I'm not sure what you mean by "DOS video mode". Mode 13h? Pretty chunky. Problem is, when we get beyond the VGA modes, the cards aren't standardized, and the manufacturers aren't too free with the information. If your OS is capable of using BIOS, look into the vesa specs.
What I'd do for a "toy OS" is put the card in a 640 x 480 mode (easy to draw characters) - probably "linear frame buffer", if I could get it - *before* switching to protected mode, using the bios. I don't know if that would fit in with your plans, though...
If you plan to write your own drivers, taking advantage of "hardware acceleration" where available and all - Whew! Big job! But it *can* be done, of course, and Nasm should be a suitable tool...
Best,
Frank