NASM - The Netwide Assembler

NASM Forum => Programming with NASM => Topic started by: lukassevc on May 16, 2019, 06:23:51 AM

Title: how to switch video modes (switching to VESA video modes too) in protected mode
Post by: lukassevc on May 16, 2019, 06:23:51 AM
Hi, I am Lukas,
and I am writing my operating system but I have problem with switching video modes ( switching to VESA modes too ) because I don't know how to do it and I don't found any tutorial on it so I need help. I know that in real mode it can by done by using ah = 0x00 and with interruption 0x10, but I am in protected mode so I don't know how to do it. Can someone help me please?
Thanks for any help!
Title: Re: how to switch video modes (switching to VESA video modes too) in protected mode
Post by: fredericopissarra on May 16, 2019, 11:43:33 AM
http://www.petesqbsite.com/sections/tutorials/tuts/vbe3.pdf

See "Obtaining the Protected Mode Entry Point", page 21 -- this is NOT available in all implementations and I was never been able to make it work properly...
Title: Re: how to switch video modes (switching to VESA video modes too) in protected mode
Post by: lukassevc on May 16, 2019, 03:58:44 PM
http://www.petesqbsite.com/sections/tutorials/tuts/vbe3.pdf

See "Obtaining the Protected Mode Entry Point", page 21 -- this is NOT available in all implementations and I was never been able to make it work properly...

I read it, and it seems to that it can work properly, but can you please tell me how to do it in C language?
Thanks!
Title: Re: how to switch video modes (switching to VESA video modes too) in protected mode
Post by: fredericopissarra on May 17, 2019, 12:29:30 PM
I read it, and it seems to that it can work properly, but can you please tell me how to do it in C language?
I could, but I thought this forum was about assembly language, isn't it?
Title: Re: how to switch video modes (switching to VESA video modes too) in protected mode
Post by: ace012k8 on April 09, 2025, 01:00:47 PM
I ran into the same headache back when I was experimenting with writing a small OS kernel for fun. Switching to VESA modes in protected mode easily becomes a rabbit hole. I had to do a bunch of hacks just to get the BIOS to cooperate. The legacy BIOS interrupt (INT 0x10) just won't fly once you're out of real mode, so I remember relying on setting up a trampoline to call into real mode temporarily or using something like a VBE 2.0+ Linear Framebuffer approach after querying video modes beforehand.

For videos and demo recording, I always used a lightweight editor to polish up footage or build quick walkthroughs. If you’re working on dev-related content and want to make clean quick edits, especially on Windows, I picked up a solid one here: https://www.movavi.com/imovie-for-windows/ (https://www.movavi.com/imovie-for-windows/). Easy to use and doesn’t overcomplicate things.
Title: Re: how to switch video modes (switching to VESA video modes too) in protected mode
Post by: fredericopissarra on April 13, 2025, 01:01:59 PM
I ran into the same headache back when I was experimenting with writing a small OS kernel for fun. Switching to VESA modes in protected mode easily becomes a rabbit hole. I had to do a bunch of hacks just to get the BIOS to cooperate. The legacy BIOS interrupt (INT 0x10) just won't fly once you're out of real mode, so I remember relying on setting up a trampoline to call into real mode temporarily or using something like a VBE 2.0+ Linear Framebuffer approach after querying video modes beforehand.
Yep... OR you could setup and use Virtual 8086 mode (NOT available in x86-64 mode!) to do it... Anyway, it is a nightmare.

Another problems are: Getting the Linear Framebuffer Physical address from ModeBlockInfo strucuture... Often this field is zeroed in VMs like QEMU, for example... And page switching, for double buffering... another nightmare.

This and the Protect Mode Interface, described in VBE 3.0 documentation usually are not present (not even in Option-ROM that a nVidia card puts there)...
Title: Re: how to switch video modes (switching to VESA video modes too) in protected mode
Post by: marknoble on April 25, 2025, 04:37:33 PM
For VESA in protected mode, you gotta use VBE functions, not BIOS interrupts. Check the VBE specs on OSDev wiki for mode info and framebuffer setup. It’s tricky but doable. What bootloader you using? Might help narrow it down.
Title: Re: how to switch video modes (switching to VESA video modes too) in protected mode
Post by: ben321 on April 29, 2025, 07:31:19 AM
Hi, I am Lukas,
and I am writing my operating system but I have problem with switching video modes ( switching to VESA modes too ) because I don't know how to do it and I don't found any tutorial on it so I need help. I know that in real mode it can by done by using ah = 0x00 and with interruption 0x10, but I am in protected mode so I don't know how to do it. Can someone help me please?
Thanks for any help!

That's because the interrupts work differently in protected mode. In protected mode the CPU ignores the IVT (interrupt vector table) that was used in 16bit real mode, and instead requires you to set up an IDT (interrupt descriptor table). In 16bit protected mode, you might be able to (if you're lucky) get away with pointing the IDT to the same interrupt functions that the IVT originally pointed to (but if not, then you'll need to take the same steps as if you were working with 32bit protected mode). But if you mean 32bit protected mode, then that's not a possible shortcut at all, because the functions pointed to by the IVT are all written in 16bit code, not 32bit code. So you will need to write your own 32bit functions (or 16bit functions if you are talking about 16bit protected mode) for EVERY SINGLE FUNCTION that exists in the IVT. That's 256 functions you will need to write! Not worth the effort, if you are doing anything other than writing an entire operating system. Since I assume you are writing software, not an OS, I would advise you to not bother with this effort, unless you really just want to go through with it for the sake of learning how to do it (maybe as a project in a programming class at a university or something).
Title: Re: how to switch video modes (switching to VESA video modes too) in protected mode
Post by: fredericopissarra on April 29, 2025, 02:31:39 PM
For VESA in protected mode, you gotta use VBE functions, not BIOS interrupts. Check the VBE specs on OSDev wiki for mode info and framebuffer setup. It’s tricky but doable. What bootloader you using? Might help narrow it down.
You can do this only with PMI is available to you. It isn't the case of QEMU and even at ROM BIOS extentions of major modern graphics cards like nVidia's (at least I can't find it using the described method from VBE3 documentation).
Title: Re: how to switch video modes (switching to VESA video modes too) in protected mode
Post by: Tobiasz Stamborski on May 17, 2025, 02:26:01 PM
Hi, I am Lukas,
and I am writing my operating system but I have problem with switching video modes ( switching to VESA modes too ) because I don't know how to do it and I don't found any tutorial on it so I need help. I know that in real mode it can by done by using ah = 0x00 and with interruption 0x10, but I am in protected mode so I don't know how to do it. Can someone help me please?

It don't know does it helpful for you, because it's just about regular VGA modes but:
1. In protected mode you haven't access to BIOS interrupts as you already know. You must to deal directly with VGA i/o registers by self.
2. Here you have the short instruction about VGA registers (how to use them):
https://wiki.osdev.org/VGA_Hardware#VGA_Registers (https://wiki.osdev.org/VGA_Hardware#VGA_Registers)
3. Here you have a proper register settings for few VGA modes (and how you should safely set them up):
https://wiki.osdev.org/VGA_Hardware#Sample_Register_Settings (https://wiki.osdev.org/VGA_Hardware#Sample_Register_Settings)

Generally it's a lot of work before you, so don't expect a simple answer. I think links above are good place to start. BTW: Not so long ago I was trying to write my own simple OS too. I know your pain. :)