Author Topic: How does SYSENTER opcode work?  (Read 6514 times)

Offline ben321

  • Full Member
  • **
  • Posts: 182
How does SYSENTER opcode work?
« on: March 24, 2016, 04:55:08 AM »
Ok, so I'm in Windows, and I've heard that the only way to use powerful low-level commands like int for interrupt (which can easily be used directly in DOS, but not Windows) is to be in Kernel mode, which is different than the normal User mode. I've also heard that the quickest way to enter Kernel mode from user mode is to call the x86 opcode SYSENTER (and an equivalent SYSEXIT opcode will need to be called while in Kernel mode to get back to user mode). To test this out, I decided to make a simple program, where the only thing it did was enter Kernel mode. The one and only instruction it contains is SYSENTER. I then ran it in OllyDbg, to see just what will happen when SYSENTER is called. Turns out, it doesn't do anything useful at all, just immediately generates an error. The system error that OllyDbg reports appears on the status bar in the main OllyDbg window. The error says:
Access violation when reading [FFFFFFFF]

I've got a question about this. I know what access violation reading means. It means that at the specified address, it is unable to read the memory there. But my question is where is the address FFFFFFFF coming from? I haven't given it any memory address to work with (pushed no arguments onto a stack, nor moved any values into any registers). What is going on? And how do I PROPERLY use the x86 instruction SYSENTER in Windows?

Can someone maybe provide a bit of sample code that actually works? And please, no Linux samples. I've found plenty Linux things regarding the SYSENTER opcode, but not a single one for Windows. I need one for how to use it in Windows.