Debugging 32bit exe with OllyDbg

[AndrejM]

Hi,

I have the following NASM file (test.asm):

Code: [Select]

%include "asm_io.inc"

segment .text USE32
        global  _asm_main
_asm_main:
        enter   0,0               ; setup routine
        pusha

        mov eax, 0xFFFF_FFFF
        
        popa
        mov     eax, 0            ; return back to C
        leave                    
        ret
I'm compiling it with debugging symbols, and I get back a 32bit OMF object file:
nasm -l test.lst -f obj -g test.asm -I%cd%\pcasm\

I'm linking it with another object file from another language (D) which calls the exported _asm_main function. The real main() function is defined in the driver file written in D. Anyway, this whole thing compiles and links without a hitch, I can call NASM from D and vice-versa (D uses the same OMF format just like Delphi).

Long story short:
I can debug with Visual Studio, see this screenshot:
http://i.imgur.com/U7JQp.jpg

But, I've heard great things about OllyDbg. It's probably more powerful than VS's integrated debugger, and VS is a resource hog as it is. However, I haven't figured out a way to load a source file into Olly and add a breakpoint on a line. Is this possible in Olly?

I'd appreciate any help for this ASM newbie. Cheers!

[AndrejM]

The screenshot actually shows a different asm file. But I've pasted just a short example in my post so you don't have to read through all of that.

[Frank Kotler]

I dunno. What's it say in the Friendly Manual? I've got a very old version, and the manual is in RTF, which I can't really read... I can see a reference to "breakpoint", but I can't make out what it says...

From my very small experience with OllyDbg, a long time ago, I recall that Olly complained that the entrypoint was outside the code segment (but debugged it anyway, as I recall). A modification to the segment declaration fixed it:

Code: [Select]

segment .text USE32 class=CODE

This was with the entrypoint (the "..start" symbol) in my file. Since your entrypoint is elsewhere, this may not apply to you, but might make OllyDbg friendlier to your code(?).

You might want to read this topic:

http://forum.nasm.us/index.php?topic=880.0

Essentially, Nasm's implementation of 32-bit OMF is... rudimentary. Assembling to "-f win32" and converting to OMF with Agner Fog's "objconv" was working better with Borland tools, for this user.

Neither of these really addresses your question about breakpoints, but is about all I know on the subject (old politician's trick: if you don't know the answer to the question, answer some question you do know the answer to ) Maybe someone with more experience with OllyDbg can be more help.

Looks like you're converting some of Dr. Carter's stuff from C to D(?). Keep us posted on how it goes!

Best,
Frank

[AndrejM]

Looks like you're converting some of Dr. Carter's stuff from C to D(?). Keep us posted on how it goes!

Well in his book he calls assembly code from C, probably to eliminate platform specifics. I'm really interested in how C/C++/D work at a low-level, so I felt it was time to grab an ASM book.

It looks like nasm can't generate debug info with win32 (COFF, my guess) object files, but it can if I use OMF. Since D's standard compiler uses OMF, I've tried linking a D driver app with a nasm object. It seems to work fine, albeit with a few linker errors from D's linker (Optlink).

Interestingly, D provides an inline asm, so for the most part I don't nasm. But since there might be some syntax differences I chose to use nasm anyway.

I'll try to RTFM and see if there's something there. Thanks!

[AndrejM]

It seems to work fine, albeit with a few linker errors from D's linker (Optlink).

I meant warnings, not errors.

[Frank Kotler]

Well... googling for "D" is going to be useless so I tried "D optlink". All "problem" reports! This doesn't look promising. I guess if all you're getting is warnings, you're doing great!

You're right that Nasm doesn't do debug info in "-f win32" - the "-g" switch, if you use it, is silently ignored (probably should warn...). In "-f obj" output format, the debug info is in "borland" format. I'm not sure it's 100% correct. I'm not sure how either VS or OllyDbg will like it. If it seems to be working at all, that's good...

If you're using VS to invoke Nasm to assemble your code, and if there are errors, the "-Xvc" switch on Nasm's command line may help. It's a minor change in syntax in Nasm's error reporting from the default "-Xgnu". Supposedly helps MS tools locate the error in its editor. If you're not doing it that way, it won't help. Shouldn't make any difference to OllyDbg.

After you're done with C, C++, and D, we've got some examples of interfacing with Fortran around here someplace, too!

Best,
Frank

[AndrejM]

Well... googling for "D" is going to be useless so I tried "D optlink". All "problem" reports! This doesn't look promising. I guess if all you're getting is warnings, you're doing great!

Yes, D might be a bit hard to search for. I'm not surprised about you finding problem reports all over the place. Optlink is just about the most hated piece of software in the D community. Written in 100% ASM by Walter Bright. But it's being slowly rewritten to C. It's the only linker we can currently use on Windows, although on Linux we can use the GNU tool set for linking. There might be another linker that will support D soon (called "Unilink"). D was originally made by Walter Bright and it has some nice talented people contributing to it, one of which is Andrei Alexandrescu. At the moment the 1.x branch of D is kinda-stable, but most of the focus is on version 2 of the language, which is quite complex compared to v1. There's also a book out, The D Programming Language, by Alexandrescu.

I doubt any ASM programmers around here are interested in high-level languages, bit I thought I'd just mention a few words about D.

You're right that Nasm doesn't do debug info in "-f win32" - the "-g" switch, if you use it, is silently ignored (probably should warn...). In "-f obj" output format, the debug info is in "borland" format. I'm not sure it's 100% correct. I'm not sure how either VS or OllyDbg will like it. If it seems to be working at all, that's good...

Well actually there's a plugin system that wraps D support into Visual Studio. The project is called VisualD. DMD (the reference D compiler) spits out its own extended version of codeview for the debug symbols, and VisualD can convert it to a format that's readable by Visual Studio.

If you're using VS to invoke Nasm to assemble your code, and if there are errors, the "-Xvc" switch on Nasm's command line may help. It's a minor change in syntax in Nasm's error reporting from the default "-Xgnu". Supposedly helps MS tools locate the error in its editor. If you're not doing it that way, it won't help. Shouldn't make any difference to OllyDbg.

That's fantastic, thanks!

After you're done with C, C++, and D, we've got some examples of interfacing with Fortran around here someplace, too!

I don't think I'll be needing that.

[Keith Kanios]

I doubt any ASM programmers around here are interested in high-level languages, bit I thought I'd just mention a few words about D.

I'm quite interested in D. But, between the lack of 64-bit inline asm support, the seemingly perpetual design phase, ambiguity over the defacto standard library (tango vs phobos vs druntime) and immature state of alternative compilers such as LDC, it has been hard to stay enthusiastic about the language.

That being said, I have developed an initial OpenGL-based game engine in D, and it was quite a pleasant experience as compared to the inconsistent mess that is C++.

I'm not sure what advice I can offer your in regard to debugging formats, other than trying GDC or LDC.

[AndrejM]

I've found this from another part of this forum:
Debugging Nasm with OllyDbg on Windows

Thanks to whoever is the owner of that site. It looks like quite a nice set of tutorials, although most seem to be in Spanish.