Hi ej3989,
I edited your post to put "code tags" around your code. Just put the word "code" in square brackets ("[]" - like a Nasm memory reference) at the beginning of your code, and "/code" (in "[]"s) at the end. Not a big deal, but we like "code tags" around this Forum, so please humor us and use 'em, if you would.
Having said that, I don't know much about Windows programming, and don't have an answer to your question. When do you see this error message? I'm pretty sure it isn't from Nasm. From the linker? (Alink?) Or when you try to run it?
I don't know Windows, but "code is code" (so I keep claiming), and there are a couple of things about your code that look "funny" to me. I don't think it's causing your problem, but I'll point it out anyway...
cmp dword [esp+4+MSGwParam],1001
je testout
testout:
You compare a parameter on your stack to 1001. If it compares equal, you jump to "testout:". If it doesn't compare equal, you "fall through" to "testout:". It doesn't make much sense to me do do a comparison and do the same thing either way. I suspect that maybe you want to pop up that messagebox if it is 1001, and... just "ret" without doing anything if it isn't? You equate "IDC_BTN1" to 1001. You might want to use IDC_BTN1 here instead if the raw number. Just a matter of "style" (exactly the same code), but what's the point of the equate if you're not going to use it?
Just before that...
call [DialogBox]
mydialog:
You've pushed "mydialog" as a parameter to "DialogBox". When DialogBox returns (if it does - not all APIs do), you're going to "fall through" and execute "mydialog" (again?). I doubt if you want to do that. I suspect you want an ExitProcess in there. You'll need to "extern" and "import" it like the others (I think it's in kernel32.dll). Takes one parameter, zero usually indicates success...
call [DialogBox]
push 0
call [ExitProcess]
mydialog:
ExitProcess doesn't return. If DialogBox doesn't return either, my mistake. Shouldn't hurt.
One thing I remember from my very limited experience with Windows...
section .code USE32
That worked, but when I ran it in Ollydbg I got a message (Just a warning, I think, not an error) about "entrypoint outside of code segment" or so. Doing:
section .code USE32 class=CODE
used to shut Ollydbg up. Might do something useful, didn't seem to hurt...
Hopefully a real Windows programmer will be along and give you an actual answer to your problem. Good luck with it!
Best,
Frank