Author Topic: Typo in nasmx.inc and issue with invoke on win64  (Read 25771 times)

Offline miz

  • Jr. Member
  • *
  • Posts: 3
Typo in nasmx.inc and issue with invoke on win64
« on: September 10, 2013, 12:11:03 PM »
Hi,

while looking through the include file I noticed a typo (line 776)
Code: [Select]
%elifidn __OUTPUT_FORAMT__,macho64
Also, it seems that the invoke macro on win64 always writes the first four parameters to the stack, which is quite unnecessary.

Offline Frank Kotler

  • NASM Developer
  • Hero Member
  • *****
  • Posts: 2667
  • Country: us
Re: Typo in nasmx.inc and issue with invoke on win64
« Reply #1 on: September 10, 2013, 02:10:04 PM »
Good catch! Fixed in my version. Thanks, Miz!

Best,
Frank


Offline Rob Neff

  • Forum Moderator
  • Full Member
  • *****
  • Posts: 429
  • Country: us
Re: Typo in nasmx.inc and issue with invoke on win64
« Reply #2 on: September 10, 2013, 03:48:08 PM »
Thanks for the feedback, Miz!  8)
I'll correct the typo, investigate the stack usage, and update sourceforge in the near future. 

Offline Rob Neff

  • Forum Moderator
  • Full Member
  • *****
  • Posts: 429
  • Country: us
Re: Typo in nasmx.inc and issue with invoke on win64
« Reply #3 on: September 14, 2013, 07:42:49 PM »
I've updated the nasmx.inc file on sourceforge.net so the changes will be included in the next release.  I've also attached it to this message for your convenience.

The typo was obviously an easy fix.  However, the stack issue required some thought.  I don't recall the exact reasons for the current behavior but I'm pretty sure it involved possible portability issues between Win32 and Win64.  And given the fact that this framework has been out for years now we can't afford to break any existing programs which may rely on this feature.

I've added an option which is only necessary for Windows FastCall to prevent saving register parameters to shadow space ( ie: the MS definition of stack space reserved for registers ) during an INVOKE macro call.  The option can be specified on the nasm command line using a define to globally affect code generation, eg:

Code: [Select]
nasm -f win64 -DDISABLE_FASTCALL_STACK_PRELOAD

or it can be turned off and/or on again within your source code by using the syntax:
Code: [Select]
NASMX_PRAGMA FASTCALL_STACK_PRELOAD, [ ENABLE | DISABLE ]

The default INVOKE behavior on Windows is to save the register parameters to the stack before executing the call to ensure backward compatibility.  Specifying this option on BSD/Linux currently has no effect.

You may also simply remove the 3 lines in the nasmx.inc INVOKE macro where the option is located if you never want the old behavior.  Of course, you'll need to maintain your own personal copy.

Please experiment with it and share any comments you may have by replying in this thread.  I hope you find this a satisfactory solution.  Thanks again for the feedback.

Offline miz

  • Jr. Member
  • *
  • Posts: 3
Re: Typo in nasmx.inc and issue with invoke on win64
« Reply #4 on: September 15, 2013, 09:52:22 PM »
Great, thanks for the quick fix :)

Offline Bryant Keller

  • Forum Moderator
  • Full Member
  • *****
  • Posts: 360
  • Country: us
    • About Bryant Keller
Re: Typo in nasmx.inc and issue with invoke on win64
« Reply #5 on: October 18, 2013, 04:04:14 AM »
Also, it seems that the invoke macro on win64 always writes the first four parameters to the stack, which is quite unnecessary.

Actually it's not unnecessary, it's a requirement.

"The x64 Application Binary Interface (ABI) is a 4 register fast-call calling convention, with stack-backing for those registers." (MSDN)
"The caller is responsible for allocating space for parameters to the callee, and must always allocate sufficient space for the 4 register parameters, even if the callee doesn’t have that many parameters." (MSDN)

About Bryant Keller
bkeller@about.me

Offline miz

  • Jr. Member
  • *
  • Posts: 3
Re: Typo in nasmx.inc and issue with invoke on win64
« Reply #6 on: October 18, 2013, 09:12:07 AM »
Allocating the space is, writing the register parameters into that space is not.

Offline encryptor256

  • Full Member
  • **
  • Posts: 250
  • Country: lv
  • Win64 .
    • On Youtube: encryptor256
Re: Typo in nasmx.inc and issue with invoke on win64
« Reply #7 on: October 18, 2013, 09:56:27 AM »
Allocating the space is, writing the register parameters into that space is not.

Confirm!
Encryptor256's Investigation \ Research Department.

Offline Bryant Keller

  • Forum Moderator
  • Full Member
  • *****
  • Posts: 360
  • Country: us
    • About Bryant Keller
Re: Typo in nasmx.inc and issue with invoke on win64
« Reply #8 on: October 18, 2013, 05:46:05 PM »
Allocating the space is, writing the register parameters into that space is not.

I would suggest testing this quite a bit. Early versions of NASMX implemented the stack space without adding the registers on stack and IIRC we had some issues with programs that used exception handling for stack unwinding (they would crash). The fix in that case was to put the registers themselves onto the stack in proper order. The justification for this was always the two comments posted below. It worked at the time, but if there isn't a reason to do this anymore then by all means just change it to a SUB instruction (that's the way it was originally written). :D

About Bryant Keller
bkeller@about.me