Author Topic: NASM - Official Recursive Macro Support!  (Read 48075 times)

Offline Keith Kanios

  • Full Member
  • **
  • Posts: 383
  • Country: us
    • Personal Homepage
NASM - Official Recursive Macro Support!
« on: November 08, 2010, 05:26:18 PM »
This is an official announcement regarding the rewrite/revamp of the NASM preprocessor, now available as part of the official/main NASM source code starting with NASM 2.10rc2.

This rewrite addresses issues with implementing recursive macros (%rmacro/%irmacro) as previously attempted, %exitmacro and also makes room for new directives.

New NASM Directives:
  • %rmacro: Recursive Macro, can be called arbitrarily deep (really high limit [2^20] set to catch infinite loops.)
  • %irmacro: Same as above, except case-insensitive like %imacro.
  • %exitmacro: Terminates the current macro invocation, much like %exitrep; can be nested in other preprocessor directives (e.g. %if/%endif) within the macro, with exception of another macro, of course.
  • %while/%endwhile/%exitwhile: A fusion between %rep and %if.
  • %comment/%endcomment: Comment block for long/multi-line comments. A little syntactic sugar, if you will.
  • %final: Causes rest of line after %final to be preprocessed after all other "normal" preprocessing is complete. Multiple %final directives work like the x86 stack, LIFO/FILO.

This rewrite also addresses efficiency issues with heavy/nested macro invocations.

Please help us test this rewrite further by downloading/compiling the NASM 2.10rc2 and testing it against your NASM code/snippets.

NASM 2.10rc2 is available for download (source and binaries) at http://www.nasm.us/pub/nasm/releasebuilds/2.10rc2/

Please note that the official documentation will be updated over the next few weeks as we approach an official/stable NASM 2.10 release.

If there are any questions/comments/concerns, please reply to this thread.

Thanks. -Keith

Offline cm

  • Jr. Member
  • *
  • Posts: 65
Re: NASM - Official Recursive Macro Support!
« Reply #1 on: November 08, 2010, 07:28:34 PM »
A bug report. Posting it here as I guess it's on-topic enough (appears to be a 2.10rc2 issue), and the SF.net bug tracker doesn't let me post a new bug for whatever reason.

The Windows executable of 2.10rc2, as distributed, appears to crash on %fatal in included files.

Two files form a simple test case of this; assemble them with the command line "nasm test.asm".

test.asm contains:
Code: [Select]
%include "test.mac"
test.mac contains:
Code: [Select]
%fatal Test
This crashes for me with a GPF.

Notably, a local Windows build compiled by OpenWatcom 1.9 does not crash and outputs the %fatal message as expected. (I tried compiling with Visual Studio 2008 but that didn't work. Probably because of some incompatibility to VS 2005.)

Edit: This simple test case doesn't crash with my OW build, but assembling my whole project crashes the same with either the distributed build or the OW build.
« Last Edit: November 08, 2010, 07:42:25 PM by cm »
C. Masloch

Klod

  • Guest
Re: NASM - Official Recursive Macro Support!
« Reply #2 on: November 10, 2010, 03:12:38 AM »
Just downloaded Nasm 2.10rc2 win binaries.
Noted decrease in assemble time (faster)
So far all test programs I tried compiled correctly.

One observation:
Quote
%macro mac1 5
   %rep %0
      %Warning %? %0 %1 %2 %3 %4 %5
      %rotate 1
   %endrep
%endm
mac1 firstarg,2,3,4,string

:7: warning: (mac1:2) %? 5 firstarg 2 3 4 string

Loss of macroname %? %??

Regards
Klod



Offline Cyrill Gorcunov

  • NASM Developer
  • Full Member
  • *****
  • Posts: 179
  • Country: 00
Re: NASM - Official Recursive Macro Support!
« Reply #3 on: November 10, 2010, 08:19:07 PM »
A bug report. Posting it here as I guess it's on-topic enough (appears to be a 2.10rc2 issue), and the SF.net bug tracker doesn't let me post a new bug for whatever reason.

The Windows executable of 2.10rc2, as distributed, appears to crash on %fatal in included files.

Two files form a simple test case of this; assemble them with the command line "nasm test.asm".

test.asm contains:
Code: [Select]
%include "test.mac"
test.mac contains:
Code: [Select]
%fatal Test
This crashes for me with a GPF.

Notably, a local Windows build compiled by OpenWatcom 1.9 does not crash and outputs the %fatal message as expected. (I tried compiling with Visual Studio 2008 but that didn't work. Probably because of some incompatibility to VS 2005.)

Edit: This simple test case doesn't crash with my OW build, but assembling my whole project crashes the same with either the distributed build or the OW build.

The fix pushed upstream as commit 55cc4d04235cb884a885682b5a52f367ec7d50c3.
Please give it a shot by using nasm snapshot. Thanks for report!

Offline Cyrill Gorcunov

  • NASM Developer
  • Full Member
  • *****
  • Posts: 179
  • Country: 00
Re: NASM - Official Recursive Macro Support!
« Reply #4 on: November 10, 2010, 08:22:10 PM »

:7: warning: (mac1:2) %? 5 firstarg 2 3 4 string

Loss of macroname %? %??

Regards
Klod


Hi Klod, probably ;) I hope I manage to find time for this issue at this weekend.
Though no promises, not enough spare time :(

But observations from nasm users are quite important for us -- so please continue testing ;)

Offline cm

  • Jr. Member
  • *
  • Posts: 65
Re: NASM - Official Recursive Macro Support!
« Reply #5 on: November 11, 2010, 12:57:31 PM »
The fix pushed upstream as commit 55cc4d04235cb884a885682b5a52f367ec7d50c3.
Please give it a shot by using nasm snapshot. Thanks for report!

Okay, that appears to fix this problem. (Though I had to recompile on my own using OW 1.9 and the OW makefile doesn't contain the perl commands and I had to execute these manually.)

I see other issues now: Assembling my project will take a lot longer than with previous releases. During that time, NASM uses up to 50% CPU time (as I have a dual core system, this means the core NASM runs on is fully utilized) for as long as one minute. The resulting binaries appear correct.

Notably, several warnings about unterminated strings are displayed after periods of waiting. Investigating these, I found that they were generated by "code" in %if 0 blocks, which I use for long comments because of the lacking %comment in previous releases.

Further testing reveals that replacing %if 0 by %comment where appropriate does not change anything: assembling still takes long and the "unterminated string" warnings are still displayed. (This is a minor bug. I do not want these syntax checks in %comment blocks, and arguably they shouldn't be done in disabled parts of %if either.) Even more testing (with a file that contains lots of long %if 0 or %comment blocks) reveals that the time issue does not appear related to the parsing of the commented lines that results in warnings. If I find out what exactly is causing the long delay I'll let you know.

Edit: One thing I found is that in preproc.c, line 5319 (where there is a comment "sanity check: is this supposed to be here?"), the commented function call (to free_tlist) should indeed be executed. Of course I'm not very familiar with the source, but it seems the right thing to do. Assembling my project with this line commented (as in the repo) takes about 50 seconds, and peaks at a memory usage of 70 MiB. With the line uncommented, it takes about 30 seconds and peaks at 45 MiB. Notably, with 2.10rc1 the same action takes about 3 seconds and peaks at 6 MiB. All produce the same result. So the current revision certainly has some leaks and possibly other stuff slowing it down. The mentioned function call appears to solve some of these.
« Last Edit: November 11, 2010, 04:08:06 PM by cm »
C. Masloch

Offline Cyrill Gorcunov

  • NASM Developer
  • Full Member
  • *****
  • Posts: 179
  • Country: 00
Re: NASM - Official Recursive Macro Support!
« Reply #6 on: November 11, 2010, 07:42:03 PM »
The fix pushed upstream as commit 55cc4d04235cb884a885682b5a52f367ec7d50c3.
Please give it a shot by using nasm snapshot. Thanks for report!

Okay, that appears to fix this problem. (Though I had to recompile on my own using OW 1.9 and the OW makefile doesn't contain the perl commands and I had to execute these manually.)

I see other issues now: Assembling my project will take a lot longer than with previous releases. During that time, NASM uses up to 50% CPU time (as I have a dual core system, this means the core NASM runs on is fully utilized) for as long as one minute. The resulting binaries appear correct.

Notably, several warnings about unterminated strings are displayed after periods of waiting. Investigating these, I found that they were generated by "code" in %if 0 blocks, which I use for long comments because of the lacking %comment in previous releases.

Further testing reveals that replacing %if 0 by %comment where appropriate does not change anything: assembling still takes long and the "unterminated string" warnings are still displayed. (This is a minor bug. I do not want these syntax checks in %comment blocks, and arguably they shouldn't be done in disabled parts of %if either.) Even more testing (with a file that contains lots of long %if 0 or %comment blocks) reveals that the time issue does not appear related to the parsing of the commented lines that results in warnings. If I find out what exactly is causing the long delay I'll let you know.

Edit: One thing I found is that in preproc.c, line 5319 (where there is a comment "sanity check: is this supposed to be here?"), the commented function call (to free_tlist) should indeed be executed. Of course I'm not very familiar with the source, but it seems the right thing to do. Assembling my project with this line commented (as in the repo) takes about 50 seconds, and peaks at a memory usage of 70 MiB. With the line uncommented, it takes about 30 seconds and peaks at 45 MiB. Notably, with 2.10rc1 the same action takes about 3 seconds and peaks at 6 MiB. All produce the same result. So the current revision certainly has some leaks and possibly other stuff slowing it down. The mentioned function call appears to solve some of these.

ok, new preproc code needs to be polished that is why we pushed it out early and want the feedback ;)
If this is not a secret -- what kind of project you have and if it's open-sourced mind to provide the
code so we could be able to compile it and figure out what eats cpu time.

Offline Cyrill Gorcunov

  • NASM Developer
  • Full Member
  • *****
  • Posts: 179
  • Country: 00
Re: NASM - Official Recursive Macro Support!
« Reply #7 on: November 11, 2010, 07:47:25 PM »
I read inaccurate -- you've explained what is happening. Will take a look. Thanks!

Offline cm

  • Jr. Member
  • *
  • Posts: 65
Re: NASM - Official Recursive Macro Support!
« Reply #8 on: November 11, 2010, 08:42:36 PM »
Quote from: Cyrill Gorcunov
ok, new preproc code needs to be polished that is why we pushed it out early and want the feedback ;)

Eh yeah. I appreciate it I guess. It has good new features and maybe we can make the speed-ups noticeable for me too :D

Quote
If this is not a secret -- what kind of project you have and if it's open-sourced mind to provide the
code so we could be able to compile it and figure out what eats cpu time.

I read inaccurate -- you've explained what is happening. Will take a look. Thanks!

Yeah, but with that it still takes multiple times longer than the revision from before the preproc-rewrite merge. So there is still some work to do. My stuff isn't ready so there are no public releases yet. But it is free software (2-clause BSD) so if you point me to where I can find your e-mail address, or write me a PM with it or something, it's no problem for me to send a current copy to you. Uncompressed it is about 600 KiB large. (It's also a program for DOS, but since the only build tool used is NASM (with multi-section bin output :)) that shouldn't be an issue for you.)
« Last Edit: November 11, 2010, 09:37:55 PM by cm »
C. Masloch

Offline Cyrill Gorcunov

  • NASM Developer
  • Full Member
  • *****
  • Posts: 179
  • Country: 00
Re: NASM - Official Recursive Macro Support!
« Reply #9 on: November 12, 2010, 07:01:58 PM »
yup, gorcunov@gmail.com

Offline Keith Kanios

  • Full Member
  • **
  • Posts: 383
  • Country: us
    • Personal Homepage
Re: NASM - Official Recursive Macro Support!
« Reply #10 on: November 13, 2010, 03:42:10 PM »
:7: warning: (mac1:2) %? 5 firstarg 2 3 4 string

Loss of macroname %? %??

A fix has been pushed to the source code repository.

Klod

  • Guest
Re: NASM - Official Recursive Macro Support!
« Reply #11 on: November 13, 2010, 11:56:36 PM »

Quote
Keith Kanios:
A fix has been pushed to the source code repository.

I do not have the resources to compile the fix with my setup, I will wait for an updated snapshot

Offline Keith Kanios

  • Full Member
  • **
  • Posts: 383
  • Country: us
    • Personal Homepage
Re: NASM - Official Recursive Macro Support!
« Reply #12 on: December 18, 2010, 04:58:40 PM »
Notably, several warnings about unterminated strings are displayed after periods of waiting. Investigating these, I found that they were generated by "code" in %if 0 blocks, which I use for long comments because of the lacking %comment in previous releases.

Pushed a simple fix for this.

Offline Keith Kanios

  • Full Member
  • **
  • Posts: 383
  • Country: us
    • Personal Homepage
Re: NASM - Official Recursive Macro Support!
« Reply #13 on: December 18, 2010, 05:08:45 PM »
One thing I found is that in preproc.c, line 5319 (where there is a comment "sanity check: is this supposed to be here?"), the commented function call (to free_tlist) should indeed be executed. Of course I'm not very familiar with the source, but it seems the right thing to do. Assembling my project with this line commented (as in the repo) takes about 50 seconds, and peaks at a memory usage of 70 MiB. With the line uncommented, it takes about 30 seconds and peaks at 45 MiB. Notably, with 2.10rc1 the same action takes about 3 seconds and peaks at 6 MiB. All produce the same result. So the current revision certainly has some leaks and possibly other stuff slowing it down. The mentioned function call appears to solve some of these.

Uncommenting it does indeed pass tests and still builds various code. I probably commented that when I was tracking down another related bug.

I've uncommented the line and pushed to GIT.

Offline cm

  • Jr. Member
  • *
  • Posts: 65
Re: NASM - Official Recursive Macro Support!
« Reply #14 on: December 18, 2010, 07:16:48 PM »
Notably, several warnings about unterminated strings are displayed after periods of waiting. Investigating these, I found that they were generated by "code" in %if 0 blocks, which I use for long comments because of the lacking %comment in previous releases.

Pushed a simple fix for this.

I've looked into the source myself and did not yet implement a good solution. In your fix, are these error conditions still handled properly when expanding non-%comment ExpDefs later?

Besides, it isn't optimal that tokenize() is always used - even if the (untokenized, ie input) line was only read to be discarded. Anything except certain preproc directives is discarded in %comment blocks, and ignored %if parts (and some other less common cases). My current method of checking for whitespace, then a preproc ID with a letter, and discarding the line if it doesn't look like that, actually decreases performance (by a noticeable amount!) in some reference cases though. A better method might be to check there only for the specific directives and processing them directly if an expansion is currently defined. But implementing that well might require understanding the hash table creation and lookup, which I don't. Just rambling here.

Your fix for %unmacro on a currently active macro is okay but I'm currently playing around with a mechanism where the check that frees any ExpDef except macro ones will additionally check for a certain flag - set by %unmacro if it would display your error. I store that flag in the `state' field as that's only used by %if (and maybe %while).

I'll probably get that %unmacro patch (along with a bunch of other preproc patches) ready during the holidays.

EDIT: By the way, I think the linked list `expansions' wasn't properly used anywhere in the repo. Just confused me a little when I reviewed/compared one of your patches right now.
« Last Edit: December 18, 2010, 07:33:10 PM by cm »
C. Masloch