NASM - Official Recursive Macro Support!

[Keith Kanios]

This is an official announcement regarding the rewrite/revamp of the NASM preprocessor, now available as part of the official/main NASM source code starting with NASM 2.10rc2.

This rewrite addresses issues with implementing recursive macros (%rmacro/%irmacro) as previously attempted, %exitmacro and also makes room for new directives.

New NASM Directives:

• %rmacro: Recursive Macro, can be called arbitrarily deep (really high limit [2^20] set to catch infinite loops.)
• %irmacro: Same as above, except case-insensitive like %imacro.
• %exitmacro: Terminates the current macro invocation, much like %exitrep; can be nested in other preprocessor directives (e.g. %if/%endif) within the macro, with exception of another macro, of course.
• %while/%endwhile/%exitwhile: A fusion between %rep and %if.
• %comment/%endcomment: Comment block for long/multi-line comments. A little syntactic sugar, if you will.
• %final: Causes rest of line after %final to be preprocessed after all other "normal" preprocessing is complete. Multiple %final directives work like the x86 stack, LIFO/FILO.

This rewrite also addresses efficiency issues with heavy/nested macro invocations.

Please help us test this rewrite further by downloading/compiling the NASM 2.10rc2 and testing it against your NASM code/snippets.

NASM 2.10rc2 is available for download (source and binaries) at http://www.nasm.us/pub/nasm/releasebuilds/2.10rc2/

Please note that the official documentation will be updated over the next few weeks as we approach an official/stable NASM 2.10 release.

If there are any questions/comments/concerns, please reply to this thread.

Thanks. -Keith

[cm]

A bug report. Posting it here as I guess it's on-topic enough (appears to be a 2.10rc2 issue), and the SF.net bug tracker doesn't let me post a new bug for whatever reason.

The Windows executable of 2.10rc2, as distributed, appears to crash on %fatal in included files.

Two files form a simple test case of this; assemble them with the command line "nasm test.asm".

test.asm contains:

Code: [Select]

%include "test.mac"
test.mac contains:

Code: [Select]

%fatal Test
This crashes for me with a GPF.

Notably, a local Windows build compiled by OpenWatcom 1.9 does not crash and outputs the %fatal message as expected. (I tried compiling with Visual Studio 2008 but that didn't work. Probably because of some incompatibility to VS 2005.)

Edit: This simple test case doesn't crash with my OW build, but assembling my whole project crashes the same with either the distributed build or the OW build.

[Klod]

Just downloaded Nasm 2.10rc2 win binaries.
Noted decrease in assemble time (faster)
So far all test programs I tried compiled correctly.

One observation:

%macro mac1 5
   %rep %0
      %Warning %? %0 %1 %2 %3 %4 %5
      %rotate 1
   %endrep
%endm
mac1 firstarg,2,3,4,string

:7: warning: (mac1:2) %? 5 firstarg 2 3 4 string

Loss of macroname %? %??

Regards
Klod

[Cyrill Gorcunov]

A bug report. Posting it here as I guess it's on-topic enough (appears to be a 2.10rc2 issue), and the SF.net bug tracker doesn't let me post a new bug for whatever reason.

The Windows executable of 2.10rc2, as distributed, appears to crash on %fatal in included files.

Two files form a simple test case of this; assemble them with the command line "nasm test.asm".

test.asm contains:

Code: [Select]

%include "test.mac"
test.mac contains:

Code: [Select]

%fatal Test
This crashes for me with a GPF.

Notably, a local Windows build compiled by OpenWatcom 1.9 does not crash and outputs the %fatal message as expected. (I tried compiling with Visual Studio 2008 but that didn't work. Probably because of some incompatibility to VS 2005.)

Edit: This simple test case doesn't crash with my OW build, but assembling my whole project crashes the same with either the distributed build or the OW build.

The fix pushed upstream as commit 55cc4d04235cb884a885682b5a52f367ec7d50c3.
Please give it a shot by using nasm snapshot. Thanks for report!

[Cyrill Gorcunov]

:7: warning: (mac1:2) %? 5 firstarg 2 3 4 string

Loss of macroname %? %??

Regards
Klod

Hi Klod, probably I hope I manage to find time for this issue at this weekend.
Though no promises, not enough spare time

But observations from nasm users are quite important for us -- so please continue testing

[cm]

The fix pushed upstream as commit 55cc4d04235cb884a885682b5a52f367ec7d50c3.
Please give it a shot by using nasm snapshot. Thanks for report!

Okay, that appears to fix this problem. (Though I had to recompile on my own using OW 1.9 and the OW makefile doesn't contain the perl commands and I had to execute these manually.)

I see other issues now: Assembling my project will take a lot longer than with previous releases. During that time, NASM uses up to 50% CPU time (as I have a dual core system, this means the core NASM runs on is fully utilized) for as long as one minute. The resulting binaries appear correct.

Notably, several warnings about unterminated strings are displayed after periods of waiting. Investigating these, I found that they were generated by "code" in %if 0 blocks, which I use for long comments because of the lacking %comment in previous releases.

Further testing reveals that replacing %if 0 by %comment where appropriate does not change anything: assembling still takes long and the "unterminated string" warnings are still displayed. (This is a minor bug. I do not want these syntax checks in %comment blocks, and arguably they shouldn't be done in disabled parts of %if either.) Even more testing (with a file that contains lots of long %if 0 or %comment blocks) reveals that the time issue does not appear related to the parsing of the commented lines that results in warnings. If I find out what exactly is causing the long delay I'll let you know.

Edit: One thing I found is that in preproc.c, line 5319 (where there is a comment "sanity check: is this supposed to be here?"), the commented function call (to free_tlist) should indeed be executed. Of course I'm not very familiar with the source, but it seems the right thing to do. Assembling my project with this line commented (as in the repo) takes about 50 seconds, and peaks at a memory usage of 70 MiB. With the line uncommented, it takes about 30 seconds and peaks at 45 MiB. Notably, with 2.10rc1 the same action takes about 3 seconds and peaks at 6 MiB. All produce the same result. So the current revision certainly has some leaks and possibly other stuff slowing it down. The mentioned function call appears to solve some of these.

[Cyrill Gorcunov]

The fix pushed upstream as commit 55cc4d04235cb884a885682b5a52f367ec7d50c3.
Please give it a shot by using nasm snapshot. Thanks for report!

Okay, that appears to fix this problem. (Though I had to recompile on my own using OW 1.9 and the OW makefile doesn't contain the perl commands and I had to execute these manually.)

I see other issues now: Assembling my project will take a lot longer than with previous releases. During that time, NASM uses up to 50% CPU time (as I have a dual core system, this means the core NASM runs on is fully utilized) for as long as one minute. The resulting binaries appear correct.

Notably, several warnings about unterminated strings are displayed after periods of waiting. Investigating these, I found that they were generated by "code" in %if 0 blocks, which I use for long comments because of the lacking %comment in previous releases.

Further testing reveals that replacing %if 0 by %comment where appropriate does not change anything: assembling still takes long and the "unterminated string" warnings are still displayed. (This is a minor bug. I do not want these syntax checks in %comment blocks, and arguably they shouldn't be done in disabled parts of %if either.) Even more testing (with a file that contains lots of long %if 0 or %comment blocks) reveals that the time issue does not appear related to the parsing of the commented lines that results in warnings. If I find out what exactly is causing the long delay I'll let you know.

Edit: One thing I found is that in preproc.c, line 5319 (where there is a comment "sanity check: is this supposed to be here?"), the commented function call (to free_tlist) should indeed be executed. Of course I'm not very familiar with the source, but it seems the right thing to do. Assembling my project with this line commented (as in the repo) takes about 50 seconds, and peaks at a memory usage of 70 MiB. With the line uncommented, it takes about 30 seconds and peaks at 45 MiB. Notably, with 2.10rc1 the same action takes about 3 seconds and peaks at 6 MiB. All produce the same result. So the current revision certainly has some leaks and possibly other stuff slowing it down. The mentioned function call appears to solve some of these.

ok, new preproc code needs to be polished that is why we pushed it out early and want the feedback
If this is not a secret -- what kind of project you have and if it's open-sourced mind to provide the
code so we could be able to compile it and figure out what eats cpu time.

[Cyrill Gorcunov]

I read inaccurate -- you've explained what is happening. Will take a look. Thanks!

[cm]

ok, new preproc code needs to be polished that is why we pushed it out early and want the feedback

Eh yeah. I appreciate it I guess. It has good new features and maybe we can make the speed-ups noticeable for me too

If this is not a secret -- what kind of project you have and if it's open-sourced mind to provide the
code so we could be able to compile it and figure out what eats cpu time.

I read inaccurate -- you've explained what is happening. Will take a look. Thanks!

Yeah, but with that it still takes multiple times longer than the revision from before the preproc-rewrite merge. So there is still some work to do. My stuff isn't ready so there are no public releases yet. But it is free software (2-clause BSD) so if you point me to where I can find your e-mail address, or write me a PM with it or something, it's no problem for me to send a current copy to you. Uncompressed it is about 600 KiB large. (It's also a program for DOS, but since the only build tool used is NASM (with multi-section bin output that shouldn't be an issue for you.)

[Cyrill Gorcunov]

yup, gorcunov@gmail.com

[Keith Kanios]

:7: warning: (mac1:2) %? 5 firstarg 2 3 4 string

Loss of macroname %? %??

A fix has been pushed to the source code repository.

[Klod]

Keith Kanios:
A fix has been pushed to the source code repository.

I do not have the resources to compile the fix with my setup, I will wait for an updated snapshot

[Keith Kanios]

Notably, several warnings about unterminated strings are displayed after periods of waiting. Investigating these, I found that they were generated by "code" in %if 0 blocks, which I use for long comments because of the lacking %comment in previous releases.

Pushed a simple fix for this.

[Keith Kanios]

One thing I found is that in preproc.c, line 5319 (where there is a comment "sanity check: is this supposed to be here?"), the commented function call (to free_tlist) should indeed be executed. Of course I'm not very familiar with the source, but it seems the right thing to do. Assembling my project with this line commented (as in the repo) takes about 50 seconds, and peaks at a memory usage of 70 MiB. With the line uncommented, it takes about 30 seconds and peaks at 45 MiB. Notably, with 2.10rc1 the same action takes about 3 seconds and peaks at 6 MiB. All produce the same result. So the current revision certainly has some leaks and possibly other stuff slowing it down. The mentioned function call appears to solve some of these.

Uncommenting it does indeed pass tests and still builds various code. I probably commented that when I was tracking down another related bug.

I've uncommented the line and pushed to GIT.

[cm]

Notably, several warnings about unterminated strings are displayed after periods of waiting. Investigating these, I found that they were generated by "code" in %if 0 blocks, which I use for long comments because of the lacking %comment in previous releases.

Pushed a simple fix for this.

I've looked into the source myself and did not yet implement a good solution. In your fix, are these error conditions still handled properly when expanding non-%comment ExpDefs later?

Besides, it isn't optimal that tokenize() is always used - even if the (untokenized, ie input) line was only read to be discarded. Anything except certain preproc directives is discarded in %comment blocks, and ignored %if parts (and some other less common cases). My current method of checking for whitespace, then a preproc ID with a letter, and discarding the line if it doesn't look like that, actually decreases performance (by a noticeable amount!) in some reference cases though. A better method might be to check there only for the specific directives and processing them directly if an expansion is currently defined. But implementing that well might require understanding the hash table creation and lookup, which I don't. Just rambling here.

Your fix for %unmacro on a currently active macro is okay but I'm currently playing around with a mechanism where the check that frees any ExpDef except macro ones will additionally check for a certain flag - set by %unmacro if it would display your error. I store that flag in the `state' field as that's only used by %if (and maybe %while).

I'll probably get that %unmacro patch (along with a bunch of other preproc patches) ready during the holidays.

EDIT: By the way, I think the linked list `expansions' wasn't properly used anywhere in the repo. Just confused me a little when I reviewed/compared one of your patches right now.