Recursive Macros and Beyond!

[Keith Kanios]

This is a preliminary announcement regarding the rewrite/revamp of the NASM preprocessor.

This rewrite addresses issues with implementing recursive macros (%rmacro/%irmacro) as previously attempted, %exitmacro and also makes room for new directives.

New NASM Directives:

• %rmacro: Recursive Macro, can be called arbitrarily deep (really high limit [2^20] set to catch infinite loops.)
• %irmacro: Same as above, except case-insensitive like %imacro.
• %exitmacro: Terminates the current macro invocation, much like %exitrep; can be nested in other preprocessor directives (e.g. %if/%endif) within the macro, with exception of another macro, of course.
• %while/%endwhile/%exitwhile: A fusion between %rep and %if.
• %comment/%endcomment: Comment block for long/multi-line comments. A little syntactic sugar, if you will.
• %final: Causes rest of line after %final to be preprocessed after all other "normal" preprocessing is complete. Multiple %final directives work like the x86 stack, LIFO/FILO.

This rewrite also attempts to address heavy/nested macro invocation efficiency issues.

This rewrite is currently a branch [in GIT] called preprocessor-rewrite, and tentatively labeled as version 2.10rc1. Please note that NASM 2.09rc (official/main/master) is still going and NASM 2.09 should be released fairly soon.

This rewrite has been known to accurately build all NASMX demos, along with a variety of other code snippets by various people.

As positive/successful as this rewrite has been thus far, it is still a major change and we would like to catch as many bugs as possible before merging it into the main/master NASM code base.

Please help us test this rewrite further by downloading/compiling the preprocessor-rewrite branch of the NASM source code, and testing it against your NASM code/snippets.

Windows (Win32/MinGW) Binaries: http://www.nasm.us/downloads/preproc-rewrite.zip (these binaries will be updated when new preproc-rewrite updates are made, along with a corresponding reply to this thread.)

Source Download via Browser: goto http://repo.or.cz/w/nasm.git/shortlog/refs/heads/preproc-rewrite and click on the top-most tar.gz or zip "snapshot" link, decompress/unarchive accordingly.

Source Download via GIT:

Code: [Select]

git clone git://repo.or.cz/nasm.git preproc-rewrite
cd ./preproc-rewrite
git checkout origin/preproc-rewrite

Compiling on MacOSX/FreeBSD/Linux/et al.: (assumes you are in the preproc-rewrite directory of the downloaded source)

Code: [Select]

autoheader
autoconf
./configure
make

Shell Script for GIT Download + Compiling on MacOSX/FreeBSD/Linux/etc al.:

Code: [Select]

#!/bin/sh
git clone git://repo.or.cz/nasm.git preproc-rewrite
cd ./preproc-rewrite
git checkout origin/preproc-rewrite
autoheader
autoconf
./configure
make

If you are comfortable enough with making this branch of NASM your primary binary on MacOSX/FreeBSD/Linux/et al., follow the above compile instructions with: sudo make install

If there are any questions/comments/corrections/concerns, please reply to this thread.

Thanks. -Keith

[Keith Kanios]

Update: Fixed a bug regarding the emission of labels that immediately precede macro invocations. The new source/binaries are available, and can be found as instructed in the initial post.

Note: The latest x264 source successfully builds and passes checkasm with this preprocessor rewrite of NASM.

[Keith Kanios]

Update: Fixed line number handling.

[Rob Neff]

The %fatal/%error/%warning directives in preproc-rewrite no longer includes in the output the source line of the file/macro it originated from, only the line in the main .asm file where it is detected. Is this new behaviour to be expected now or was this missed?

[Keith Kanios]

I would imagine it should stay the same.

Is this with the "latest" preproc-rewrite, from early this morning?

[Rob Neff]

yes, just downloaded and installed preproc-rewrite.zip, executables timestamp 8/9/2010 1:21am

[Keith Kanios]

Alright, I'll take a look at it.

[Rob Neff]

Found the bug:

The macro name and line number are not displayed when the %fatal/%error/%warning occurs inside an %if/%endif pair.
It IS displayed when outside of the %if/%endif block.
That should help you track it down

[Keith Kanios]

OK. I think I know what needs to be done.

Thanks. -Keith

[Keith Kanios]

Update: Fixed macro-relative line number handling for %warning/%error/%fatal/etc. This update, combined with the general preprocessor redesign, now allows for %warning/%error/%fatal lines within %rep blocks, that are nested within macros, to correctly emit the exact macro-relative line number.

Update: Context-local labels will no longer implicitly search upper/outer contexts in the case of undefined labels. This feature was added in NASM 0.98.03, and is now being removed due to its unintuitive and error-prone design. There will be a deprecation warning for "usage" of this feature in the NASM 2.09. For NASM 2.10 and beyond, relying on this context-local label fallthrough will simply generate an expression syntax error. If you rely on this soon-to-be deprecated feature, please take this time to adjust your code, accordingly.

Note: The Win32 Binaries link has changed: http://www.nasm.us/downloads/preproc-rewrite.zip

[Rob Neff]

Today I happened to retest the NASMX demos using preproc-rewrite and I found what I think is a bug.
At first I thought the bug manifest itself because of multiple dots (ie: MSG.pt.x) in a define.  Then I set out to create a test case for that - the result of which proved nothing (at least not visibly).

The issue appears in relation to macro local variables. Specifically, it appears to be occurring when %assign'ing an equ'ated offset or size (using multiple dotted defnames) to a local variable which is then given to the times macro. You can see this in action in NASMX_AT or NASMX_ENDSTRUC. Try compiling demos 2 and 3 to see this behaviour as these demos both contain the MSG.pt nested structure.

The version of preproc-rewrite I used for testing is

Code: [Select]

nasm -v
NASM version 2.10rc1 compiled on Aug  9 2010
datestamped 8/9/2010 10:48pm

All current NASMX demos compile fine on Nasmv2.09rc6 and execute correctly.
Perhaps something jiggled loose?

[cm]

• %comment/%endcomment: Comment block for long/multi-line comments. A little syntactic sugar, if you will.

But it does have a specific use for which %if 0/%endif can't be used: incomplete examples of conditional assembly. For example, you can put an %if without the corresponding %endif in the %comment/%endcomment block and it will be ignored. All %if-related directives in %if 0/%endif blocks still have to be parsed and must match (i.e. same number of %if and %endif directives). Related to that, you can use unmatched %else/%elif directives in the %comment/%endcomment block without ending the comment (as these would end the %if 0 comment block).

• %final: Causes rest of line after %final to be preprocessed after all other "normal" preprocessing is complete. Multiple %final directives work like the x86 stack, LIFO/FILO.

I don't understand how this works or what the use of this is. Are these lines parsed (again?) in a kind of additional preprocessing pass? Are they just saved (like mmacros) to be executed after anything else in the input? Is the line after %final parsed twice by the preprocessor?

[Keith Kanios]

• %final: Causes rest of line after %final to be preprocessed after all other "normal" preprocessing is complete. Multiple %final directives work like the x86 stack, LIFO/FILO.

I don't understand how this works or what the use of this is. Are these lines parsed (again?) in a kind of additional preprocessing pass? Are they just saved (like mmacros) to be executed after anything else in the input? Is the line after %final parsed twice by the preprocessor?

Saved to be preprocessed last, as if it was the last line among all files... i.e. deferred.

[cm]

Saved to be preprocessed last, as if it was the last line among all files... i.e. deferred.

Okay. I played around with it a bit and found that using %if inside %final does work but is somewhat counter-intuitive because you have to write the %if directives backwards. Testing whether it works, I used the wrong order first and found a problem that occurs anywhere (not just in %final): an %endif directive without a matching %if correctly displays the NASM error about this condition, but segfaults afterwards. This can be reproduced with a source file that just contains one line with "%endif". I'm using the 2010-08-09 Win32 executable.

Am I correct in assuming that the LIFO order of %final processing is intended for setting up checks that will be done at the very end of all processing, and then using later %final directives (which actually will be executed earlier) to make these checks pass?

I found out that %final works for moving assembler lines too, although that's not that useful for most projects. (Considering that there's neither a block %final directive, nor support for nesting %final.) It may be useful for writing equ directives that need to be at the end; they can be written elsewhere this way.

[Keith Kanios]

%final can be used for post-preprocessing cleanups and whatnot, is stronger in conjunction with a macro call, and is arguably most useful when the file utilizing it is being pre-included via NASM's (-p) command line option. It's definitely not a killer feature, but can be useful in certain scenarios.

Perhaps, overall, %final should require a single/multi-line macro parameter?

As for the segfaults, that is a result of my ~90% completion of the rewrite. I still have to wrap-up certain aspects of error checking and memory management.

I'll address Rob's "bug report" as soon as I can, give the rewrite some more time to marinate, and then start wrapping-up the loose ends so that it is production ready.