NASM - The Netwide Assembler
NASM Forum => Announcements => Topic started by: Keith Kanios on August 06, 2010, 05:36:06 AM
-
This is a preliminary announcement regarding the rewrite/revamp of the NASM preprocessor.
This rewrite addresses issues with implementing recursive macros (%rmacro/%irmacro) as previously attempted, %exitmacro and also makes room for new directives.
New NASM Directives:
- %rmacro: Recursive Macro, can be called arbitrarily deep (really high limit [2^20] set to catch infinite loops.)
- %irmacro: Same as above, except case-insensitive like %imacro.
- %exitmacro: Terminates the current macro invocation, much like %exitrep; can be nested in other preprocessor directives (e.g. %if/%endif) within the macro, with exception of another macro, of course.
- %while/%endwhile/%exitwhile: A fusion between %rep and %if.
- %comment/%endcomment: Comment block for long/multi-line comments. A little syntactic sugar, if you will.
- %final: Causes rest of line after %final to be preprocessed after all other "normal" preprocessing is complete. Multiple %final directives work like the x86 stack, LIFO/FILO.
This rewrite also attempts to address heavy/nested macro invocation efficiency issues.
This rewrite is currently a branch [in GIT (http://repo.or.cz/w/nasm.git)] called preprocessor-rewrite (http://repo.or.cz/w/nasm.git/shortlog/refs/heads/preproc-rewrite), and tentatively labeled as version 2.10rc1. Please note that NASM 2.09rc (official/main/master) is still going and NASM 2.09 should be released fairly soon.
This rewrite has been known to accurately build all NASMX (http://www.asmcommunity.net/projects/nasmx/) demos, along with a variety of other code snippets by various people.
As positive/successful as this rewrite has been thus far, it is still a major change and we would like to catch as many bugs as possible before merging it into the main/master NASM code base.
Please help us test this rewrite further by downloading/compiling the preprocessor-rewrite branch of the NASM source code, and testing it against your NASM code/snippets.
Windows (Win32/MinGW) Binaries: http://www.nasm.us/downloads/preproc-rewrite.zip (http://www.nasm.us/downloads/preproc-rewrite.zip) (these binaries will be updated when new preproc-rewrite updates are made, along with a corresponding reply to this thread.)
Source Download via Browser: goto http://repo.or.cz/w/nasm.git/shortlog/refs/heads/preproc-rewrite (http://repo.or.cz/w/nasm.git/shortlog/refs/heads/preproc-rewrite) and click on the top-most tar.gz or zip "snapshot" link, decompress/unarchive accordingly.
Source Download via GIT:
git clone git://repo.or.cz/nasm.git preproc-rewrite
cd ./preproc-rewrite
git checkout origin/preproc-rewrite
Compiling on MacOSX/FreeBSD/Linux/et al.: (assumes you are in the preproc-rewrite directory of the downloaded source)
autoheader
autoconf
./configure
make
Shell Script for GIT Download + Compiling on MacOSX/FreeBSD/Linux/etc al.:
#!/bin/sh
git clone git://repo.or.cz/nasm.git preproc-rewrite
cd ./preproc-rewrite
git checkout origin/preproc-rewrite
autoheader
autoconf
./configure
make
If you are comfortable enough with making this branch of NASM your primary binary on MacOSX/FreeBSD/Linux/et al., follow the above compile instructions with: sudo make install
If there are any questions/comments/corrections/concerns, please reply to this thread.
Thanks. -Keith
-
Update: Fixed a bug regarding the emission of labels that immediately precede macro invocations. The new source/binaries are available, and can be found as instructed in the initial post.
Note: The latest x264 source successfully builds and passes checkasm with this preprocessor rewrite of NASM.
-
Update: Fixed line number handling.
-
The %fatal/%error/%warning directives in preproc-rewrite no longer includes in the output the source line of the file/macro it originated from, only the line in the main .asm file where it is detected. Is this new behaviour to be expected now or was this missed?
-
I would imagine it should stay the same.
Is this with the "latest" preproc-rewrite, from early this morning?
-
yes, just downloaded and installed preproc-rewrite.zip, executables timestamp 8/9/2010 1:21am
-
Alright, I'll take a look at it.
-
Found the bug:
The macro name and line number are not displayed when the %fatal/%error/%warning occurs inside an %if/%endif pair.
It IS displayed when outside of the %if/%endif block.
That should help you track it down ;)
-
OK. I think I know what needs to be done.
Thanks. -Keith
-
Update: Fixed macro-relative line number handling for %warning/%error/%fatal/etc. This update, combined with the general preprocessor redesign, now allows for %warning/%error/%fatal lines within %rep blocks, that are nested within macros, to correctly emit the exact macro-relative line number.
Update: Context-local labels will no longer implicitly search upper/outer contexts in the case of undefined labels. This feature was added in NASM 0.98.03, and is now being removed due to its unintuitive and error-prone design. There will be a deprecation warning for "usage" of this feature in the NASM 2.09. For NASM 2.10 and beyond, relying on this context-local label fallthrough will simply generate an expression syntax error. If you rely on this soon-to-be deprecated feature, please take this time to adjust your code, accordingly.
Note: The Win32 Binaries link has changed: http://www.nasm.us/downloads/preproc-rewrite.zip (http://www.nasm.us/downloads/preproc-rewrite.zip)
-
Today I happened to retest the NASMX demos using preproc-rewrite and I found what I think is a bug.
At first I thought the bug manifest itself because of multiple dots (ie: MSG.pt.x) in a define. Then I set out to create a test case for that - the result of which proved nothing (at least not visibly).
The issue appears in relation to macro local variables. Specifically, it appears to be occurring when %assign'ing an equ'ated offset or size (using multiple dotted defnames) to a local variable which is then given to the times macro. You can see this in action in NASMX_AT or NASMX_ENDSTRUC. Try compiling demos 2 and 3 to see this behaviour as these demos both contain the MSG.pt nested structure.
The version of preproc-rewrite I used for testing is
nasm -v
NASM version 2.10rc1 compiled on Aug 9 2010
datestamped 8/9/2010 10:48pm
All current NASMX demos compile fine on Nasmv2.09rc6 and execute correctly.
Perhaps something jiggled loose?
-
- %comment/%endcomment: Comment block for long/multi-line comments. A little syntactic sugar, if you will.
But it does have a specific use for which %if 0/%endif can't be used: incomplete examples of conditional assembly. For example, you can put an %if without the corresponding %endif in the %comment/%endcomment block and it will be ignored. All %if-related directives in %if 0/%endif blocks still have to be parsed and must match (i.e. same number of %if and %endif directives). Related to that, you can use unmatched %else/%elif directives in the %comment/%endcomment block without ending the comment (as these would end the %if 0 comment block).
- %final: Causes rest of line after %final to be preprocessed after all other "normal" preprocessing is complete. Multiple %final directives work like the x86 stack, LIFO/FILO.
I don't understand how this works or what the use of this is. Are these lines parsed (again?) in a kind of additional preprocessing pass? Are they just saved (like mmacros) to be executed after anything else in the input? Is the line after %final parsed twice by the preprocessor?
-
- %final: Causes rest of line after %final to be preprocessed after all other "normal" preprocessing is complete. Multiple %final directives work like the x86 stack, LIFO/FILO.
I don't understand how this works or what the use of this is. Are these lines parsed (again?) in a kind of additional preprocessing pass? Are they just saved (like mmacros) to be executed after anything else in the input? Is the line after %final parsed twice by the preprocessor?
Saved to be preprocessed last, as if it was the last line among all files... i.e. deferred.
-
Saved to be preprocessed last, as if it was the last line among all files... i.e. deferred.
Okay. I played around with it a bit and found that using %if inside %final does work but is somewhat counter-intuitive because you have to write the %if directives backwards. Testing whether it works, I used the wrong order first and found a problem that occurs anywhere (not just in %final): an %endif directive without a matching %if correctly displays the NASM error about this condition, but segfaults afterwards. This can be reproduced with a source file that just contains one line with "%endif". I'm using the 2010-08-09 Win32 executable.
Am I correct in assuming that the LIFO order of %final processing is intended for setting up checks that will be done at the very end of all processing, and then using later %final directives (which actually will be executed earlier) to make these checks pass?
I found out that %final works for moving assembler lines too, although that's not that useful for most projects. (Considering that there's neither a block %final directive, nor support for nesting %final.) It may be useful for writing equ directives that need to be at the end; they can be written elsewhere this way.
-
%final can be used for post-preprocessing cleanups and whatnot, is stronger in conjunction with a macro call, and is arguably most useful when the file utilizing it is being pre-included via NASM's (-p) command line option. It's definitely not a killer feature, but can be useful in certain scenarios.
Perhaps, overall, %final should require a single/multi-line macro parameter?
As for the segfaults, that is a result of my ~90% completion of the rewrite. I still have to wrap-up certain aspects of error checking and memory management.
I'll address Rob's "bug report" as soon as I can, give the rewrite some more time to marinate, and then start wrapping-up the loose ends so that it is production ready.
-
I'll address Rob's "bug report" as soon as I can, give the rewrite some more time to marinate, and then start wrapping-up the loose ends so that it is production ready.
Thanks, Keith. I did not feel the need to add to bug tracker as you were still coding, but rather simply providing a heads up. ;)
-
Thanks, Keith. I did not feel the need to add to bug tracker as you were still coding, but rather simply providing a heads up. ;)
Yep, that's why I quoted "bug report" as it wasn't official, but still a high priority.
Thanks. -Keith
-
Today I happened to retest the NASMX demos using preproc-rewrite and I found what I think is a bug.
At first I thought the bug manifest itself because of multiple dots (ie: MSG.pt.x) in a define. Then I set out to create a test case for that - the result of which proved nothing (at least not visibly).
The issue appears in relation to macro local variables. Specifically, it appears to be occurring when %assign'ing an equ'ated offset or size (using multiple dotted defnames) to a local variable which is then given to the times macro. You can see this in action in NASMX_AT or NASMX_ENDSTRUC. Try compiling demos 2 and 3 to see this behaviour as these demos both contain the MSG.pt nested structure.
The version of preproc-rewrite I used for testing is
nasm -v
NASM version 2.10rc1 compiled on Aug 9 2010
datestamped 8/9/2010 10:48pm
All current NASMX demos compile fine on Nasmv2.09rc6 and execute correctly.
Perhaps something jiggled loose?
Rob, do you have an updated or more concise demonstration of the issue... specifically one that is compliant with local-context fall-through deprecation?
-
Testing whether it works, I used the wrong order first and found a problem that occurs anywhere (not just in %final): an %endif directive without a matching %if correctly displays the NASM error about this condition, but segfaults afterwards.
I've pushed a change that fixes the segfault issue with %endif.
-
Rob, do you have an updated or more concise demonstration of the issue... specifically one that is compliant with local-context fall-through deprecation?
Unfortunately the current state is not yet stabilized enough and quite a few changes have occurred since the last report that I'm currently unable to provide what you request. However, I have already fixed the context-fall-through issue and will be revisiting the local macro variable issue again soon - testing against Nasm v2.10rc's - so look to hear back from me ;D