Hi Josh,
Okay - you can get answers a lot of places. Actually "or byte ptr [eax], [upperMask]" had a couple of problems... Besides "[mem], [mem]" instructions not existing - it applies to more than just "or" (we have only one data bus) - Nasm doesn't use the "ptr" keyword... just "or byte [eax], ...". The "..." can't be memory - could be a register like bl (in which case you wouldn't need to tell Nasm "byte" - the register tells Nasm the size), or could be an "immediate" - a plain number (in which case Nasm *can't* determine the size, you have to tell it).
If "upperMask" is a variable...
upperMask db 0x40
Then your solution will work (assemble, at least). If you defined "upperMask" as a "constant"...
upperMask equ 0x40
then you could do "or byte [eax], upperMask"... (this is what "Arargh" was talking about)
Now... perhaps my memory fails me, but 0x40 is *not* the value I recall for "upperMask". I remember 0x20... If your routine is giving "weird" results, try that.
Best,
Frank