Well... this is a fairly complicated question, Amumu!
As you know, the 16-bit version of AoA is "old", and for dos. It might be like trying to learn automotive repair using a Model A repair manual. The pistons still go up and down, but practically everything else has changed. Dos isn't going to be making that "comeback" that some of us were hoping for. As an OS, dos is pretty lame. This has an advantage, in that dos allows direct access to the hardware, which no protected-mode OS will let you do. Dos won't "help" you much, but it won't get in your way, either. To maximize this advantage, I'd install "real dos" on a separate partition, rather than count on an emulator to get it right... but maybe I'm just suspicious of emulators. I have not been able to install dosbox on my systen (a long story). You mention on SO that you tried dosbox but it "didn't work". In what way did it not work? (I can't help you, but perhaps others can)
It is possible to use Nasm in conjunction with AoA, but (16-bit) AoA targets Masm/Tasm syntax. Nasm's syntax is different. It is possible to overcome these differences... not too difficult... but you probably don't need any "extra" complications at this stage. I think I'd suggest using the "intended" assembler. If you don't like MS (some people don't, I'm told
), Japheth's Jwasm may work...
The 16-bit AoA is useless for Linux. The 32-bit AoA will work on Linux, but it uses HLA (the "high level assembler", an invention of Randy's) which uses a syntax all its own - unlike any other syntax you'll encounter. Last I knew, the Windows version will output Nasm syntax, the Linux version will not. Not that useful anyway - HLA won't "use" Nasm syntax.
Getting back to the 16-bit version, I think you'll find that the UCR Standard Library will come pre-assembled and linked. If not, it shouldn't be difficult... using the intended tools. Nasm won't do it. (I have some of the routines translated to Nasm syntax, but not all... I "might" still be able to find 'em...). As I recall, there's no difficulty linking Nasm-generated code ("-f obj") against the UCR library, if that's what you want to do. If a problem crops up, we might be able to help...
It isn't clear whether you're running Windows and want to run Linux in a VM, or if you're running Linux and want to run dos in a VM. Either should work, but might introduce "extra" complications that you don't need. The native code of the machine is the same in any case. Even a 64-bit machine will run 16-bit code (AFAIK), but a 64-bit OS will prevent it... except in an emulator. I'd install "real dos" (OpenDos seems pretty good) on a separate partition if you want "real" access to "real" hardware. That may be due to my unfamiliarity with emulators...
If you want to get started in Linux, we can discuss that more - I'll leave it for a separate answer. I think that's a fine idea! But the 16-bit AoA is not relevant.
I'm watching SO to see if you get any good answers. I'm betting that they'll close your question. Apparently they don't consider "how should I learn this?" to be a "real question". Maybe they'll fool me. We're willing to discuss it here, even if the answer is "don't use Nasm". (we'll try to avoid that, of course)
Best,
Frank