Ah! Okay. I still may not quite understand what you want to do, but I'm closin' in on it... maybe...
If you use a Windows-format .dll, you'll be limited to those design decisions (obviously) - but I'm not sure exactly what those limitations are. You might want to adopt that format in spite of it, since it's quite popular.
Linux has the "shared object" format. I think it does essentially the same thing as a Windows .dll - but not compatible, AFAIK. I don't know details of either format.
Nasm's own format would be "rdoff" - I guess that's what you mean. Although it's got "dynamic" in the name, I'm not sure it has "dynamic libraries" in the sense I understand it. The documentation indicates it's suitable for static or dynamic linking, "depending on the complexity of the loader". Maybe the supplied loader is "complex" enough - or could be enhanced. I don't see the "-shared" option to rdfld, so I don't know how you'd do it. The only guys that know much about rdoff are Julian Hall and Yuri Zaparogets. Should be able to find (current???) e-mails for them in the docs for rdoff. Yuri's RADIOS site (SF) bears a notice that it's no longer under active development. (another one bytes the dust)
If you want to encapsulate your audio system into a dynamic library, linkable from "any OS", that may be tough. Probably could be done with enough conditional assembly, but if it's "nearly all" conditional, there isn't much point. If what you need is to be able to call a dll to handle OS-specific stuff from your audio system, the C standard library may be your best bet - msvcrt.dll for Windows, and libc.so for Linux. Not sure about the other OSen. You should be able to write fairly "portable" code that way.
I suspect you're looking for more than that. The correct answer to your question (from me) is probably "I don't know". I'm happy to discuss it further - if *one* of us learns something, I don't care which one. "Both" would be nice. :)
Best,
Frank