The DLL should simply have one function (which would use the STDCALL calling conventions), and this function would have one parameter. That parameter would be a pointer to a null terminated ANSI string that contained the ASM code to assemble. The function's return value would be a pointer to a structure with 2 fields. The first field would be the number of bytes of compiled code. The second field would be the pointer to the first byte of compiled code. All pointers, and sizes would be 32bit ints (not 64bit ints), to make sure that it is compatible with both 32bit and 64bit versions of Windows. This NASM DLL would only compile 32bit code (not 16bit or 64bit code).
While it would be lacking a lot of features of the full NASM application, this DLL version would make it very easy for a person to write a program in a high level language such as C++ or VB6, and include "thunks" of machine code, as ASM code stored in string consts at designtime. At runtime these text strings of ASM code would then be compiled by this DLL when the DLL file was called by the program. Then one just needs to use Windows API function like DispCallFunc or CallWindowProc to get the program to start executing this raw machine code thunk which was the compiled output of the NASM DLL function.