I figured out a way to do it. You need set NASM to output to a "win32" format object file, and then use a program I wrote separately to edit the object file output by NASM. In NASM, use EXTERN with the function name you want to hook (such as SetPixel). Then create a new function with an altered version of that function name (such as HOOK_SetPixel). Then run NASM to assemble this into an object file. Then use the program, and tell it to search in the object file for the string with the altered function name (in this case "HOOK_SetPixel") and rename it to the string name of the function you are trying to hook (in this case "SetPixel"). After the object file has been edited, you use GoLink to link it, which in my example would be:
"golink.exe /entry dllstart /dll myobjectfile.obj gdi32.dll"
After that, just rename your DLL to gdi32.dll and put it in the folder of the program that you expect to call that function in gdi32.dll and it will instead call your hacked version of the function, which will then do something you want it to do with the data passed to it. After your function does what it is supposed to do, your function then will call the real SetPixel function in the real gdi32.dll in the Windows\System32 folder so the program produces the expected result, but at the same time your function can send data to a different program or log it to a file. Of course this may not work on actual system DLLs like gdi32.dll (Windows always makes sure programs use the official DLLs, not alternate versions of the DLLs), but it will work on non-system DLLs (even some graphics DLLs that are often included with Windows like the DirectX DLLs).
This has the potential to let you actually do things like hack games, making it easier to see enemy targets (if you hook DirectX graphics functions and hack them to make enemies always appear bright red).
Not ideal, but until NASM allows you to use EXTERN to reference an external function of a particular name, while calling it a different name in the assembly code itself (an alias), this hacking the object file that is created by NASM is the ONLY way to do it.