Whew. THIS Wednesday? Not a chance.
What OS? I suppose it doesn't matter too much, you'll need similar routines... Simple user I/O... file create, read, write, seek... some sort of "search" routine... some sort of "sort" routine... as a bare minimum. Keep it simple. How much of this can you already do? What's the most advanced asm program you've written? Can you do it in some other language?
I suppose the first thing to do, if you haven't, is to define the contents of your files. Some sort of user-number, first name, last name... maybe an address so you can track 'em down if they don't bring the book back. Title, author (first/last?), who's got it (or is it "on the shelf"?). I guess Dewey Decimal is obsolete, maybe ISBN number? I'd definitely go with fixed record length. It's a PITA to change 'em after the files are created, so try to make 'em long enough to begin with.
Maybe the very first thing to do is to try to get an extension on this. I don't think you've got enough time to do a decent job. Show us what you've got and we'll try to help you when you get stuck. Nobody's going to do it for you!
Best,
Frank