We can clean up the code in a few ways. Using EQU or %define to create symbols can clarify your code without adding any additional overhead, so use them as much as possible. Second, as Frank stated, you should clean up the stack after each C function call. Another thing to note (and I think this *might* be what's confusing you) is that the parameters to C functions are pushed onto the stack in reverse order. In other words:
push DWORD [ebp-X]
push DWORD [ebp-Y]
push msg2
call printf
add esp, (3 * DWORD_size)
Is equal to:
printf (msg2, Y, X);
Below is a cleaned up version of your program, I hope some of the things I said here have helped you.
DWORD_size equ 4
section .data
msg: db "The value of the variable is: %d",10,0
msg2: db "%d y is greater than %d x",10,0
msg1: db "%d x is greater than %d y",10,0
section .text
global main
extern printf
main:
X equ 4 ; First DWORD Variable
Y equ 8 ; Second DWORD Variable
push ebp
mov ebp,esp
sub esp,(2 * DWORD_size) ; Allocate 2 DWORDS
mov DWORD [ebp-X],0x10
mov eax,DWORD[ebp-X]
mov DWORD [ebp-Y],0x11
mov ebx,DWORD[ebp-Y]
cmp eax, ebx
jg .xGy
.yGx: ; for y greater than x
push DWORD [ebp-X]
push DWORD [ebp-Y]
push msg2
call printf
add esp, (3 * DWORD_size)
jmp .done
.xGy: ; x greater than y
push DWORD [ebp-Y]
push DWORD [ebp-X]
push msg1
call printf
add esp, (3 * DWORD_size)
jmp .done
.done:
mov esp,ebp
pop ebp
ret