I guess I am more of a bigger picture listener. I hear the hooks, the melody, the riffs, and how it all comes together. I hear the song as a whole.
Although I do understand music theory and time signatures, I don't notice odd sigs right away.
Here's an analogy; if I eat a cookie, I taste the cookie. Not the flour, butter, eggs, and sugar individually, even though I know they're in there.
Then again, I do hear the individual instruments, I know there is guitar, bass, drums, keys, and voice. If those are analogous to the basic ingredients of the cookie, then maybe the time signatures are the chocolate chips?