You should be able to hear something of a difference between 128kbps and lossless, but you shouldn't particularly be able to hear any difference between 320kbps and lossless, as 320kbps is way faster than the average human ear can process information.
Lossless is more important for storage than for listening. As you don't lose any information whatsoever, your entire storage of CDs could be totally obliterated and you'd still have all the information you paid for, so that's a grand thing about lossless. It's also handy for ripping CDs. You could burn a 320kbps MP3 to a disc for a friend, and then when they ripped it to their hard drive it'd cascade, because the dropped information won't necessarily match up.
So yeah, it's nifty if you've got the space for it, but don't let anyone tell you you're missing out by listening to your MP3s, because you really, really aren't. Your ears are fine. Normal, in fact. Possibly slightly above average!
You know how films are just an endless flipbook of teeny, tiny snapshots? Music's the same. And just as you can't see empty frames between images at the cinema, you can't hear the empty frames between the noises of a high-bitrate MP3. A pigeon's eye works faster than a human one - they wouldn't understand a film at all. But the human eye is deceived by film, and in exactly the same way, the ear is deceived by a 320kbps MP3.
If you can't tell the difference, it's not 'cause your ears are untrained, it's because you can not tell the difference.