Very good point. It's taken me a lot longer to REALLY get jazz than just about anything else. Maybe it's gotten a little easier to get into different things after I have more of a base to work from, but it still seems to take almost the same amount of work.
That's really the trick to getting into jazz (and any genre, I guess), building up a base to work from.
When I first started getting into rock seriously in my early teens, I'd already heard a fair bit, because the fact is, rock is widely popular, and it's everywhere. Even buying my first albums, I could identify Oasis, the Smashing Pumpkins, Nirvana, Pearl Jam, the Chilis, even older bands like the Beatles, Guns N Roses, Led Zeppelin etc. Just from general knowledge, I had a bit of an idea which bands were older than which, what a rock n roll band sounded like, what rock bands from the late 60s/early 70s, the 80s, and the 90s sounded like. How punk and metal sounded different from rock. I think most people would already know a little bit to navigate themselves, and they could identify differences in styles and eras instinctively fairly well.
With jazz, it's almost never gonna be the case. Of course, everyone's heard it, but no-one has any real pre-knowledge, and you can't really pick artists and styles well by ear, unless you have music training, maybe; it's all just spicy, instrumental music played with brass instruments that has a lot of soloing. Beyond that, the extent of your ability to identify different styles is things like separating pre-50s jazz from 50s-and-later jazz by the production, big-band from smaller groups, and quieter, slower jazz from faster, louder jazz. None of which is all that much to go on.
Which is why it all sounds samey, until you spend some time with it, really delve, build up a body of albums, learn about their context, develop an ear for the differences in the subgenres, and get a sense of orientation within the genre as a whole. It's essentially building the foundation you already have for rock from the beginning.
I pull out Brubeck's Take Five about once a year and keep hoping I get what other people seem to get out of it. I realize the distinction of it being landmark album and I respect that, but I just haven't been able to get much of anything else out of it consistantly.
Well, just like any genre, at the end of the day, it's all about your own taste. I wouldn't say it's my one of favourite jazz albums, although I do like it.