I'm a fairly big fan, but weirdly their more recent catalogue is what I know and love. I first heard them back in '09, with The Wildhearts Must Be Destroyed. It is still my favourite album of theirs, and is just chock full of hits!
I believe a lot of fans rate that as their worst (I certainly do), though it depends on what they make of Endless Nameless, which I love but not everyone does, for obvious reasons!
I then got a greatest hits album and a three disc collection spanning their career at that point. I bought their live album The Wildhearts Strike Back, and have then followed every album they've released since then. So yea, I've never listened to their early albums, but I do know all the hits from them.
If you're sticking to hits and album releases then you're missing a big part of the joy of The Wildhearts, which is some of their best songs are B-sides and for a time some of their best ever material (Fishing For Luckies) was languishing on a fan-club only release! I'd cheerfully rate Chutzpah! Jnr! as comfortably better than Chutzpah! though Chutzpah! still has a few stormers.
Also, the thing with the Wildhearts is that most of the albums are just about all Ginger, so you're really missing out if you don't investigate his solo/side project endeavours, albeit that material can get quite eclectic and be some way off the "Beatles meets Metallica"* of The Wildhearts.
Here's some lesser known non-Wildhearts gems I'd recommend:
Unlucky In Love (Clam Abuse)
Not Bitter, Just A Little Disappointed (B-side from the singles club solo release Cars & Vaginas)
Walk Like A Motherf*cker (Silver Ginger 5 "demo")
Drinking In The Daytime (from Valor Del Corazon)
The Pendine Incident (from month 5 of the GASS project / The Year Of The Fanclub)
Toxins & Tea (from month 8 of the GASS project / The Year Of The Fanclub)
Don't Lose Your Tail Girl (from month 12 of the GASS project / The Year Of The Fanclub)
Zap (from Headzapoppin)
Why Aye (Oh You) (from The Pessimist's Companion)
I Need You (from Albion... a chunk of the fanbase doesn't agree with me as they didn't vote that onto the commerical release despite it being my favourite song off the album!**)
...and just about anything off both Hey! Hello! albums.
* Though I've always preferred "Cheap Trick meets Sepultura".
** And one of the very, very few songs that's ever brought a tear to my eyes despite it sounding superficially cheery.