Oh, I'll certainly pick up The Destruction..., I'd just like to let the other three completely sink in first, especially as the general opinion is that The Destruction... is their least solid effort (but still very good).
Part of the appeal of The Fall of Math for me is that it sounds like post-rock mixed with Aphex Twin, my favorite songs being Install A Beak..., I Swallowed Hard..., The Fall of Math, This Cat Is A Landmine, and Hole.
One Time For All Time is just barely below it, perhaps equal. I agree that the keyboards add a lot. My favorite songs from that one are Await Rescue, Mean Low Water, 65 Doesn't Understand You and Radio Protector.