I tend to agree. Best response would probably just have been a polite, "Nice shirt." Or if he wanted to joke, something along the lines of, "Hey, nice shirt. I hear those guys used to be pretty good." Actually, if it was clear he was joking, the latter would actually be pretty funny.