You have also to consider his upbriging. Golden child, everyone praising and asskissing him, the heir to the Lannister fortune, an awesome swordfighter... he was born into wealth, praised for it, and fucked up literally and figuratively by her stupid hot sister, for whom he decided to join the Kingsguard to stay close to her.
He served under a mad king (duh, his actual nickname) and resorted to kill him to spare the entire capital from exploding in a mayhem of wildfire, and he's forever branded "kingslayer", "man without honor", with everyone thinking he's the biggest asshole in Westeros. There's only so much of this that he can take before eventually going "ah, screw it" and live up to his reputation, because why bother trying to explain what you did.
He had always that honorable side of him, he was just fucked up for life (in more ways than once) by his sister, he just had to get away from her to start behaving better, thanks also to the positive influence of other people.
If you look back knowing what we know, he basically did his "confession" like he did to Brienne to Catelyn as well, while he was her prisoner; when confronted with breaking his oaths, he said that you swear so many oaths that one contradicted the other, especially if the king would massacre thousands of innocents. It's just that he told in such a smug way that the actual implication was not understood, but he basically told Catelyn what he told Brienne - "Look, the king was a major asshole, I did King's Landing a favor by killing him".
Which, as was said above, made his final remark "I never cared for the innocent people" more infuriating.