On the early Rush albums, Geddy hardly does what would be conventionally recognized as singing. I mean, it is singing, but it's really squeaky and heavy on the wailing. Later on, he stops doing that quite so much, but his voice is still sort of throaty and squeaky.
Even if one doesn't mind this (I do mind it in the early stuff, not so much later), I don't see why it should be such a mystery that other people do mind? He doesn't have the sort of tone that people typically love. He's not a particularly talented singer in the easily measurable aspects like range. I have a hard time imagining any band seeking him out just for his singing, leaving aside his ability as a bass player or composer. Rush is, like, a prototypical case of a band deciding not to hire a dedicated singer and having one of the members whose voice they find suitable [edit: meant to say: "serviceable"] take on the lead vocals.