I've thought about that before, and it does seem silly at times to go to the transporter room when they don't need to.
There are reasons for having the designated transporter room though. First of all, it still contains all of the specific components of the transporter system itself, and they have the specialized transporter controls and someone qualified to operate the transporter system, in cases of difficult or unknown conditions etc.
As for why people need to go to the transporter room to transport off the ship, it's probably easier because it's the area designed for the transporter beam, so there's no messing around with locating people and objects and making sure they're ready and not moving, and you're not transporting anything you're not supposed to, or making any mistakes where you transport part of something. It's easier to have everyone and everything there in front of you where you can visually check they're in place and verbally tell people what to do.
And a site to site transport might use more power, since you have to transport them from one distant location to the transporter buffer, then transport them to another distant location. Having them right over the transporter when de-materializing may use less power.
And also, formality, procedure, and habit.
That said, I'm sure there are numerous instances where they go to the transporter room when it would be more appropriate to do a site to site transport.