I have never used the "Fly Hole" on my boxers. First, it's not the largest hole, and sometimes trying to manuever Vlad the Impaler around to fit neatly through that little hole can be a bit time consuming, and uncomfortable. Second, once the "Fly Hole" has been repeatedly stretched out, it can have a tendency to open when not needed, and "Renfeld and Van Helsing"(the dangly bits) in their constant struggle for domination in the trousers, can then pop out for a peek through said hole, which can be really uncomfortable, and embarassing if you're somewhere where an easy adjustment is impossible to do. Lastly, who has time to be trying to manuever Vlad out of his confines to give the old bladder some relief? Not me.
It is far easier to pull the elastic down and let Vlad spring forth into the fresh air to do his thing. It's easier to shake him off afterward that way too.
I am shocked actually by the number of people on this thread who actually use the "fly hole" for anything other than helping adjust everything if need be in a rush. I've always viewed the "Fly Hole" as the underwear equivalent of an "In Case of Emergency, Break Glass" sign.