I think it starts to become pretty ridiculous when you start spending more time hunting down mods, installing them, and getting them to work with other mods than you do playing the game.
I say this as someone who did this very thing with Oblivion ALL THE TIME.
"Huh, I really wanna play some Oblivion... oh it's not installed? Damn."
*hunts down mods and patches, installs, 3 hours pass*
"Okay, everything is installed... WTF? Where's my textures? Why is everything a barrel? Goddamnit."
*1 hour of fixing, reinstalling, uninstalling*
"Okay, time to play..."
*Get through intro and start wandering around, takes about 45 minutes*
"Eh, I'll get back to this tomorrow..."
*Never gets back to it*