Whenever I have an evening of chilling out with friends, I have to really think about what music to put on. 90% of the music that I listen to isn't fit to being good background/party music.
This is the biggest problem I've encountered. I've played some of my favorite music for friends, family, co-workers (e.g. Warmen, Michael Pinnella, Kamelot, Symphony X, Children of Bodom, Yngwie Malmsteen, Tony MacAlpine), and each time I got strange looks and was told it sounded like video game music.
There's no way this music would go over well at parties. In college, I had to get used to hearing rap/hip hop at parties.
It's also harder to relate to people and have a conversation about music because less people share in common my taste in music. I went from strictly listening to metal, to also listening to indie, electronic, folk, and hip-hop so I can be/feel more sociable in this way. I have to say that doing this had its rewards 'cause I discovered a lot more great music. It did help socially at times, too, I suppose.