Even if you're allowed to use a certain symbol, you can't just ignore a meaning that the majority of people associate with it. Fly the flag if you want, but be aware that many, many people interpret it and those who fly it a certain way.
You can't just ignore the majority view because you disagree with it. I'm not going to walk around with an image of a swastika, and then feign ignorance when people get offended. Sure, the swastika was a symbol of Hinduism, and many other things, before it gained its more notorious reputation, but it DOES carry that negative interpretation now for most people. Even if my intent was entirely based around one of the innocent uses of the symbol, I know damn well what it means to most people.
A person may fly the confederate flag solely because they're proud of living in the south, without intending anything racist. That doesn't change the fact that most people do consider it a racist symbol, and it's well known that it has that reputation. Yes, you have the freedom to display it, but it isn't unreasonable for people to be offended if you do.