Here in Germany, caching in the woods becomes more and more of a problem. Land owners regularly complain about the downsides (disturbing the animals, destroying nature). The problem is that geocaching does not have some kind of a federation that could react in the press. And we cachers don't want any more attention, tbh... But my guess is that facing it out won't be successful.
Anyway, what also causes the negative attitude of some people is the growing number of cachers - especially those how use their smartphone and "only try" it out. Often, these are the unexperienced people who dig up whole areas in the woods and leave unnecessary trails.