When something like that happens to me, I try to come up with a different section that sounds right to my ears. If I really like the section that gets replaced because it didn't fit, I construct a new song around it.
I would certainly not stick to it if I felt it does not flow. No matter what the casual listener says, if I myself do not think it works, I'll never be satisfied with the outcome and eventually the song will end up not getting recorded / kicked out of the live set.
I had the same problem with a song that I'm currently finishing. I had to come up with about 15 different approaches until I was satisfied, but that way I made some really cool stuff that I can use in the future
But that's just the way I work. You'll have to find out what works best for you. Good luck