Yea, it could have to do with flow and overall atmosphere of the album.
Exactly. Another perfect example of this was when Steven Wilson wrote the song Drown With Me. He thought it was an awesome song and would be a perfect single. However, after he finished writing everything else that he intended for the In Absentia album, he recognized that Drown With Me wouldn't fit with the overall feel of the rest of the songs, and so he chose not to include it on the album.
Another example: DT planned on writing a dark, heavy album, and they ended up with ToT. However, during the writing sessions, they developed some ideas that became an upbeat song. However, they realized that it wouldn't work on the album, and so they discarded those ideas. In this case, MP remembered this song (dunno how close it was to completion) when they were working on Octavarium, and they ended up using them on the album. The song was I Walk Beside You.
So in both these cases, it's not that the songs were bad or of lesser quality, but rather that they didn't fit with the rest of the songs.
I think RtK and the other non-album tracks from Falling into Infinity are a different animal.
They were a different animal, but not because they intended to write/record a double album. It was only because the label wouldn't allow them in the studio to start recording what became FII and was pushing for more radio-friendly songs, that DT continued writing more.
In the same way, if the guys are "in the zone" and are cranking out great song after song, why stop just because they have an album's worth of material? Why not continue writing to see where it takes you? Maybe you'd come up with some even better songs. Or maybe the feel/vibe of the album might be different. At the very least, you'd have more material to pick and choose from. Had they not continued to write more material while waiting to get the OK from the label to go into the studio, Trial of Tears, Cover My Eyes, Hollow Years, New Millennium and Speak to Me wouldn't exist. How many other awesome songs might have existed if DT didn't stop writing the moment they had enough material to fill a CD during other writing sessions? Obviously we'll never know, but the idea to just write enough to fill a CD instead of following your muse (especially given that they're not writing the album in a full-blown studio) and seeing what more you can come up with seems to be selling yourself short.