Looks like it is my turn to do a top 50 list. As much as I am enjoying senecadawg's unique list, I am afraid mine will be a bit more pedestrian than that. I actually put this list together about 6 months ago and I haven't revisited it since. I am sure that if I went through and rebuilt it some things would move and change, but after reviewing my list I am comfortable with the song placements and my feelings.
As for the actual method I used to rank my top 50 DT songs, I went through their complete catalog (including all the unreleased and b-sides I could find) and listened to it several times over the course of a few weeks to refresh my memory and to get passingly familiar with some of the band's rare cuts. Then I sat down over the course of several days and listened again with a more critical ear and rated the songs. Finally, I listened through my top 50(ish) again to do my write-ups and make any final adjustments.
I had considered several methods of song rating and finally decided on what I have called the "Food Network" style of rating. If you ever watch competition shows on the Food Network (like Iron Chef) they use three categories to judge food, Plating, Originality, and Taste. They use those three basic categories to rate the food and then combined those scores into a final score. I liked the idea of rating songs on several different metrics. So I took the idea and broke it down as follows:
"Plating" (1-5): I look at Plating as being the presentation of the song, how appealing it is to listen to and experience. This includes things like song structure, instrumentation, mix, etc. anything that enhances or detracts from the auditory appeal of the song.
"Originality" (1-5): Originality was a harder concept to judge. I tried to judge a song's "originality" against not only music as a whole, but also Dream Theater's catalog in general. Items that make a song original could include time signatures, creative use of samples, etc.
"Taste" (1-10): Unlike a real food or music critic, presentation and originality can only take art so far in my mind. To me Taste, or how satisfied I am after the song is over, is the most important category. Taste to me was far more subjective, but in the end I listen to music I like and that leaves me happy in the end.
Combined, I averaged out these scores to get a rating from 0-10. In the long run there were quite a few songs that had the same scores. Usually, I rated songs that "Tasted" better higher than ones that scored better on the technical aspects. Being a rabid Dream Theater fan my overall scores were probably higher than a non-fan or more critical person would rate them (the average score was 8.18 and the median was 8.5), but its my list and my scores.
I had a pile of songs that clocked in at 8.5 and had to cull those down to start the list, as a teaser I am presenting my "almost-rans" to wet your whistle and will put up the first 5 shortly.
The 51s:This is the Life
Far From Heaven
The Silent Man
Surrounded
Through Her Eyes