About Scenes vs ToT.
I think you guys are mixing up "popular" with "critically acclaimed" though both terms are not always mutually exclusive, of course.
St. Anger is one of Metallica's most popular albums, because it's such an objectively bad album (no offense to anyone who enjoys it)
Everyone knows about St. Anger, along with Master of Puppets and TBA. Casual music listeners probably don't know Load/Reload or their last 2 albums as well as those albums.
ToT was really popular because it was something a lot of fans disliked, and the move to full-metal album was very controversial.
Looking at the sales numbers again, I see Scenes sold less in the US than FII as well, along with SDoIT and ToT.
ToT may have turned a lot of fans off as well, as Octavarium sold much less in the US compared to the previous albums, but later album sales could also be affected by illegal downloading, and eventually streaming. Again, this is why I wish we could look at overall sales around the world.
"objectively bad"
Yeah, no. You can't quantify music quality in an objective way. You can have music that appeals to a particular subculture (or offends them) and with some practice, you can get consistently positive results. This is why a lot of supposedly 'jump the shark' moments tend to be instances in which the band simply changes style towards something the subculture is not accustomed to. I've heard people make pretty compelling arguments on paper for why St Anger could be seen as great stuff. I don't agree with them, but am glad that they found a way to connect with it.
Also, I don't know why we're looking at sales when it's kind of an antiquated way of looking at music popularity nowadays. This isn't two decades ago, this is today. Anyone getting into DT or revisiting them are not particularly likely to be straight up buying the album, but streaming it or even going to YouTube.
I think you're combining two arguments I made that had little to do with each other. I admitted that there are some jazzy moments on ADTOE earlier, and duel solos still exist in the Mangini era. Octavarium was largely a reaction to ToT, lighter songs, lighter production, etc.. You're taking the "jazz influence" in DT's music too deep here. I merely said there was some jazz influence in DT's music, but there was less of it with each successive album, or at least the melting pot of their music was more homogenized (is that the right word?) by Octavarium.
Alright, so we've got (and only for the MM era)... classic prog metal (On the Backs of Angels, At Wit's End, so on), elements of chill jazz rock (This is the Life's second verse really brings this out), 80s style neo-prog (Barstool Warrior), melodic thrash (Fall Into the Light, Behind the Veil), Tchaikovsky style orchestral stuff (The Embracing Circle), jazz fusion freakouts (Surrender to Reason's solo, S2N, arguably the intro of A Life Left Behind, Untethered Angel's mid-section arguably wouldn't be that out of place on an Al Di Meola album), musical theatre (literally The Astonishing), chunky alt-metal (BMUBMD, Paralyzed), spaghetti western style acoustics (Fall Into the Light again), shades of modern prog metal / djent (Pale Blue Dot, arguably The Enemy Inside, Illumination Theory and Enigma Machine), flighty prog rock (Breaking All Illusions), modernised hard rock (Viper King), almost hymn / churchy material or military march stuff (Brother Can You Hear Me, idk what to call it at this moment precisely). folksy tunes (Hymn of a Thousand Voices), arena rock (Our New World, The Looking Glass), elements of textural film score type stuff in the vein of someone like Hanz Zimmer (Pale Blue Dot's spacey atmospherics and general harmonic style, False Awakening Suite), piano balladry (Far From Heaven), experimental electronic elements (NOMAC tracks, though a tamer integration in Outcry and BMUBMD with the drum loops), stepping into pop territory at times (Beneath the Surface, Out of Reach, Chosen also wouldn't look that out of place next to something like Let it Go as far as pop/theatre material is concerned), even psychedelia if you squint hard enough (Room 137's choruses and maybe pre-choruses have elements of this). Sure sounds homogenous, doesn't it?
Awake wasn't based on metal? Scenes? They are both easily more metal than ADTOE and The Astonishing. Maybe even Distance Over Time.
WDaDU too, which is, other than Status Seeker, pretty much entirely 80s style prog metal.