But I know very well what an American accent sounds like, and I know very well what an Australian accent sounds like.
And that's why you don't hear it. Your brain "corrects" what it hears automatically. To us, "day" and "die" do in fact sound very similar. But you know that they're completely different words and thus hear them differently.
Also, there's no such thing as an "American accent." This country is large enough that there are countless regional accents. Folks from one part of the country always think folks from another part of the country speak differently. As I understand it, the closest thing to a "neutral" American accent is the Midwest dialect. Very bland, practically boring, without many distinctive vowel affectations which characterize Southern, or New England, or even West Coast.