Let's put it this way:
Am, G, F are chords built on the notes of the A natural minor scale (i.e. the C Major Scale): A, B, C, D, E, F, G
When the progression goes to E, it's borrowing a dominant chord (parallel sostitution) from the A harmonic minor scale, which features A, B, C, D, E, F, G#
So, if you are constantly using the A harmonic minor scale, G# is a chord tone only on E, hence a reliable note to lean on melodically, while on the other chords it tends to sound "better" as a quick passing tone.