I don't have a really good ear, but I'm adept at playing by ear. It is a complicated process, but if you want to go about starting, I'd tell you to do this:
1) Learn about chords and chord structure. I imagine you're talking about rock/metal songs when you say "playing by ear", and even the most complicated rock songs usually come down to a linear list of chords. Some chord progressions are more standard/common than others, and once you're able to pick out the chord structure underneath, things get a whole lot easier. For example, say you're having trouble picking a note out of a harmony, but you know the chord under that measure is an E; at that point, except in unusual cases, you can rule out the notes which definitely don't make any sense, like F, G, and Bb. You can also use your knowledge about the chord to take a stab at what the note could possibly be; since it's an E chord, the natural place to start would be an E, G#, or B (though it could be another tone).
2) Learn about intervals and scale tones. Once you've got this down, you're pretty much set to play all but the most complicated or unusual rock music by ear. Intervals are the distances between notes, and scale tones are the particular notes of the scale in comparison to the root or tonic note (D, for example, is the second scale tone of the C major scale, as it is the second note of that scale starting on C). By playing a lot of music, it's pretty easy to figure out what the 5th scale tone of any scale sounds like, as well as the major and minor 3rds, the major and minor 7ths, the 4th, the diminished 5th, etc. Locating these notes in a harmony or melody can pretty reliably point you in the right direction to transcribing the rest of the phrase.
In short, playing by ear is about recognizing common patterns in popular music, be they rhythmic, melodic, or harmonic. In music where there is no pattern -- say, a speedy piano solo where there is no tonal center to speak of and which utilizes none of the common scales -- I would be lost and unable to replicate it (though there are, of course, people who have perfect pitch and who would have no problem doing so).