I think it's just being cautious. Yes, the odds of a knock at the door being benign are much higher than it being someone who's there to hurt you and was stupid enough to knock and give you a warning, but there are definitely areas where any knock on the door must be met with caution. I don't know anybody who would just stop by my house without calling first or having something pre-arranged. Even if they're in the neighborhood and feel like dropping in, they'd still call first to make sure we're home, otherwise they're wasting their time anyway.
I'm not "terrified" by a knock at the door, but if I'm not expecting anyone, I don't just open it up, especially late at night. I look through the window first at the very least. Also, door-to-door people selling shit or wanting to convert me to their religion or whatever, fuck that. I don't even answer the door.