Your story about New York reminds me a comparable story i had in Paris long time ago (in twenties also for me ) near la fontaine Saint-Michel .I was photographing people around the fontaine with evening light .A guy coming to me .open his jacket and menacing me with a gun to not photographing him.Surely i said no no it was the fontaine and the guy go elsewhere (A friend say me it maybe because it was an  illegal immigrant)
At Montreal where i live people are more calm about photographersbut you must be discret

Joyeux Noël et bonne année!