This is what some people who are afraid of cats complain about :-
Get Fuzzy comic strip
The real reason in all likelihood is that cats don't like it when people make direct eye contact with them. As a result, the people who DO like cats and are trying to get their attention are the ones the cats view as most threatening. On the other hand, people who are frightened and avoiding eye contact are seen as 'safer'.