I am a straight man, but I've noticed and I've been told that a lot of men are actually attracted to me. I get hit on surprisingly often when I go out or at my job by guys and I don't think much of it, but my colleagues see it and tease me about it. I don't want my surroundings to start thinking I am homosexual.

It is somewhat flattering to get hit on by guys, but it is not something I'm looking for.

What could be the reason of this and how can I change it?