Defamation / harrassment Case potential?

I work in a large sales organization and I am at the top of my field. We reguraly attend semiannual meetings. One employee who is essentially on the cultural level of nascar and pro wrestling inferred directly that I am gay and repeated the accusation. I am completely not and have had many girlfriends and have never even remotely engaged in such activity. He's basis is because I dress well and I workout. Rather than simply 'beat him up'lol what is the best way to handle this idiot. Is this defamation per se of character as from my understanding he goes out of his way to tell others behind my back also and has directly accused me of such which I simply responded that I've forgotten more about women than he'll ever know and have slept with more in a weekend then he'll ever hand dream to. I have witnesses etc. Thanks for any feedback.