defamation?

I worked for a temp service several years ago (2001) and was removed from an assignment not knowing the specific reason. I believe now that it was because a coworker told management that I was having an affair with my married supervisor. I am also married. Although we liked each other we never had an affair or even spoke more than a few words to each other. At any rate, I moved on and returned to school and went about applying for jobs but was never hired by anyone until recently at a place that did not check my references. I have received many strange looks and comments at interviews and in responses from places where I have applied with no success which leads me to believe that the temp service has made this rumor public knowledge to prospective employers, either through direct references or through one of the background checking services. I wrote them a letter requesting them to stop but I rather don't think they will. My reputation is damaged because of this and the chances of getting a better job are essentially nil. I don't know exactly what is being said but it is something that is casting me in a bad light and keeping me from obtaining a job where references are checked. It also has gained me disfavor at the university and community college I attended after I applied for jobs at both of those places. Is there anything I can do?