Did I Defamate my Boss?

My boss has a reputation around our workplace that she is a user of cocaine, many people have told me stories about her being a user. The other day I watched a fellow employee give her what appeared to be a rolled up baggie. Given her reputation, and his (also being a drug user) I assumed that this was some kind of drug transaction. I told a co worker what I thought I saw, and that comment made its way back to my boss.
The other employee and my boss told me that she had been giving him back a lighter that she had borrowed earlier after denying for several minutes of any contact at all that night. She then told me that I was ruining her reputation and that it was defamation of character and that I would be hearing from her lawyers.
I don't care what they say the were passing to each other, I believe that he hand her a rolled up baggie, it was definately not a lighter, it was much bigger.
Does she have a case against me?
(Sorry I am from Ontario, I just didn't know who else to ask about this, even though this is a USA site)