Strong accusations, shifting in working relationship - is this defamation or a revenge situation?

Hello. I have worked off and on for a temp agency after wrapping up a masters degree and moving cities. I do many different things, but occasionally work with labor temp agency when I am not trying to build my own business. The temp agency is notorious for not paying people on time, and occasionally one needs to call to ensure they will be paid several days after the payment is expected to come. After not being paid for a week, I called back and after a brief line of questioning the agency staffer "hung up on me" - I called back saying that was unwarranted and he hung up on me again. Later that night I called corporate offices and filed a formal complaint - which is supposed to be protected. The affected individual later texted me saying I only gave one side of the story (I never responded), however, he was then transferred to another location and I happily did not have to deal with him again. I took a job out of state, and then nearly a year later came back in town for family and resumed sporadic work with the agency. I always work hard and take pride in my work, even when doing menial tasks. After being assigned to work one particular job, I realized the supervisor at the location changed the start time I wrote down on my sheet to a later time - when I was set to end my work for the day and sign out I informed the closing supervisor that I arrived earlier than what was written and the closing supervisor said "how about 7 hours - does that work?" "I worked approximately 5 hours". I said that is generous but just disputing the start time, but eventually ceded and said thank you. Thinking that was that. The following week I realized I was paid about half of the 7 hours and nicely inquired with the office about the reasons why (again, they often mistake hours paid, so they give me a reason to call and check up). The nice office staff member said they would email and find out and call back. A week later the office called to see if I would be returning to another site (but now it was the individual I had not talked to in a year since the hanging up / complaint incident) I remained professional, told him I could not return to the new site for another day because of my other work, but then informed him I had not heard back about the earlier hours work at the job a week prior. He said he would go through the notes and get back to me. The individual called me back 20 minutes later accusing me of leaving the job early and calling about hours not worked which was "fraud" and that would not be tolerated, etc. I called back and clarified that the supervisor wrote down my hours and I was simply asking for clarification and that I did not leave early. He said you're not suspended etc - but I know he took satisfaction out of the situation. Since that time, I've taken 3 or 4 jobs, and the company has gotten progressively condescending and less workable. Telling me they called the employer for the exact time I will be at the location, texting me additionally and calling me additionally to make sure I am in route. I work hard and have never had a negative employment situation, and now with a company that seems to pray on individuals not making much money, I feel like my reputation is being twisted and that maybe this individual is seeking revenge for my previous complaint being filed. I will never list this company on my resume, but I still don't want to be bullied - made out to be someone I am not - and unfortunately still use these guys for the short term. Is there any thing I can do / you recommend doing in this situation? Isn't accusing someone of "Fraud" without solid proof and overreach and damaging to the employee? Thank you for your time.