Are these actions by former supervisor harassment?

I work with a program which helps seniors (I am 62) and people with barriers to employment (I have a felony conviction for a white collar crime and am on disability) gain skills that will lead to permanent employment. Salaries are paid by this program, not by the business we are placed at. I was placed at a privately run non-profit agency (the Executive Director was aware of my background and told me she had shared that information with very few others). I was provided a job description which involved learning computer skills. I did routine office work and took on the task of helping the non-profit develop a presence on social networks; however, on an almost daily basis I was asked to perform tasks NOT in the job description (either on my own or with a volunteer) I cleaned the bathrooms and kitchen, organized storerooms, prepared lunch for board of directors meetings, set up the foot bath spa for clients and cleaned the room afterwards, etc.). I did not complain or even mention this to my program coordinator when I had a job review because I was grateful to be working. The Executive Director allowed me to extend my assignment there from 6 months to 17 months. At the first meeting with a new program coordinator it was stressed that we should only be working in our job descriptions because we could put the program and our host agencies at risk if we were injured on the job, etc. I was asked if I was working within my job description and improving my skills, I shared some of the things I did. The program coordinator decided to move me to another agency. After my former supervisor met with the program coordinator she was extremely upset and would not talk to me. Since I left her agency she has called my new supervisor and asked if she knew about my past (of course she does), sent me a letter telling me I was banned from coming into her agency because she had heard rumors that I had alleged "elder abuse" at her agency,(untrue), and that she was considering filing a "defamation of character" suit or other legal action against me because I am responsible for the program not assigning her any other workers. A friend who still works there told me that the director had announced at staff meeting that given my past, she was glad I was gone. To say I am upset is an understatement. I live in a small community and have worked hard to re-establish myself here. The program coordinator tells me not to worry, but I am having panic attacks. Is there anything I can, or should do? Thank you for your time.