My previous employer has called my part time employer making threats about contacting clients

I am a Managing Esthetician. I began working for a dentist who wanted to offer skin care in his practise. I was hired around the 10th of November, 2009. I had a following that I brought with me. I have never signed a non compete nor have I ever even signed an employee handbook. The dentist didn't have one at the time. One year later, he lets me go and tells me that he didn't feel it was profitable enough(skin care) for him and that he was loosing moneys. I left and contacted my clients and let them know where I will be working. He got wind of this and has made calls to my part time employer with threats of a law suit and had his attorney write me a threatening letter about Trade Secrets. I never knew anything about trade secrets or anything else. The only papper work I ever filled out was a W-2 form to receive my pay. Since I never signed any kind of legal document, can he stop me from contacting clients I serviced? Most I brought with me and even the ones I brought, referred others. He did make a profit off of the revenue, so I do not understand why he even made a comment about loosing money. Also, I work 30 miles from his practice and I work for plastic surgeons and not a dentist. Also, he has since hired another esthetician, however, she doesn't have my skill set and he is paying her less money. Does he hve a case against me? His attorney wants me to sign papers that I will hand over client information and swear in an oath that I will not contact them. Why would I do that when that was not wht was agreed to when started working there