Can I be sued for taking clients from a prior employer?

I have recently left my job, which was in the accounting field. A non-competition agreement was never signed between myself and my employer. I have now started up my own business. My previous employer is threatening to sue me if I talk to any of their clients for service. In my position with this company, they requested that I have all employees sign an employment agreement which was comparable to a non-compete agreement. The employees did sign this agreement, but it was never signed by the employer. However, I never signed one myself. Now I am being accused of knowing that I should have signed one of these agreements. Do they have any legal action against me for not signing this agreement, leaving and starting my own business? If their clients do come to me for service, can I be sued?