Non-Compete

Hello,
Background:
I had to sign a Non-Compete Agreement in order to work at my previous company. I had told the owner that I was uncomfortable signing it, but he said it was just part of the "new hire process". After I worked at that employer for 2.5 years, I quite because of health reasons. My employer sent me all over the tri-state area and for over 9 months I was not living at home but in a hotel. At that point my health was affected and I started to get "panic attacks". I saw a doctor and the doctor told me to get a more stable job that would not require travel. I told this to my employer and at that same time a client called and specifically requested me to work on a 1-year project at their site. My employer turned the job down because the client would not pay the requested hourly rate. I told my employer I needed a project like this that would not require travel. He still said no, and I was no working in NYC and CT every week. After two months the client contacted me directly and asked if I would like to work for them. I told them I would consider it.
I quite my job and started my own company working locally in my hometown. At the same time I discussed working with the client and we started contract negotiations. Their attorneys reviewed my "non-compete" agreement and said there would be no problem.
I had a new role at the client as "Project Team Lead" and had 5 other consultants report to me to do the project. My previous employer found out I was working there and has threaten to sue based on breach of contract. They said I disclosed company information and solicited business with a former client.
I never used any of the company information or instruction.
Question:
Do they have a valid case?