no compete convenant

i worked for a house cleaning co. for 3 years, recently new ownership took over and i lost my health benefits, i had to sign a non compete agreement or they would have let me go, i made it clear that i would have to look for new employment adventually. During my 3 years of employment with prior owners i had mentioned to my clients that i would eventually be leaving this company, and would like to just work for myself, several told me that if i decided to do that they would perfer to cancel with this company and hire me, Upon giving this new company my two week notice and letting my clients that i cleaned know i would be leaving the company try to work for myself, a few said they were very dissatified with this company and would cancel if i would consider cleaning them, otherwise they would cancel anyway as they didn't want anyone else in their homes, that they knew my work and trusted me. I never once asked any of them to cancel and go with me meaning i did not solicit them, they asked me. I told them i had signed an agreement not to solicit their customers their reply to me was you aren't soliciting me i am asking you,i have the right to hire who i want to clean my home. I have wanted to go on my own for along time now, but stayed with prior co. for the benefits, which i no longer have, i feel it is time to better myself. Guess my question is, if this people choose to fire this company and hire me [ it's only a few, and the company will loose them anyways ] can the employer sue me. Do I not have the right to work where i choose and at what i choose to make a better living? It's not like I'am giving these people any confidential information or anyting like that. Nor did I solicit them.