When terminated can you work for a client that contacted you for employment?

I am a non-CPA accountant that has worked for a local accounting firm for three years and I was let go in April for reasons not involving work product. I do not recall ever signing a non-compete agreement with this firm, when I first was hired it was not ready, I do not have a copy of one in my file and I do not remember one being presented to me. After the firm notified the client that I was no longer there I have been approached by the client to work for them...they have been a client at the firm for almost two years and I did the majority of the work for them while there. I have been told by the client that they have been thinking about bringing all of their accounting work in house for years, so it is perfect for them and myself. I have been told by several other accountants that work at various firms that if I am an "employee" that my employment with them is OK - that I cannot consult or go to another firm and then take them away. With this information provided is this correct? I have been an accountant/bookkeeper for 30 years so this is my known trade/profession and what I did at this firm is no more than I learned at my previous positions.