Can i work for a client indirectly through agency

I have been working for a third party logistics company for almost 4 years. My first account I worked for 2 years. these last two years I'm working on a different account. I informed my employer that I'm looking for another position so I started applying for temp agencies. I know there is a contract with the company and client that employees can't leave your current job to go work for a client. I just interview with the temp agency and found out that the position is with the client that I have worked the first two years. am I in violation of the contract between the company and the client if I go work for the temp agency and they just happen to place me there? The contract position with the temp agency is for one year before the client could hire me. I was the shipper and receiver on their account and was the direct contact for our client.