Is this a legal tactic

I have been working for my current employer for 3 years now without a written or sighned contract. I started working for him for a set weekly rate which meant if we didn't work I didn't get payed by him. After two years without a raise, he offered to put me on Salary because besides doing the job he hired me for, I was doing alot of work for him and his family. Of course once on salary I was making less per week, but I knew that if work got slow that would benefit me.
Recently our work did slow down and now he has decided that I should not be on salary any longer. It doesn't seem like a fair practice, to raise or lower my salary whenever it suits him. I took a weekly pay cut while I worked almost every day for him. And Now that the hard work is done and there is less to do the benefits of a salary position are gone. There is also a rumor of me being terminated because he wants to downsize our crew.
Do I have any recourse. Or should I just quit, since this has all been based on oral agreements.
I would be willing to pay for legal services if I were to win some sort of settlement.