Can employer prevent me from making a living by working other place.

I am an on-call/as needed sign language interpreter at a local agency. I am also a student at a local community college. In the past the college had contracted with the agency to provide interpreter services. The college has not had a deaf student and not used the agencies services for at least the past two years. The college now has a possible deaf student and hired me as a student tutor/interpreter. I checked with the HR department at the agency and was told that they do not have a non-compete policy. The CEO at the agency is now questioning my actions as possible conflict of interest. Additionally, to clarify the actions of the agency, within the past several months I have been assigned less than 25 hours of work, and, per a recent policy newly established, can be called off of an assignment up to four hours prior in favor of being replaced by a staff interpreter if the staff interpreters schedule becomes open. I feel that my accepting an employment position at the college is not a conflict of interest in that I was told by the agencies HR that the agency does not have a non-compete policy and that I did not independently contract with the school but rather applied for and was hired as a employee/student tutor/interpreter. Also, I feel due to the agencies actions, I am being prevented from making a reasonable living by not giving me hours and possibly attempting to prevent me from working at the school.