Can an employer dictate who employees fraternize with outside of work?

I work for a union. (Directly for the union, that is - not as a member of that particular union.) One of my superiors recently resigned, evidently as a result of internal politics and conflict with the union. He was beloved by all the rest of us non-union people, and we wanted to have a party, outside of work, on a weekend, to say goodbye to him. The party was going to be at a restaurant where he lives, in Sacramento, over two hours' drive from our workplace. It was being organized through word of mouth, not over work email, but everyone in our office was invited. Including three co-workers who are representatives of the union which employs us.

Unfortunately, the party was canceled. We were told that we could not fraternize with our ex-boss, and that anyone who did could be fired.

Is this legal? Or: HOW is this legal?

This is not a first; one co-worker got in trouble for writing a heartfelt email to all-staff when this guy left, lauding him. Word on the street has it that he was nearly fired too. That's one thing, I suppose, because it's company email. But how can they dictate who we talk to or hang out with outside of work hours? As far as I remember, there's nothing in our contract about our leisure time, but I suppose now I'd better double-check!

Thank you for your help!

- Dani