Making team members work after saying its voluntary

This week at work my supervisor said that we had the option to work or not work the upcoming week. I decided to spend time with family and plan to go out of town, so my decision was not to work. In the interim my supervisor took names of individuals that wanted to work so they could get the amount of work done needed. Another team member signed up to work and the next day decided that she didn’t want to work and asked to have her name taken off the list. My supervisor did tell her that he couldn’t take her name off since the schedule was set. After some back and forth between the two, he decided to take her name off. He later approached me and told me that since that team member wasn’t working I had too. I then told him that I have already made plans and didn’t want to work since it was voluntary. He didn’t budge and told that I had to be there. Also, there was a different team member there that wants to work but our supervisor refuses to let them come in. What can I do about this, am I legally obligated to come in to work? Can my supervisor single me out only and make me work? Should the first team member who signed up be obligated to work? Any info you can give will be greatly appreciated.