My boss kept a spreadsheet of my appointments etc on his desktop without my knowledge, legal?

While looking for a contract in a shared administration folder, I found that my boss was keeping a spreadsheet of my appointments (both work comp related and personal) on his desktop computer. A copy is saved on the shared drive that only he, another manager and myself have access to. There are no spreadsheets of other employees, just me. I am the only female that works here. I feel that he did this to give me lower scores/low raise during our annual review. He has never showed me the spreadsheet nor does he know that I have seen it but he commented during my review "that my appointments and personal issues (family related) were interfering with my work". My appointments were for a work injury for which I was treating with 2 different providers every week. These appointments cannot be held against me, that I know. I have been told that what he did, keeping a spreadsheet of only me, in a place that he doesn't think I have access to, is illegal. Is this true?