Can I be forced to sign a non-compete on my way out of a company?

My employer claims to require all prospective employees to sign a non-compete before commencing employment with the company. I initially refused to sign the non-compete, but my employer put me to work right away anyway. Several months later, I was given the non-compete agreement to sign again, and I took it home to read privately. I didn't sign it, return it, or initiate any sort of conversation about it, and the non-compete agreement was never mentioned to me again as long as I was employed by the company.

Now that I am leaving the company, my employer is asking me to sign the non-compete on my way out, claiming I am still liable to sign it and adhere to it because a non-compete is "required of all employees" under company policy and I took the non-compete home and didn't return it as a "subversive tactic" to avoid signing it.

Can my employer take any legal action against me for working for the company without signing the agreement? What validity does the "subversive tactic" argument have?