How about copyright infringement, false advertising, sharing classified documents, tobacco advertising? There are a LOT of special cases.
Maybe there shouldn’t be and all of those things should be free from government interference, but the status quo is that plenty of speech is regulated.
Copyright violation is not speech.
Copying someone else's work and presenting it as your own is not protected speech, it is fraud. If you want to still consider this speech, someone is using it to actively and intentionally harm others.
If you're so intent on free speech, how would you feel if someone followed you around screaming profanities and abuse? Are you engaging in harmful and immoral censorship by calling the cops to remove this person? Are you suppressing his right to free speech by taking out a restraining order?
Copyright enforcement is not a form of censorship.