The efforts to avoid output with copyright violations in the Claude system prompt are interesting for a few reasons, one being the desperate attempt to get LLMs to obey hard constraints. LLMs work off correlations and similarity. It's not constraint-based search. Prompt engineering efforts keep hitting this and are frustrated by it but aren't going to be able to overcome it.
https://simonwillison.net/2025/May/25/claude-4-system-prompt/#seriously-don-t-regurgitate-copyrighted-content