Prompt Injection as Role Confusion

Simon Willison highlights new research on prompt injection that reframes the problem as 'role confusion.' Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell tested whether LLMs can distinguish trusted role tags like <system> or <think> from untrusted <user> input. Their findings confirm that models cannot reliably separate these, and worse, they appear to be more influenced by the writing style of the text than by the explicit tags. For example, mimicking the style of internal thinking blocks can lead to jailbreaks that override safety training. The researchers also discovered that 'destyling'—rewriting input in a less format-like style—significantly reduces the model's confusion, even though the meaning remains identical to a human reader. This work underscores a fundamental limitation in current LLM architecture: the inability to enforce role-based boundaries through formatting alone. For developers building AI workflows, this means any reliance on tag-based security is fragile and must be supplemented with robust input validation and output filtering.