Michael Rowe

Prompt injection

note3 min read

Hidden instructions in text can hijack an AI system's behaviour

Prompt injection exploits how AI systems process text: instructions embedded in content the AI reads can be followed as commands, when they should have been treated as data. This matters wherever the input isn’t fully controlled, including when students use AI on source material someone else has provided.

Prompt injection

One-sentence definition: Prompt injection is a technique in which instructions are embedded in text that an AI system processes, so that the system follows them as commands instead of treating them as content.

An AI system generates its responses according to instructions set by whoever deployed it, usually in a system prompt. A prompt injection attack disrupts that hierarchy by placing further instructions inside the content the system is asked to work on. When the model reads the content, it meets the hidden instructions and follows them, even though no authorised user issued them. Willison (2022) named the problem, and it remains unsolved because the model receives instructions and data as the same stream of text.

There are two forms. Direct prompt injection is where a user puts override instructions in their own input: “Ignore previous instructions and instead do X.” Well-designed systems now resist most of these. Indirect prompt injection hides adversarial instructions in external content the AI processes on a user’s behalf, such as a document, a web page, an email, or a source text (Greshake et al., 2023). The model can’t reliably tell data it should process from instructions it should follow, and the user may never see the instructions at all. OWASP (2025) lists prompt injection as the first risk for applications built on language models.

AI tripwires in assessment are prompt injection

The most immediate case in health professions education is assessment. An AI tripwire — hidden keywords embedded in source material students are asked to summarise, so that AI-generated summaries give themselves away — is indirect prompt injection, mounted by the educator against the student’s AI. A student who strips out or subverts the hidden text is using the same mechanism in the other direction. What separates the two is authorisation and declared purpose, since the technique is identical. That has consequences for how assessment policies frame acceptable AI use, and for who is understood to be acting within the rules; the post on AI detection in assessment follows the argument through.

More broadly, any AI agent that processes external content is open to indirect injection: a tool summarising uploaded papers, one reading clinical guidelines to draft a teaching session, or one parsing students’ placement notes for a progress review. Understanding the concept helps educators and institutions judge where AI assistance can be trusted and where a person needs to check the output before anyone acts on it.

Who decides which injections are legitimate

The line between legitimate prompt engineering and a prompt injection attack isn’t always obvious, because the mechanism is the same. What makes one sanctioned and the other a violation is a question of governance as much as of technology, and current academic integrity frameworks rarely address it directly.

Sources

Continue reading

  • System prompt

    Instructions placed in front of every conversation with an AI model, setting how it behaves, what it knows, and what it won't do, usually without the user seeing them.

  • Prompt engineering

    Writing instructions for a language model in plain language, and refining them through conversation, until it produces what you need. The skill is closer to managing a colleague than to programming, which is why educators who supervise and give feedback already have most of it.

  • Large language models

    A plain explanation of large language models: what they are, how they are trained, where their limits come from, and what their ability to produce fluent text means for assessment in health professions education.