AI threat modelling: what it covers that a standard penetration test does not

A penetration test is an authorised attack on a running system. It finds exploitable weaknesses in what you have deployed: injection flaws, broken access control, misconfiguration, exposed credentials. It answers the question "can an attacker break in?" with evidence.

AI features introduce failure modes that are not bugs in the traditional sense. A prompt injection that makes the model ignore its instructions is not a code flaw a scanner will flag. A model that answers questions using one customer's data inside another customer's session can pass every conventional test and still be a breach waiting for a trigger.

These weaknesses live in the design: which data the model can reach, what instructions it trusts, what actions it can take, and what happens to its output. By the time the system is running, the expensive decisions are already made.

A structured AI threat model walks the system against the attack classes that apply to language models and agents: prompt injection through user content and retrieved documents, data exfiltration through model output, abuse of tool and function-calling permissions, training and context data leakage, and supply-chain risk in models, plugins and embeddings sourced from third parties. Frameworks such as the OWASP Top 10 for LLM Applications give these categories names and structure.

Threat modelling comes first, while the design can still change cheaply. Penetration testing comes before go-live, to prove the controls survive a real attack. Skipping the threat model means the pen test finds design problems you can no longer afford to fix properly. Skipping the pen test means the design was never verified against a determined attacker. AI systems need both, in that order.