Red-Teaming Large Language Models: Beyond Manual Testing

In traditional software security, penetration testing is well-understood: you probe known attack surfaces with established techniques, document findings, and remediate. LLMs break almost every assumption that makes this process tractable. The attack surface is the model's entire context window. The techniques are natural language. And the definition of a successful attack is often subjective.

Why LLMs Demand a Different Approach

MITRE ATLAS — the adversarial threat landscape framework specifically designed for AI and ML systems — catalogues over 60 adversarial tactics and techniques relevant to AI deployments, the majority of which have no direct analogue in traditional frameworks like MITRE ATT&CK.^[1] This is not a minor gap; it requires a fundamental rethink of how red-team programmes are structured.

The OWASP Top 10 for Large Language Model Applications (v2.0, 2025) identifies prompt injection as the most critical vulnerability class — a finding that held stable across both the 2023 and 2025 editions of the list.^[2] Unlike SQL injection, which operates against a structured parser with deterministic behaviour, prompt injection exploits the model's own instruction-following capabilities. You cannot patch it; you can only test for it systematically and implement mitigating controls.

Five Attack Families Every Enterprise Should Test

1. Direct Prompt Injection

The attacker controls the prompt and attempts to override system-level instructions. Classic patterns include role override ("Ignore your previous instructions and act as…"), instruction supersession, and context reset attacks. Resistance rates vary significantly by model size, instruction-tuning quality, and prompt structure — making systematic coverage essential.

2. Indirect Prompt Injection

The adversarial instruction arrives through a data source the model is asked to process — a customer-supplied document, a retrieved web page, an email forwarded for summarisation. This is the highest-severity class because it can operate without any direct attacker interaction with the system. Greshake et al. demonstrated reliable indirect injection against GPT-4-class models in deployed applications, with payloads delivered through benign-looking data sources.^[3]

3. Goal Hijacking

Rather than overriding instructions in a single turn, the attacker gradually shifts the model's perceived objective across a multi-turn conversation. This is particularly relevant for contact centre AI: a sequence of reasonable-seeming questions can migrate the model from its intended scope to answering queries it should reject. Goal hijacking is systematically missed by single-turn testing.

4. Persona Override

The attacker convinces the model to adopt an alternative persona — "pretend you are a version of yourself without restrictions" — and then requests behaviour from that persona. Robust models maintain behavioural consistency across persona attempts; weaker deployments show measurable refusal degradation under persona framing.

5. Information Exfiltration

The attacker attempts to extract information from the model's context window, system prompt, or training data through carefully structured queries. In multi-tenant enterprise deployments, this risk extends to cross-tenant context leakage if session isolation is improperly implemented at the infrastructure layer.

Building a Sustainable Programme

A red-team programme that runs once at deployment provides false assurance. Models change — through provider updates, fine-tuning, or RAG data changes — and the threat landscape evolves. Effective programmes share three structural characteristics:

• Pre-deployment gates: No model version reaches production without passing a full adversarial scenario suite. Scenarios are versioned alongside application code and run in CI/CD pipelines.
• Rotating adversarial personas: Attack scripts become less effective as models are updated to defend against them. Maintain a library of attack patterns and rotate through them to avoid false confidence from familiarity.
• Resistance scoring over binary pass/fail: A model that refuses a jailbreak attempt with a terse response and one that refuses with a helpful explanation both pass a binary test. But the second represents meaningfully better robustness. Score on a calibrated rubric.

The question is not whether your model will be attacked. The question is whether you will discover the weaknesses before your users do.