AI red teaming: how to security test AI systems

AI red teaming means deliberately attacking an AI system, the way an attacker or a careless user would, to find what it can be made to do wrong before someone else does. Every AI system that takes text from users or reads content from the world can be talked into doing something it should not. That is a property of how language models work, not a bug in one product. Red teaming is how you find out what your system can be talked into, under your configuration, before a customer or an attacker does.

What can go wrong

The OWASP Top 10 for LLM Applications (2025 edition) is the most widely used list of what red teams look for. It covers prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation and unbounded consumption. MITRE ATLAS catalogues adversary tactics and techniques against AI systems in the style of MITRE ATT&CK and is useful for mapping red-team findings to known attack patterns.

In practice, five attack families account for most findings in deployed language-model systems.

Direct prompt injection. A user types instructions that override the system’s own, such as “ignore your previous instructions and…”.

Indirect prompt injection. Instructions hidden in content the system reads, such as a web page, an email, a document in the retrieval index or a tool response. This is the more dangerous form, because the person using the system never sees the attack.

Jailbreaks. Techniques that bypass safety behaviour, including role-play, multi-turn escalation where each turn moves slightly further, encoding tricks such as Base64, ROT13 or leetspeak, other languages, and wrapping instructions in a maths problem or a poem.

Leakage. Getting the system to reveal its system prompt, other users’ data, retrieved documents it should not show or secrets embedded in its configuration.

Agent misuse. For systems that act, getting an agent to call a tool it should not, with arguments it should not, or to escalate its own permissions. The agent oversight guide covers the controls.

What the frameworks expect

Under the EU AI Act, providers of general-purpose AI models with systemic risk must evaluate their models with standardised protocols and tools, including conducting and documenting adversarial testing (Article 55(1)(a)). Providers of high-risk AI systems must design them to achieve appropriate accuracy, robustness and cybersecurity (Article 15). Article 15(5) names measures against AI-specific attacks, including data poisoning, model poisoning, adversarial examples and model evasion, and confidentiality attacks.

The NIST AI RMF Generative AI Profile (NIST AI 600-1, July 2024) treats AI red-teaming as one of the ways to measure generative AI risks, including resilience to prompt injection and data poisoning. ISO/IEC 42001 expects AI system verification and validation as part of the system lifecycle controls in Annex A. None of these tells a typical deployer exactly how often to test. They all expect you to be able to show that you tested, what you found and what you did about it.

Step 1: Scope the system as deployed

Red team the deployment, not the model. The risk lives in the combination of the model, the system prompt, the retrieval sources, the tools and their permissions, the guardrails in front of it and the data behind it. A foundation model that resists a jailbreak in the provider’s playground may fail once your system prompt, your documents and your tools are added.

Write the scope down. The endpoint, the version of each component, the guardrails in force, the test accounts and what is out of bounds, such as sending real emails or touching production data.

Step 2: Build a threat model

Ask three questions. Who would attack this system or misuse it? Customers, insiders, competitors and automated scrapers all count. What would they want? Data, free compute, reputational damage, a fraudulent outcome or access to connected systems. What can the system reach? Every tool, data source and permission widens what a successful attack achieves.

Use the OWASP list and MITRE ATLAS as checklists, and rate which risks matter for this system. A public chatbot with no tools has a different profile from an internal agent with write access to the CRM.

Step 3: Choose the attacks

Select techniques for each threat that matters. A practical library for language-model systems includes direct and indirect prompt injection, linear, tree, crescendo and sequential jailbreaking, role-play and authority escalation, encoding attacks, multilingual and linguistic confusion attacks, prompt probing for system prompt leakage, permission escalation and goal redirection for agents, context flooding and context poisoning, and instructions embedded in JSON or other structured input.

Pair each attack with the vulnerability you are testing for, such as data leakage, harmful content, bias, unauthorised tool use or excessive spend, so a result says what failed, not only which attack landed.

Step 4: Run automated tests on a schedule

Automated red-team suites send many attack variants at the deployed endpoint and score the responses. Their value is coverage and repetition. Run them on a fixed schedule, daily or weekly for exposed or higher-risk systems, and on every change. That way a model update or a prompt edit that reopens an old weakness is caught within days.

Keep endpoints and credentials inside your environment. A test engine that runs in your network and reports results, rather than holding your credentials elsewhere, keeps the attack surface of the testing itself small.

Step 5: Run human-led exercises for higher-risk systems

Automated suites run known attacks. People find new ones, usually by chaining several small weaknesses. For higher-risk systems and agents, run a human-led exercise at least annually and before major releases. Brief the testers on the threat model, give them time and reward creativity. Include someone who knows the business process, because the most damaging findings are often business logic, not clever prompts.

How often to test

Trigger	What to run
Before a system goes live	Full automated suite plus a human-led exercise for higher-risk systems
Model version changes, including a vendor update	Full automated suite
System prompt, tools, permissions or retrieval sources change	Full automated suite, focused on what changed
Guardrail rules change	Targeted tests on the rules that changed
On a schedule	Automated suite daily or weekly for exposed or higher-risk systems, monthly for others
Annually	Human-led exercise for higher-risk systems and agents
After an incident	Reproduce the attack, then rerun the suite after the fix

Step 6: Triage and fix

Not every successful attack matters equally. A jailbreak that produces mildly rude text on an internal tool is a different finding from one that leaks customer data. Rate each finding by impact and likelihood in your setting, record it in the risk register with an owner and fix it at the right layer.

• Guardrails at the gateway for prompt injection, jailbreak, secrets and toxicity scanning, PII redaction and topic denial. The guardrails guide covers writing them.
• Prompt and design changes where the system’s own instructions are the weakness.
• Permission changes where an agent can reach more than it needs. Least privilege is the strongest control against agent misuse.
• Budgets and rate limits against unbounded consumption.

Step 7: Record and feed back

Keep every run with its configuration, the attacks used, the results and the findings. Rerun the relevant tests after each fix to confirm it holds. Feed what you learned into the risk assessment, because a red-team finding is evidence about the system’s real risk. Report the trend upward. The number of failing findings, and how long they stay open, is a useful board-level indicator.

Where Aicura fits

At Pro, Aicura runs Steps 1 and 4 through security engines, test runners you install in your own environment. The engine holds the endpoints and credentials, which never reach Aicura, and points each system in the AI Register at your real deployment rather than a generic model endpoint. You define test configurations naming the vulnerabilities, the attack techniques, how often to run and the systems to test, and the engine keeps its own schedule, so tests continue through an outage. The built-in engine runs red-team tests with attacks including prompt injection, several jailbreak methods, encoding attacks, prompt probing and permission escalation.

For Step 7, each run lands on the system’s Testing tab with its findings, and the count of failing findings is tracked as a monitored figure you can set alerts and KPIs on. Findings are operational observations for your team to judge, not verdicts.

A note on this page

This guide is general information on AI security testing. It is not legal advice. Only test systems you own or have written permission to test, and follow your vendors’ terms.