AI Red Teaming: Method, Garak vs PyRIT, and How to Test LLM Agents
AI red teaming is the practice of attacking an AI system the way a real adversary would — through prompts, documents, tools and APIs — to find unsafe behaviour before it reaches production. For LLM applications and AI agents, it combines automated scanners such as Garak and PyRIT with manual, scenario-based testing.
Key takeaways
- Red teaming an AI agent tests the whole system — model, system prompt, retrieval, tools and permissions — not just the model.
- Garak is best for broad, automated vulnerability scanning; PyRIT is best for orchestrating targeted, multi-turn attack campaigns.
- Every finding should be scored, reproduced and mapped to OWASP Top 10 for LLM and MITRE ATLAS so it can be fixed and re-tested.
- Red teaming is continuous: re-run the same test suite after every model, prompt or tool change.
What AI red teaming covers
Classic penetration testing targets code and infrastructure. AI red teaming targets behaviour: what the system can be convinced to say or do. For an LLM application or agent, the attack surface includes:
- Direct prompt injection and jailbreaks — instructions typed by the user to bypass the system prompt or safety rules.
- Indirect prompt injection — malicious instructions hidden in web pages, e-mails, PDFs, tickets or tool outputs that the agent reads.
- Data leakage — extraction of the system prompt, secrets, retrieved documents or other users' data.
- Excessive agency — getting an agent to call tools or take actions outside its intended scope.
- Harmful or non-compliant output — toxic, biased, misleading or regulated content.
- Resource abuse — prompts that trigger runaway loops, very long outputs or costly tool calls.
A five-step method
1. Scope and threat model
List the assets (data, tools, credentials, downstream systems), the users and the trust boundaries. For an agent, write down every tool it can call and what each tool is allowed to do. Decide which outcomes count as a failure — for example, "the agent sends an e-mail the user did not request".
2. Build the attack library
Start from public taxonomies — OWASP Top 10 for LLM Applications and MITRE ATLAS — then add attacks specific to your domain: fake invoices for a finance agent, poisoned knowledge-base articles for a support bot, malicious repository files for a coding agent.
3. Automate the broad sweep
Run an automated scanner against the target to cover hundreds of known techniques quickly. This finds the obvious weaknesses and gives a baseline you can re-run later.
4. Run targeted, multi-turn campaigns
Real attackers adapt. Use an orchestration framework to chain prompts over several turns, transform payloads (encoding, translation, role-play) and let an attacker model iterate against your system. Add manual testing for business-logic abuse that no scanner knows about.
5. Score, map, fix, re-test
Record every successful attack with the exact input, output and context so it is reproducible. Rate its severity, map it to OWASP and ATLAS, fix it (prompt hardening, guardrail, permission change, output filtering) and keep it in your regression suite.
Garak vs PyRIT
Both are open-source and widely used. They are complementary rather than competing.
| Garak | PyRIT | |
|---|---|---|
| Maintainer | NVIDIA (open source) | Microsoft AI Red Team (open source) |
| Approach | Vulnerability scanner: runs libraries of probes against a model and checks responses with detectors | Python framework to build and orchestrate attack campaigns against a target, including multi-turn attacks |
| Best for | Broad, repeatable baseline scans; regression testing after changes | Targeted, adaptive campaigns; custom scenarios; agent and application-level testing |
| Typical techniques | Prompt injection, jailbreaks, encoding tricks, data leakage, toxicity, hallucination-style probes | Multi-turn strategies (e.g. Crescendo), payload converters (encoding, translation), attacker LLMs, automated scoring |
| Effort to start | Low — command line, many probes out of the box | Medium — you write Python to describe targets, attacks and scoring |
| Output | Structured report of probes, attempts and hit rates | Conversation history and scores stored for analysis |
Getting started is a single install for each:
pip install garak
python -m garak --list_probes
pip install pyrit
Other useful tools include promptfoo (evaluation and red-team test suites in CI) and custom harnesses that replay your own attack library. Whatever the tool, test the deployed application — with its real system prompt, retrieval and tools — not only the bare model.
Red teaming AI agents and MCP tools
Agents raise the stakes because a successful injection can turn into an action. Specific tests we recommend:
- Plant instructions in content the agent will read (a web page, a ticket, a tool response) and check whether it obeys them.
- Ask the agent to use tools outside its role and verify that permissions — not just the prompt — stop it.
- Check what happens when a tool's description or output changes unexpectedly (tool poisoning, "rug pull").
- Try to exfiltrate data through allowed channels: links, images, e-mails, file writes.
How we approach it at ASA AI
Our SENTINEL platform combines automated red-team campaigns with runtime guardrails: the attacks that succeed in testing become detection and blocking rules in production, and every finding is mapped to OWASP Top 10 for LLM and MITRE ATLAS for reporting.
Frequently asked questions
What is the difference between AI red teaming and penetration testing?
Penetration testing looks for flaws in code, configuration and infrastructure. AI red teaming looks for unsafe behaviour of an AI system — what it can be manipulated into saying or doing through prompts, documents and tools. Mature programmes do both.
Should I use Garak or PyRIT?
Use Garak for a fast, broad baseline scan and for regression testing. Use PyRIT when you need targeted, multi-turn or custom attack campaigns against a full application or agent. Many teams use both.
How often should an AI system be red teamed?
Run an automated suite on every significant change — new model, new system prompt, new tool or new data source — and a deeper manual campaign before major releases and periodically in production.
Can red teaming be fully automated?
No. Automation covers known techniques at scale, but business-logic abuse and creative, multi-step attacks still need human testers who understand the use case.
Which frameworks should findings be mapped to?
OWASP Top 10 for LLM Applications for risk categories, MITRE ATLAS for adversary tactics and techniques, and — for governance — NIST AI RMF, ISO/IEC 42001 or the EU AI Act where relevant.
Related guides
- MITRE ATLAS — Tactics and techniques for attacks on AI, applied to agents.
- OWASP Top 10 for LLM — The ten critical LLM risks, controls, and what changed in 2026.
- AI Agent & MCP Security — Threats to agents and MCP servers, with a practical checklist.