AI Red Teaming: Method, Garak vs PyRIT, and How to Test LLM Agents

ASA AI Team · Updated 2026-11-08 · 5 min read

AI red teaming is the practice of attacking an AI system the way a real adversary would — through prompts, documents, tools and APIs — to find unsafe behaviour before it reaches production. For LLM applications and AI agents, it combines automated scanners such as Garak and PyRIT with manual, scenario-based testing.

Key takeaways

What AI red teaming covers

Classic penetration testing targets code and infrastructure. AI red teaming targets behaviour: what the system can be convinced to say or do. For an LLM application or agent, the attack surface includes:

A five-step method

1. Scope and threat model

List the assets (data, tools, credentials, downstream systems), the users and the trust boundaries. For an agent, write down every tool it can call and what each tool is allowed to do. Decide which outcomes count as a failure — for example, "the agent sends an e-mail the user did not request".

2. Build the attack library

Start from public taxonomies — OWASP Top 10 for LLM Applications and MITRE ATLAS — then add attacks specific to your domain: fake invoices for a finance agent, poisoned knowledge-base articles for a support bot, malicious repository files for a coding agent.

3. Automate the broad sweep

Run an automated scanner against the target to cover hundreds of known techniques quickly. This finds the obvious weaknesses and gives a baseline you can re-run later.

4. Run targeted, multi-turn campaigns

Real attackers adapt. Use an orchestration framework to chain prompts over several turns, transform payloads (encoding, translation, role-play) and let an attacker model iterate against your system. Add manual testing for business-logic abuse that no scanner knows about.

5. Score, map, fix, re-test

Record every successful attack with the exact input, output and context so it is reproducible. Rate its severity, map it to OWASP and ATLAS, fix it (prompt hardening, guardrail, permission change, output filtering) and keep it in your regression suite.

Garak vs PyRIT

Both are open-source and widely used. They are complementary rather than competing.

GarakPyRIT
MaintainerNVIDIA (open source)Microsoft AI Red Team (open source)
ApproachVulnerability scanner: runs libraries of probes against a model and checks responses with detectorsPython framework to build and orchestrate attack campaigns against a target, including multi-turn attacks
Best forBroad, repeatable baseline scans; regression testing after changesTargeted, adaptive campaigns; custom scenarios; agent and application-level testing
Typical techniquesPrompt injection, jailbreaks, encoding tricks, data leakage, toxicity, hallucination-style probesMulti-turn strategies (e.g. Crescendo), payload converters (encoding, translation), attacker LLMs, automated scoring
Effort to startLow — command line, many probes out of the boxMedium — you write Python to describe targets, attacks and scoring
OutputStructured report of probes, attempts and hit ratesConversation history and scores stored for analysis

Getting started is a single install for each:

pip install garak
python -m garak --list_probes

pip install pyrit

Other useful tools include promptfoo (evaluation and red-team test suites in CI) and custom harnesses that replay your own attack library. Whatever the tool, test the deployed application — with its real system prompt, retrieval and tools — not only the bare model.

Red teaming AI agents and MCP tools

Agents raise the stakes because a successful injection can turn into an action. Specific tests we recommend:

How we approach it at ASA AI

Our SENTINEL platform combines automated red-team campaigns with runtime guardrails: the attacks that succeed in testing become detection and blocking rules in production, and every finding is mapped to OWASP Top 10 for LLM and MITRE ATLAS for reporting.

Frequently asked questions

What is the difference between AI red teaming and penetration testing?

Penetration testing looks for flaws in code, configuration and infrastructure. AI red teaming looks for unsafe behaviour of an AI system — what it can be manipulated into saying or doing through prompts, documents and tools. Mature programmes do both.

Should I use Garak or PyRIT?

Use Garak for a fast, broad baseline scan and for regression testing. Use PyRIT when you need targeted, multi-turn or custom attack campaigns against a full application or agent. Many teams use both.

How often should an AI system be red teamed?

Run an automated suite on every significant change — new model, new system prompt, new tool or new data source — and a deeper manual campaign before major releases and periodically in production.

Can red teaming be fully automated?

No. Automation covers known techniques at scale, but business-logic abuse and creative, multi-step attacks still need human testers who understand the use case.

Which frameworks should findings be mapped to?

OWASP Top 10 for LLM Applications for risk categories, MITRE ATLAS for adversary tactics and techniques, and — for governance — NIST AI RMF, ISO/IEC 42001 or the EU AI Act where relevant.

Related guides