AI Guardrails for LLMs and Agents: Types, Open-Source Tools and Limits
AI guardrails are runtime controls placed around a large language model or AI agent to check what goes in and what comes out — blocking prompt injection, jailbreaks, sensitive data leaks, off-topic requests and unauthorised tool calls. They are the production counterpart of red teaming: red teaming finds the weaknesses, guardrails enforce the rules.
Key takeaways
- Guardrails act at four points: user input, retrieved or tool content, model output and tool calls.
- Open-source building blocks include NVIDIA NeMo Guardrails, Guardrails AI, Meta Llama Guard and Protect AI LLM Guard.
- Guardrails reduce risk but can be bypassed: they complement least privilege and human approval, they do not replace them.
- Measure them like any security control: detection rate on your red-team suite, false-positive rate and added latency.
Where guardrails sit
| Checkpoint | What it checks | Typical action |
|---|---|---|
| Input rails | User prompts: prompt injection, jailbreak patterns, abusive or off-topic requests, personal data | Block, rephrase, redact or route to a human |
| Retrieval and tool-content rails | Documents, web pages and tool outputs before they reach the model (indirect prompt injection) | Strip or quarantine suspicious instructions; flag the source |
| Output rails | Model answers: secrets, personal data, harmful or non-compliant content, hallucinated facts, unsafe code | Redact, block, add a disclaimer or regenerate |
| Execution (tool-call) rails | Which tool an agent wants to call, with which parameters, for which user | Allow, deny, or require human approval |
Detection techniques
- Rules and patterns — regular expressions, keyword lists, secret and personal-data detectors. Fast and predictable, easy to bypass alone.
- Classifier models — small models trained to detect prompt injection, jailbreaks or unsafe content. Good coverage, some false positives.
- LLM-as-a-judge — a second model evaluates the request or answer against a policy. Flexible, but slower and itself attackable.
- Policy engines — deterministic allow-lists for tools, destinations and data, independent of any model. The strongest control for agents.
Open-source guardrail tools
| Tool | Maintainer | Approach | Strong for |
|---|---|---|---|
| NeMo Guardrails | NVIDIA | Programmable rails (input, output, dialog, retrieval, execution) described in configuration and the Colang language | Conversation flows, topical control, orchestrating several checks |
| Guardrails AI | Guardrails AI (open source) | Python framework of validators applied to model inputs and outputs | Structured output, format and content validation |
| Llama Guard | Meta | Open-weight safety classifier model for prompts and responses | Content-safety classification against a taxonomy |
| LLM Guard | Protect AI | Library of input and output scanners (prompt injection, secrets, PII, toxicity…) | Quick, broad scanning in a Python pipeline |
Commercial cloud services offer similar capabilities (for example prompt-injection shields and content-safety APIs). For regulated or sensitive data, running guardrails on-premises keeps prompts and responses inside your perimeter.
Limits you must plan for
- Bypass — encoding, translation, multi-turn manipulation and novel phrasing defeat filters. Test guardrails with red teaming, not only with examples.
- False positives — overly strict rails frustrate users and push them towards shadow AI.
- Latency and cost — every extra model call adds time. Run cheap checks first and expensive ones only when needed.
- False confidence — a guardrail is not an authorisation system. An agent that holds broad credentials stays dangerous even behind good filters.
A deployment checklist
- Write the policy first: what is forbidden, for whom, and what happens on detection.
- Combine layers: deterministic rules, a classifier and tool-call policies.
- Scan untrusted content (RAG documents, tool outputs) as carefully as user input.
- Log every decision with the reason, and send blocks to your SIEM.
- Measure detection and false-positive rates on your red-team suite after every change.
- Keep least privilege and human approval for high-impact actions, whatever the guardrails say.
Frequently asked questions
What are AI guardrails?
Runtime controls around an LLM or AI agent that check inputs, retrieved content, outputs and tool calls against a policy, and block, redact or escalate what violates it.
Is NeMo Guardrails open source?
Yes. NVIDIA NeMo Guardrails is an open-source toolkit for adding programmable input, output, dialog, retrieval and execution rails to LLM applications.
Can guardrails stop prompt injection completely?
No. They significantly reduce it, but determined attackers can find bypasses. Combine guardrails with least privilege, isolation of untrusted content, human approval for sensitive actions and continuous red teaming.
Should guardrails run in the cloud or on-premises?
Both work. On-premises deployment keeps prompts and responses — which often contain sensitive data — inside your own infrastructure, which matters for regulated sectors.
Related guides
- AI Red Teaming — Method, Garak vs PyRIT, and how to test LLM applications and agents.
- AI Agent & MCP Security — Threats to agents and MCP servers, with a practical checklist.
- OWASP Top 10 for LLM — The ten critical LLM risks, controls, and what changed in 2026.