AI Observability: What to Monitor in LLM Applications and AI Agents
AI observability is the ability to understand what an LLM application or AI agent is doing in production — which prompts it receives, what it retrieves, which tools it calls, what it answers, how much it costs and whether that behaviour is correct and safe. It extends classic logs, metrics and traces with signals specific to AI.
Key takeaways
- Trace every request end to end: user input, retrieval, model calls, tool calls and final answer.
- Monitor four families of signals: performance, cost, quality and security.
- Use OpenTelemetry and its generative-AI conventions to avoid vendor lock-in.
- Redact sensitive data before storing prompts and responses, and send security events to your SOC.
Why classic monitoring is not enough
An LLM application can return HTTP 200 in 800 ms and still be wrong, leaking data or following an attacker's instructions. Uptime and latency say nothing about whether the content is correct and safe. AI observability adds visibility into prompts, context, reasoning steps, tool usage and output quality.
What to monitor
| Family | Signals | Why it matters |
|---|---|---|
| Traces | Spans for each step: retrieval, model call, tool call, guardrail check, final answer | Reconstruct exactly what happened for any request or incident. |
| Performance | Latency per step, time to first token, error and timeout rates, retries | Find slow tools, overloaded models and failing integrations. |
| Cost | Input and output tokens per request, user, feature and model; tool-loop counts | Control spend and detect runaway agents or abuse. |
| Quality | Evaluation scores (relevance, groundedness, correctness), user feedback, refusal rate | Detect regressions after a model or prompt change. |
| Security | Prompt-injection and jailbreak detections, guardrail blocks, sensitive data in outputs, unusual tool calls | Detect attacks and data leaks early and alert the SOC. |
| Drift | Changes in input topics, retrieval hit rates and answer distributions over time | Spot when real usage diverges from what was tested. |
Instrumenting with OpenTelemetry
OpenTelemetry has semantic conventions for generative-AI operations, so model calls, token usage and tool invocations can be captured in a standard way and sent to any compatible backend. Instrument at three levels:
- Gateway level — every call to a model provider or local model passes through one point that records model, tokens, latency and errors.
- Application level — spans for retrieval, prompt assembly, tool calls and post-processing.
- Agent level — each reasoning step and tool decision, linked to the user and session.
Open-source building blocks
- OpenTelemetry for collecting traces, metrics and logs.
- Langfuse and Arize Phoenix for LLM tracing, prompt management and evaluations.
- OpenLLMetry for OpenTelemetry-based instrumentation of LLM frameworks.
- Your existing SIEM for security events, so AI incidents are handled like any other incident.
Privacy and retention
Prompts and responses often contain personal or confidential data. Redact or tokenise sensitive fields before storage, restrict who can read raw conversations, and set retention periods aligned with your data-protection obligations.
From observability to security
The same traces that explain a bad answer also reveal an attack: a sudden spike in tool calls, an agent reaching an unusual destination, a response containing secrets. Feeding AI telemetry into security monitoring — with detections mapped to OWASP and MITRE ATLAS — closes the loop between AI engineering and the SOC.
Frequently asked questions
What is the difference between AI observability and AI monitoring?
Monitoring tells you that something is wrong, through predefined metrics and alerts. Observability lets you understand why, by tracing every step of a request — prompts, retrieval, tool calls and outputs — and exploring it after the fact.
Should prompts and responses be logged?
Yes, for debugging, quality and security, but with redaction of sensitive data, strict access control and a defined retention period.
Which open-source tools support LLM observability?
OpenTelemetry for collection, and tools such as Langfuse, Arize Phoenix and OpenLLMetry for LLM-specific tracing and evaluation.
How does AI observability help security?
It provides the data to detect prompt injection, data leakage, abnormal tool usage and cost abuse, and to investigate incidents. Sending these events to the SIEM lets the SOC handle AI incidents like any other.
Related guides
- AI Agent & MCP Security — Threats to agents and MCP servers, with a practical checklist.
- EU AI Act & AI Governance — Timeline after the Digital Omnibus and a governance checklist.
- AI Red Teaming — Method, Garak vs PyRIT, and how to test LLM applications and agents.