This product is not supported for your selected
Datadog site. (
).
Overview
Agent Observability offers several ways to support evaluations. They can be configured by navigating to AI Observability > Evaluations.
Custom LLM-as-a-judge evaluations
Custom LLM-as-a-judge evaluations allow you to define your own evaluation logic using natural language prompts. You can create custom evaluations to assess subjective or objective criteria (like tone, helpfulness, or factuality) and run them at scale across your traces and spans.
Managed evaluations
Datadog builds and supports managed evaluations to support common use cases. You can enable and configure them within the Agent Observability application.
Submit external evaluations
You can also submit external evaluations using Datadog’s API. This mechanism is great if you have your own evaluation system, but would like to centralize that information within Datadog.
Building custom evaluators
For developers building custom evaluators, see the Evaluation Developer Guide.
Evaluation integrations
Datadog also supports integrations with some 3rd party evaluation frameworks, such as NeMo.
Annotation Queues
Annotation Queues provide a structured workflow for systematic human review of LLM traces.
Sensitive Data Scanner integration
In addition to evaluating the input and output of LLM requests, agents, workflows, or the application, Agent Observability integrates with Sensitive Data Scanner, which helps prevent data leakage by identifying and redacting any sensitive information. For a list of the out-of-the-box rules included with Sensitive Data Scanner, see Library Rules.
Security
Get real-time security guardrails for your AI apps and agents
AI Guard helps secure your AI apps and agents in real time against prompt injection, jailbreaking, tool misuse, and sensitive data exfiltration attacks. Try it today!
Permissions
Agent Observability Write permissions are necessary to configure evaluations.
Retrieving spans
Agent Observability offers an Export API that you can use to retrieve spans for running external evaluations. This helps circumvent the need to keep track of evaluation-relevant data at execution time.
Further Reading
Additional helpful documentation, links, and articles:
