Learn how to implement evaluation-driven AI observability for agents to enhance monitoring, testing, and decision-making in production environments.
César Miguelañez
The rise of AI agents has introduced transformative possibilities for businesses, but running these agents in production is fraught with challenges. Developers and AI engineers often find themselves grappling with unreliable outputs, unpredictable behavior, and a lack of proper feedback loops. Traditional software observability tools fall short when it comes to managing the functional complexity of AI agents. In his insightful talk, Shri, a product lead at Datadog, explores how eval-driven development can bridge the gap, providing a structured methodology for improving AI agent reliability and functionality.
This article unpacks the key ideas from the talk, outlining practical strategies for building observability into AI agents and ensuring their outputs are not just operationally sound but functionally accurate. Whether you’re an engineer running LLMs in production or a technical lead responsible for scaling AI systems, this guide will illuminate how to manage AI agent complexity with confidence.
What Makes AI Agents Different?
At first glance, building AI agents may seem similar to standard software development. After all, AI agents are still applications running on infrastructure. But here’s the twist: the fundamental question shifts from "Is it running?" to "Is it right?" This added layer of complexity stems from the non-deterministic nature of AI agents and the challenges of evaluating their outputs at scale.
The Missing Feedback Loop
In traditional software engineering, a robust feedback loop ensures reliability. You write code, test it, and validate it through CI/CD pipelines. If a test fails, you know exactly what to fix. However, for AI agents, this loop is broken:
The problem lies in decision logic, not just in code.
Non-deterministic behavior means outputs can vary unpredictably.
There’s no easy way to write tests for AI outputs, making it difficult to assess whether the agent’s decisions are consistently correct.
Two Metrics Engineers Must Balance
Shri introduces the concept of operational performance and functional performance as two key metrics to manage AI agents effectively:
Operational Performance: Is the system up? Is it fast? Are there 500 errors? This is familiar territory for most engineers.
Functional Performance: Is the output correct? Is it useful? Is the model hallucinating? This is the new frontier for AI observability.
The challenge lies in managing these metrics together to ensure that AI agents are not only operating but delivering reliable and accurate results.
Introducing Eval-Driven Development
To address the challenges of AI observability, Shri proposes a methodology called Eval-Driven Development (EDD). This approach adapts the best practices of software development to the unique requirements of AI agents.
The Eval-Driven Development Loop
At its core, EDD introduces a systematic feedback loop into the AI agent lifecycle:
Develop: Make changes to prompts, models, or tools used by the AI agent.
Evaluate: Write tests (evals) that measure the functional quality of the agent’s outputs. These tests act as the equivalent of unit or integration tests for AI agents.
Iterate: Analyze eval scores to assess the impact of changes, tweaking the agent as needed.
This process reduces iteration cycles, enabling engineers to fine-tune AI agents quickly and confidently.
Why Evals Are the New Tests
Evals are the cornerstone of EDD. They allow engineers to evaluate whether an AI agent is meeting functional requirements. Shri emphasizes the importance of treating evals as code, meaning they should be version-controlled, logged, and integrated into CI/CD pipelines. By doing so, engineers can:
Maintain a consistent testing framework.
Track changes to eval criteria over time.
Ensure that new iterations of an AI agent meet functional performance benchmarks.
Building Observability for AI Agents
While eval-driven development provides the methodology, observability tools are essential to implement it effectively. Shri highlights three core features needed for AI observability:
1. Agentic Tracing
Agentic tracing captures every decision an AI agent makes, including inputs, outputs, and intermediate steps. This level of granularity helps engineers diagnose failures and understand why certain decisions were made.
For example, if an agent calls a tool incorrectly, agentic tracing can pinpoint the exact step where the error occurred. This provides transparency and enables root cause analysis.
2. Data Sets as Golden Examples
A robust data set acts as a reference point for testing AI agents. By storing inputs with expected outputs, engineers can validate whether the agent is functioning as intended. Data sets also evolve as real-world edge cases and failure scenarios are identified in production, ensuring that testing reflects the complexities of actual usage.
3. Experimentation at Scale
Experimentation is a systematic process of tweaking prompts, models, or tools and measuring their impact. Every change is logged as an experiment, creating an institutional memory of decisions made. By comparing eval scores across experiments, engineers can identify which changes lead to improvements and which ones introduce regressions.
The Role of Automation
Shri discusses how automation can further accelerate these processes. For instance:
Prompt Optimization: AI tools can automatically iterate on prompts to improve eval scores.
Autotuning LLM Judges: Instead of manually calibrating evaluation models, automation can optimize them based on feedback and examples.
Production Root Cause Analysis: Automated systems can detect failure patterns in production traces, surfacing actionable insights.
Practical Tips for Success
Shri shares lessons learned from working with customers and internal teams developing AI agents. These insights can help teams avoid common pitfalls:
Ship Small, Ship Early: Instead of building massive data sets upfront, release early versions of the agent to real users (even internal users). This accelerates feedback and highlights real-world failure modes.
Focus on End-to-End Evals: While it might be tempting to test every individual step, prioritize evaluations that measure the final outcome. This allows the agent to leverage its non-deterministic nature while ensuring the end result is correct.
Avoid Over-Specificity: Keep evals broad enough to accommodate new use cases without constant rewrites. This ensures scalability as the AI agent evolves.
Key Takeaways
AI Agents Require a New Observability Paradigm: Unlike traditional applications, AI agents demand functional observability in addition to operational monitoring.
Functional Performance Is Critical: Ensure outputs are accurate, reliable, and useful by balancing operational and functional metrics.
Eval-Driven Development Is the Key: Use evals to create a feedback loop that accelerates iterations and improves agent quality.
Agentic Tracing Enhances Transparency: Capture every decision your AI agent makes to understand and diagnose failures.
Data Sets Reflect Real-World Scenarios: Build and evolve golden data sets to test agents effectively.
Experimentation Drives Progress: Treat every change as an experiment, using eval scores to guide decisions.
Automate the Loop: Leverage tools for prompt optimization, LLM judge autotuning, and root cause analysis to accelerate development.
Start Small, Learn Fast: Begin with simple end-to-end evals and expand testing based on real-world usage.
Conclusion
Building reliable AI agents requires a shift in mindset and tools. Eval-driven development and AI observability frameworks offer a structured way to ensure both operational and functional performance. By implementing principles like agentic tracing, experimentation, and version-controlled evals, teams can achieve the confidence needed to deploy AI agents in production at scale.
The journey from prototype to production doesn’t have to be a black box. With the right methodologies and tools in place, engineers can transform their AI agents from high-functioning prototypes into trusted, production-ready applications.
Source: "Practical AI-Enabled Observability for Agents and LLMs" - Datadog, YouTube, Apr 7, 2026 - https://www.youtube.com/watch?v=Xe60gkyDtGw
