By Latitude · March 29, 2026

Key Takeaways

• LLM evaluation tools score individual responses; agent evaluation must assess goal-level outcomes across multi-turn sessions — an agent can rate "good" on every turn and still fail the user's intent.

• Compounding errors make per-step scoring deceptive: a 20-step workflow with 95% per-step reliability succeeds only 36% of the time overall.

• Latitude is the only platform in this comparison with GEPA auto-generated evals from annotated production failures — the eval library grows from your product's actual failure distribution, not generic benchmarks.

• LangSmith's Insights, Braintrust's Topics, and Langfuse all require manual steps to convert production observations into eval cases — there is no automatic loop.

• Product-aligned evals outperform generic benchmarks because they capture the failure modes that actually appeared in your system with your users — not the ones the team anticipated during development.

• The eval quality metric that matters: Matthews Correlation Coefficient (MCC) — measures whether your evals actually detect the failures they're designed to catch.

This piece was prompted by several conversations I've had with AI engineers who discovered, the hard way, that the eval tools that worked brilliantly for their LLM features produced false confidence when they shipped agents. This is an attempt to explain why that happens structurally — not as a critique of any particular tool, but as a guide for choosing the right evaluation approach for the specific system you're building.

The Problem: Your Eval Stack Wasn't Built for This

In 2024 and 2025, most teams evaluating LLM-powered applications were working with a simple architecture: one system prompt, one user message, one completion. Evaluating this is a tractable problem. You build a golden dataset, run your prompt against it, score each output with a rubric or LLM-as-judge, and track your score across model versions and prompt changes. Tools like Braintrust, Langfuse, and LangSmith were built for exactly this workflow — and they do it very well.

Then teams started shipping agents. Not "LLM with a retrieval call," but real agents: systems that plan multi-step tasks, call external tools, manage state across turns, spawn sub-agents, and make branching decisions based on intermediate results. And suddenly the eval stack broke.

The symptom: eval suites showed green, but production kept failing. Teams would improve their LLM-as-judge scores by 15%, deploy, and see no meaningful improvement in the metric that actually mattered — whether the agent completed the task correctly.

The root cause isn't a tooling bug. It's a category mismatch. Agents have four properties that make traditional LLM evaluation fundamentally inadequate:

1. Non-deterministic execution paths. The same input to an agent can produce completely different sequences of tool calls, reasoning steps, and intermediate outputs across runs. A golden dataset score assumes the evaluation surface is stable. For agents, it isn't.

2. Compounding errors across turns. A mildly incorrect answer in step 2 of a 12-step agent run doesn't just affect step 2's output — it corrupts the context that every subsequent step operates on. By step 8, the agent is reasoning about a world that was subtly misconstrued six steps ago. LLM eval tools score individual outputs; they don't detect this cascade.

3. Tool use and function calling. An agent that calls the wrong API endpoint, passes malformed arguments, or misinterprets a tool response has failed — but not at the text generation level. Its final output might look plausible while being based entirely on a silent tool failure. Evaluating the completion alone misses the failure completely.

4. State management across sessions. Multi-turn agents carry state — memory of previous turns, accumulated context, user preferences established earlier in the session. Failures that involve state corruption (the agent "forgets" a constraint the user established in turn 3, applies a rule that expired two sessions ago) are invisible to single-turn evaluation frameworks.

Hamel Husain, whose writing on evaluation has become required reading in AI engineering, makes a related point: "The most common mistake I see is teams building evals that measure what's easy to measure rather than what matters." For agents, what matters is almost always harder to measure than what traditional tools surface.

The Evaluation Framework: 5 Criteria That Actually Matter for Agents

Before comparing tools, we need to define what "good" evaluation looks like for agent systems specifically. Here are the five criteria I use when assessing any evaluation platform for agent work.

Criterion 1: Multi-Turn Conversation Tracing

Does the platform trace a full agent session — not just individual LLM calls — as a single, coherent unit? This means linking tool invocations, sub-agent spawns, memory reads, and intermediate reasoning steps into a unified trace that you can inspect as one object. Without this, you're evaluating snapshots of a movie, not the movie itself.

Why it matters: Most agent failures are emergent — they don't exist in any individual step, only in how steps relate. A trace view that surfaces the full conversation structure lets you ask: "What was the agent's state when it made this wrong decision?" That question is unanswerable if your tool only stores individual LLM call records.

Criterion 2: Tool Use and Function Calling Observability

Can you inspect every tool call — its inputs, outputs, return values, error states — as first-class objects in your trace? Not as metadata attached to an LLM span, but as spans in their own right with full context.

Why it matters: Tool failures are often the proximate cause of agent failures. If a retrieval call returns an empty result because of a malformed query, the agent might hallucinate a plausible-looking answer rather than surfacing the failure. Without tool-call observability, you see a confident wrong answer and have no idea why.

Criterion 3: Issue Discovery vs. Synthetic Benchmarks

Does the platform help you discover failure modes you didn't know to look for — or does it only score outputs against failure modes you predefined? Synthetic benchmarks are valuable for regression testing known issues. They're useless for discovering unknown issues, which is where most production agent failures live.

Why it matters: You can't write an eval for a failure mode you haven't observed yet. Any platform that requires you to define your evaluation surface in advance is systematically blind to novel failure patterns. Production AI systems fail in ways their developers didn't anticipate — your eval stack needs to surface those patterns, not just verify pre-hypothesized ones.

Criterion 4: Auto-Generated Evals from Production Data

Can the platform convert observed production failures directly into regression tests, without requiring you to manually write eval criteria? And crucially: does it do this in a way that incorporates human judgment, rather than just auto-scoring with an LLM?

Why it matters: Manual eval writing is the bottleneck for most teams. The gap between "we saw this failure in production" and "we have a regression test that catches this failure" is often weeks of engineering time. Platforms that close that gap automatically — while maintaining the human judgment loop that Hamel and others have correctly argued is essential — fundamentally change the eval velocity of the team.

A note on skepticism: Hamel has written critically about auto-generated evals that replace human judgment rather than supporting it. That critique is correct and worth internalizing. The goal isn't to automate away the domain expert's judgment about what "good" means — it's to automate the mechanical work of translating that judgment into runnable tests once the judgment has been made. The human stays in the loop; the tool eliminates the manual scaffolding work.

Criterion 5: Agent-Specific Failure Mode Clustering

Does the platform automatically group similar failure patterns together — so you can see "15% of sessions exhibit this specific type of tool call loop" rather than "here are 847 individual failed traces"? Clustering transforms an unmanageable volume of production failures into a prioritized list of addressable issues.

Why it matters: At production scale, you can't manually review every failed session. The teams that improve fastest are the ones that can identify the highest-impact failure patterns quickly, prioritize them, and verify fixes. Platforms that present raw trace data without clustering require expensive manual analysis at every step of this loop.

Tool Comparison: How 5 Platforms Perform Across These Criteria

Let me be direct about my position: I co-founded Latitude because I believe existing tools have a gap. But I've used LangSmith, Braintrust, and Langfuse extensively, and my assessment of their strengths is genuine. The goal here isn't to diminish competitors — it's to help you understand where each tool fits the specific requirements of agent evaluation.

Latitude

Latitude is the platform I built, so you should weight my assessment accordingly — but I'll try to be precise about where it's strong and where it's still maturing.

The core architectural bet in Latitude is what we call the Reliability Loop: production traces flow in → domain experts annotate failure cases → GEPA auto-generates evals from those annotations → evals run continuously and catch regressions. The key word is "loop" — each component feeds the next, and the system improves automatically as the team annotates more cases.

For multi-turn agent tracing, Latitude captures full session objects with every tool call, sub-agent invocation, memory read, and reasoning step linked into a single trace. Tool calls are first-class spans with their own inputs, outputs, and error states — not metadata attached to an LLM completion. This means failures that originate in tool calls (which, in our experience, are responsible for a substantial proportion of agent failures) are surfaced directly rather than buried in downstream LLM behavior.

The issue tracking system is what differentiates Latitude most clearly from other tools in this list. When you observe a failure pattern in production, you don't just log it — you track it as a named issue with a lifecycle: first observation, root cause investigation, fix deployment, regression verification. Issues are grouped by frequency and severity, giving you a prioritized queue rather than a raw stream of anomalies.

Here's what agent-specific tracing looks like in practice with Latitude's SDK:

from latitude_sdk import Latitude, TraceOptions

client = Latitude(api_key="your-key")

# Full agent session trace — all steps linked automatically
with client.trace_session(session_id="user-123-session-456") as session:

    # Tool calls captured as first-class spans
    with session.span("retrieve_context", span_type="tool_call") as span:
        context = retrieval_api.query(user_query)
        span.set_output(context)
        span.set_metadata({"source": "vector_db", "results": len(context)})

    # LLM call with full context captured
    with session.span("reason", span_type="llm_call") as span:
        response = llm.complete(
            system=agent_prompt,
            context=context,
            history=session.history
        )
        span.set_input({"context_length": len(context), "history_turns": len(session.history)})
        span.set_output(response.content)

    # Issue flagging from annotation queue
    session.flag_for_review(
        reason="tool_returned_empty",
        severity="high",
        metadata={"retrieval_results": 0, "user_query": user_query}
    )

from latitude_sdk import Latitude, TraceOptions

client = Latitude(api_key="your-key")

# Full agent session trace — all steps linked automatically
with client.trace_session(session_id="user-123-session-456") as session:

    # Tool calls captured as first-class spans
    with session.span("retrieve_context", span_type="tool_call") as span:
        context = retrieval_api.query(user_query)
        span.set_output(context)
        span.set_metadata({"source": "vector_db", "results": len(context)})

    # LLM call with full context captured
    with session.span("reason", span_type="llm_call") as span:
        response = llm.complete(
            system=agent_prompt,
            context=context,
            history=session.history
        )
        span.set_input({"context_length": len(context), "history_turns": len(session.history)})
        span.set_output(response.content)

    # Issue flagging from annotation queue
    session.flag_for_review(
        reason="tool_returned_empty",
        severity="high",
        metadata={"retrieval_results": 0, "user_query": user_query}
    )

from latitude_sdk import Latitude, TraceOptions

client = Latitude(api_key="your-key")

# Full agent session trace — all steps linked automatically
with client.trace_session(session_id="user-123-session-456") as session:

    # Tool calls captured as first-class spans
    with session.span("retrieve_context", span_type="tool_call") as span:
        context = retrieval_api.query(user_query)
        span.set_output(context)
        span.set_metadata({"source": "vector_db", "results": len(context)})

    # LLM call with full context captured
    with session.span("reason", span_type="llm_call") as span:
        response = llm.complete(
            system=agent_prompt,
            context=context,
            history=session.history
        )
        span.set_input({"context_length": len(context), "history_turns": len(session.history)})
        span.set_output(response.content)

    # Issue flagging from annotation queue
    session.flag_for_review(
        reason="tool_returned_empty",
        severity="high",
        metadata={"retrieval_results": 0, "user_query": user_query}
    )

What's happening here isn't just logging. Every span is linked to the parent session, which means you can query: "Show me all sessions where a tool call returned empty and the agent's next LLM call produced a hallucination." That cross-span, cross-turn analysis is what makes failure patterns discoverable rather than hidden.

Where Latitude is still maturing: Integration breadth. LangSmith and Langfuse have been around longer and have deeper framework integrations. If you're using a less common framework or need very specific integration customization, you may need to do more manual instrumentation with Latitude than with established alternatives. This is improving rapidly, but worth acknowledging.

Best for: Engineering teams running agents in production who need to close the loop between observability and quality — not just see failures, but build a systematic process for preventing them.

LangSmith

LangSmith is the best observability tool for LangChain and LangGraph applications, full stop. If your agents are built on LangChain or LangGraph, LangSmith's native integration means you get full tracing — including agent steps, tool calls, and intermediate reasoning — with zero additional instrumentation. The framework generates the trace structure automatically.

Where LangSmith's agent evaluation support shows its origins as an LLM-first tool: issue discovery is manual (you curate a dataset from observed failures rather than having the platform surface failure clusters for you), eval generation requires human authoring or manual prompting of an LLM to create evaluation criteria, and failure mode clustering doesn't exist as a native feature. You can build these workflows on top of LangSmith using its dataset and annotation primitives, but it's engineering work rather than a built-in capability.

LangSmith is excellent at what it was designed for: tracing LangChain apps, curating evaluation datasets, running human review workflows, and comparing prompt versions against those datasets. For a pure LLM workflow, this is a complete and well-designed stack. For a complex agent system where you're trying to discover unknown failure patterns and auto-generate evals from production data, you'll find yourself building scaffolding that other tools provide out of the box.

Best for: Teams primarily using LangChain or LangGraph who want the best possible integration depth and are willing to build manual eval workflows on top of a solid observability foundation.

Braintrust

Braintrust takes an eval-first philosophy, and it executes that philosophy extremely well. The core Braintrust workflow — define an eval dataset, score it with automated criteria, compare scores across model versions and prompt changes, review diffs to make shipping decisions — is one of the most polished in the industry. If you've built an eval culture on your team and want a dedicated platform to run eval experiments, Braintrust is genuinely excellent.

For agent evaluation specifically, Braintrust's session grouping handles multi-turn tracing, and tool calls can be logged as spans. But the evaluation model is centered on predefined criteria: you define what "good" means in advance, then score against that definition. This works well for known failure modes and regression testing. It's structurally limited for discovering unknown failure patterns — the eval-first model requires you to know what you're looking for before you look for it.

Braintrust also doesn't have automatic eval generation from production annotations. You can run LLM-as-judge scoring over logged data, but translating a newly-discovered failure pattern into a persistent, versioned regression test requires manual eval authoring. For teams shipping fast, this friction adds up.

Best for: Engineering teams who have defined their quality criteria clearly and want a sophisticated platform for running structured eval experiments against those criteria. Excellent CI/CD integration for regression testing on known failure modes.

Langfuse

Langfuse is the most popular open-source LLM observability platform, and for teams with data residency requirements or preferences for self-hosted infrastructure, it's often the default choice. Its session threading groups multi-turn conversations, its annotation workflows support human review, and it integrates with essentially every major LLM framework through lightweight SDK wrapping.

For agent evaluation specifically, Langfuse's core gap is the same as LangSmith's and Braintrust's: evaluation is driven by human-curated datasets rather than platform-surfaced issue discovery. The platform shows you traces; it doesn't tell you which traces represent systematic failure patterns worth tracking. Building that capability requires either additional tooling or significant manual analysis.

Langfuse's eval capabilities have grown significantly (LLM-as-judge, custom scoring, dataset management), but they're applied to user-defined evaluation surfaces rather than dynamically discovered failure patterns. For teams running complex agents with novel failure modes, this means your eval coverage is bounded by what you've thought to look for.

Best for: Teams needing open-source, self-hosted observability with a complete annotation and evaluation stack. The best choice when data sovereignty is a hard requirement or when you need a community-supported, extensible platform.

Deepchecks

Deepchecks comes from the ML testing world — its original product was a data validation library for classical ML, and its LLM evaluation features reflect that origin. It's strongest on data quality monitoring: detecting input distribution shift, validating that retrieval results meet quality thresholds, and monitoring for feature drift over time. Its test suite generation capabilities are more mature than most LLM-native tools for this specific category.

For full agent observability, Deepchecks's multi-turn tracing is partial, and its tool use observability requires more manual setup than purpose-built agent tools. Where it shines is as a complementary layer: if you're running a RAG agent and need to monitor whether your retrieval quality is degrading over time, or if you need comprehensive data validation before traces reach your LLM, Deepchecks fills that niche well.

Best for: Teams with existing ML data validation workflows who want to extend quality monitoring into their LLM/agent pipeline. Particularly strong for RAG applications where retrieval data quality is a primary concern.

Selection Framework: When to Use Which Tool

The question isn't "which tool is best" — it's "which tool fits your specific system and team." Here's how I think about the decision:

Use a traditional LLM eval tool (LangSmith, Braintrust, Langfuse) when:

• Your system is primarily stateless LLM calls. If your application is a chatbot with simple retrieval, a document summarizer, or a classification pipeline, traditional eval tools handle this well. The multi-turn and tool use complexity that breaks traditional evals isn't present in your architecture.

• You're early in development with undefined production behavior. Before you have production traffic, you're writing evals against hypothetical failure modes anyway. Braintrust's structured eval experiment workflow or Langfuse's annotation workflow both work well for this phase. Start here, then reassess when you have real production data.

• You're deep in the LangChain ecosystem. LangSmith's integration depth with LangChain is genuinely hard to replicate. If LangChain is your primary development framework, the native tracing quality is a meaningful advantage.

• You need self-hosted deployment with zero vendor dependency. Langfuse is the clear choice here. Open-source, active community, self-hosted, GDPR-compliant by default.

Use an agent-specific platform (Latitude) when:

• You're running multi-step agents in production with real user traffic. If your agent makes 5+ tool calls per session, spawns sub-agents, or manages state across turns, you're in territory where traditional eval tools will give you false confidence.

• Your failure modes are unknown. If you're regularly surprised by production failures — you fix one issue and a new category of failure appears — you need issue discovery, not just regression testing. Platforms that surface failure patterns you haven't seen before are what close this loop.

• You want evals that grow with production data. Teams using GEPA-based eval generation report that their eval coverage expands automatically as the team annotates production cases, without manual eval writing. If eval velocity is your bottleneck, this changes the math.

• You have domain experts who define quality but aren't writing eval code. Latitude's annotation queues are designed for domain experts — the people who know what "correct" looks like — not just engineers. If your quality definition lives in the heads of product managers, legal reviewers, or subject matter experts, this workflow captures their judgment directly.

The honest hybrid reality:

Most production teams end up with both. They use LangSmith or Langfuse for framework-level tracing and basic logging (especially during development), and add an agent-specific observability layer when they hit production at scale. The two stacks aren't mutually exclusive — they solve adjacent problems.

Case Example: The Failure That Synthetic Benchmarks Miss

Here's a real class of failure we've seen repeatedly with production agents that illustrates why issue discovery matters.

A customer support agent for a SaaS product has a tool call that retrieves the user's current subscription plan. In 4% of sessions, this API call returns a cached response from a previous session — stale data that reflects a plan the user downgraded from two weeks ago.

The agent, receiving what looks like valid plan data, proceeds to offer features that aren't available on the user's current plan. The final output — a confident, helpful, well-formatted response — passes every synthetic benchmark in the team's eval suite. LLM-as-judge scores it highly on helpfulness, accuracy (given its context), and tone.

The failure is invisible to traditional eval tools for a simple reason: no one wrote an eval for "did the agent use stale subscription data?" That failure mode wasn't anticipated during eval design.

Here's how Latitude's issue tracking surfaces this:

1. Production trace ingestion. Every agent session — including tool inputs and outputs — flows into Latitude. The retrieval call's response time (120ms vs. the normal 340ms for a fresh API call) is logged as a trace attribute.

2. Anomaly surfacing. Latitude's clustering groups traces by behavior pattern. A cluster emerges: "Sessions where account retrieval was fast AND the agent offered premium features to non-premium users." The platform surfaces this cluster to the annotation queue.

3. Human annotation. A domain expert reviews 12 examples from this cluster. They confirm: these are failures. The agent is citing plan features the user doesn't have.

4. GEPA eval generation. Based on the annotation, GEPA generates an eval: "Flag sessions where the agent asserts premium feature availability when the user's subscription tier is Basic." This eval didn't exist before the production pattern was observed. It now runs continuously against every new session.

5. Regression prevention. The team identifies the caching bug, deploys a fix, and the eval confirms the fix holds across subsequent production traffic.

The critical step here is step 2 — the platform surfaced a failure pattern no one knew to look for. All the downstream steps (annotation, eval generation, regression testing) could potentially be replicated in LangSmith or Braintrust, but they require you to already know what you're looking for. Issue discovery is what makes unknown failures findable.

This is the direct answer to Hamel's concern about auto-generated evals: GEPA doesn't replace human judgment about what constitutes a failure. The domain expert's annotation in step 3 is what tells the system "this is wrong." What GEPA automates is the translation of that human judgment into a runnable, persistent regression test — the mechanical work that sits between "I know this is a failure" and "I have a test that catches it."

Conclusion

Evaluating AI agents is a different problem than evaluating LLM completions. The tools that work well for the latter were not designed for the former, and applying them to agent systems produces a predictable result: eval suites that pass while production keeps failing.

The five criteria outlined here — multi-turn tracing, tool use observability, issue discovery, auto-generated evals from production data, and failure mode clustering — represent the dimensions where agent evaluation diverges from LLM evaluation. Before choosing an eval platform for an agent system, I'd encourage you to evaluate each tool against these criteria for your specific use case.

LangSmith, Braintrust, and Langfuse are genuinely excellent tools — for the workflows they were designed for. If you're building complex agents that manage state across turns, make dozens of tool calls per session, and fail in ways you haven't anticipated, you need an evaluation approach designed around that architecture.

The goal isn't perfect evals on day one. It's building a system that gets better as your agents improve — one where production failures become future regression tests automatically, and where your eval coverage grows with your understanding of your domain. That's the loop worth closing.

Frequently Asked Questions

Which tool auto-generates evals aligned to product requirements from production data?

Latitude is the only platform in 2026 that auto-generates evaluations from annotated production failures using the GEPA (Generative Eval from Production Annotations) algorithm. Domain experts annotate production traces through prioritized queues; GEPA converts those annotations into runnable evaluation criteria automatically and refines them as more annotations accumulate. The eval library grows from real production failures specific to your product — not from generic benchmarks. LangSmith, Braintrust, and Langfuse all require manual eval creation from production observations; none auto-generate evals from annotations.

Why do LLM evaluation tools fall short for AI agent evaluation?

LLM evaluation tools score individual prompt-response pairs — correctness, relevance, fluency. This approach has four structural gaps for agents: (1) It misses failures that emerge across turns, not at individual responses. (2) It can't detect tool use misinterpretation — when a tool returns a valid response the agent interprets incorrectly, corrupting downstream reasoning. (3) It can't assess goal-level outcomes — whether the agent accomplished the user's actual intent over the full session. (4) It can't model compounding errors in multi-step workflows where per-step reliability of 95% yields only 36% end-to-end success over 20 steps.

What is product-aligned evaluation for AI agents?

Product-aligned evaluation uses quality criteria derived from your specific product's actual failure patterns — not generic benchmarks. Generic benchmarks test hypothetical failure scenarios based on the developer's assumptions. Product-aligned evals grow from real production failures annotated by domain experts who understand what your users actually need. The gap matters because production always generates failure modes the team didn't anticipate. Latitude's GEPA algorithm creates product-aligned evaluations automatically from annotated production failures, ensuring the eval suite stays aligned with the actual distribution of production failures rather than the team's prior assumptions.

Latitude is an observability and quality platform for production AI agents. Start your 30-day free trial →

Related Blog Posts

• Top 5 AI Agent Evaluation Tools in 2026

• Best AI Evaluation Tools for Agents in 2026: Agent-First vs LLM-Only Platforms

• Agent Evaluation Tools Compared: Why Generic Benchmarks Fail Production AI (2026)

• Best AI Evaluation Tools for Agents in Production (2026)