Do you need evals if you already monitor your AI agent?
Yes. Monitoring tells you what happened with real users and reveals failures you did not anticipate. Evals turn a defined quality expectation into a repeatable score or test, so you can check a change before release and keep watching the same behavior afterward.
An online eval can be part of monitoring. An offline eval runs against a controlled dataset before a prompt, model, tool, or code change reaches production. The useful system connects both: production failures become regression cases, and the same quality definition stays active after the fix ships.
What is the difference between monitoring and evals?
Monitoring is the evidence and discovery layer. It captures traces, model and prompt versions, tool calls, latency, cost, user feedback, and outcomes from real traffic. Its strength is exposure to reality, including new failure modes and distribution shifts absent from a test suite.
An eval is explicit grading logic. It takes an input, output, trace, or outcome and decides whether a defined criterion was met. Anthropic’s engineering guide defines an eval as a test that gives an AI system an input and applies grading logic to the output. That definition covers a schema check, a task-completion judge, a human rubric, and a regression suite.
| Dimension | Monitoring | Evals | Online evals | Offline evals |
|---|---|---|---|---|
| Main question | What happened with real users? | Did the system meet a defined criterion? | Does this live trace match the criterion? | Does this candidate version meet expectations? |
| Data | Production traces, errors, feedback, and metadata | Inputs, outputs, traces, or outcomes | Production traces, usually without reference answers | Curated cases, often with expected outputs |
| Timing | During and after production use | Whenever grading logic runs | As production work completes or in batches | During development, CI, or release review |
| Output | Evidence, trends, incidents, and unknown failures | A score, label, or pass/fail result | A detected production behavior | A comparison or release gate |
| Main blind spot | Reactive and noisy without quality definitions | Only checks criteria that were defined | Sampling and grader error | Misses failures absent from the suite |
Anthropic summarizes the division well: effective teams combine automated evals for fast iteration, production monitoring for ground truth, and periodic human review for calibration.
Do online evals count as monitoring?
Yes, when an evaluator scores production traces. The terminology overlaps because monitoring names the broader operational activity, while online evaluation names one scoring method inside it.
A trace can be useful without receiving a quality score. An engineer may inspect the tool sequence, compare a model version, or follow an exception through several spans. An online evaluator adds a repeatable question such as:
- Did the agent select the correct tool?
- Did it respect the refund policy?
- Did it resolve the user’s request?
- Did the final answer remain grounded in retrieved context?
Official LangSmith documentation makes the data boundary clear. Reference-free evaluators can work online and offline. Reference-based evaluators need an expected output, which production traces usually lack, so they normally belong in an offline dataset.
How do monitoring and evals form one closed loop?
The minimum useful loop has seven steps:
- Capture the production trace. Preserve the model, prompt version, tools, inputs, outputs, and relevant metadata.
- Detect and name the failure. Combine repeated examples under a stable behavioral definition.
- Label representative cases. Keep clear matches, boundary cases, and the expected behavior.
- Create an online check. Use a reference-free detector when the behavior can be recognized in live traffic.
- Promote real traces into an offline dataset. Preserve the input, expected outcome, metadata, and failure context.
- Gate the change. Replay the dataset after a prompt, model, tool, or code change.
- Watch production recurrence. Keep the detector active and reopen the issue if the behavior returns.
This loop prevents two common gaps. A monitoring-only team repeatedly discovers the same class of failure. An eval-only team protects known cases but misses new behavior from real users.
The production-to-test step matters because an apparently sensible change can damage a specific task. Our guide to catching regressions after prompt or model updates covers that implementation in detail.
Which production failures should become regression cases?
Promote a failure when it has a stable behavioral definition, meaningful impact, representative evidence, and an expected outcome reviewers can judge consistently.
Strong candidates include:
- a recurring tool-selection or sequencing error
- a policy, safety, or compliance breach
- an agent declaring success after a failed tool call
- a supported workflow that breaks for one model, release, or user cohort
- a fixed issue likely to return as prompts, tools, and models change
Avoid filling a regression suite with duplicate traces or disputed examples. If experts cannot agree on the pass criteria, calibrate the rubric first. Include negative cases too, especially for refusal, safety, and policy checks, because an evaluator can become so broad that it blocks valid behavior.
Can the same evaluator run online and offline?
Sometimes. The deciding factor is whether it needs a reference answer.
| Grader | Best use | Online | Offline | Main limitation |
|---|---|---|---|---|
| Deterministic code or conditions | JSON validity, required fields, tool errors, state checks | Yes | Yes | Brittle for semantic variation |
| Reference-free LLM judge | Tone, resolution, policy adherence, groundedness | Yes, usually sampled | Yes | Cost, nondeterminism, and calibration |
| Reference-based evaluator | Correctness against a known answer or expected tool sequence | Rarely | Yes | Needs ground truth |
| Human review | Ambiguous or high-stakes judgment and judge calibration | Sampled | Yes | Slow and expensive |
Use deterministic checks first. They are cheap, explainable, and can cover every eligible trace. Use an LLM judge when the boundary depends on meaning across variable outputs. Calibrate that judge against human annotations and review disagreements instead of treating its score as truth.
What is the smallest credible setup?
For a team that already captures traces, the smallest credible setup is one self-contained package:
Start with Anthropic’s practical guidance of 20 to 50 simple tasks drawn from real failures, while treating that range as guidance rather than a universal minimum. Define explicit pass criteria, include positive and negative cases, version the prompts and models, use deterministic checks first, add one reference-free online evaluator for the recurring semantic behavior, run an offline regression dataset before relevant changes ship, and periodically calibrate evaluator disagreements with human review.
A small suite that protects the team’s highest-impact workflows is more useful than a large dataset full of vague or duplicated cases.
What does this look like on a real Signal?
This redacted Latitude product example tracks a retail agent mishandling return eligibility. It demonstrates detection, impact, sampling, and evaluator alignment. It does not show that this Signal triggered Agent Dispatch, produced a regression test, caused a code change, or opened a PR. The definition covers returns processed, promised, or refused without grounding the decision in order status, item identity, customer confirmation, and allowed refund methods.
The screenshot spans July 7 to July 20. It shows 12 occurrences across 12 sessions, representing 4% of sessions in that window, with six affected users. The evaluator ran at 20% sampling. The displayed alignment metric is 92%, but the artifact does not reveal the number of human annotations, class balance, precision, recall, or false-positive and false-negative rates. It should not be read as general evaluator accuracy.
The sampling number is part of the result. Latitude captures 100% of telemetry on ingest, but that does not mean every session receives an LLM-based judgment. Deterministic conditions can run broadly. Semantic judges cost a model call and are often sampled. Any reported rate should identify the eligible traffic, sampling policy, and time window.
At Latitude, the production object connecting this workflow is a Signal. It keeps the behavioral definition, matching traces, impact, trend, lifecycle, and evaluation together. A team or coding agent can promote selected production traces into dataset rows. An evaluator can be reused during regression testing when its required inputs are available, especially for reference-free checks.
Monitoring can also reveal emerging usage patterns before a team has named a failure. Our guide to finding how users actually use an AI agent explains that discovery layer, while the tool-call error guide shows a deterministic production failure that can become repeatable coverage.
How do you know the fix worked and will stay fixed?
Use two forms of evidence.
First, replay the known failure and the broader regression set before release. A passing result shows the candidate handles those cases under the test harness.
Second, keep the production detector active after deployment and compare the occurrence rate against a meaningful baseline. A declining rate supports real-world improvement. A count without its denominator can mislead when traffic changes.
Through Agent Dispatch, Latitude can hand a Signal’s context and sample traces to the team’s own coding agent through Agent Dispatch. The coding agent investigates the repository, implements the smallest correct fix, adds a regression test where appropriate, and can open a PR. Latitude provides the trigger and evidence. It does not edit or merge the code.
No single passing test proves a failure will stay fixed permanently. Staying fixed means running the regression suite after relevant changes and continuing to monitor the production pattern.
FAQ
Can monitoring replace evals if I inspect traces manually?
Manual trace review is useful for discovery and diagnosis, but it does not create repeatable release coverage. An eval preserves the quality criterion so the same behavior can be checked across many versions and cases.
Can evals replace production monitoring?
No. An offline suite covers cases the team already knows how to represent. Production monitoring exposes new user behavior, distribution shifts, tool failures, and edge cases absent from that suite.
Should every production trace receive an LLM judge?
No. Run deterministic checks broadly and sample expensive semantic judges according to traffic, risk, and cost. Always report the sampling policy with the result.
What is a regression eval?
A regression eval checks whether the agent still handles behavior it handled before. It protects known capabilities and fixed failures after prompts, models, tools, or code change.
How often should I update the eval dataset?
Add representative production failures as they appear, remove duplicates, and review whether the suite still matches current traffic. The important cadence is event-driven: every meaningful fixed failure should leave behind repeatable coverage.