▣

If your AI agent is in production, some fraction of its conversations are going wrong right now. Users rephrasing the same question three times. The model quietly refusing. A tool call erroring out and the agent bluffing past it. The instinct is to train a classifier for each of these. Don’t. Bespoke per-mode classifiers are the most expensive and most brittle way to solve this, and they miss the failures you haven’t named yet.

Here’s the short version. You can auto-flag problematic conversations with no trained classifier at all, using a funnel of three layers you already have the pieces for. Cheap deterministic signals catch the obvious cases for free. An LLM-as-a-judge labels the ambiguous middle across every category at once. Periodic embedding-and-clustering sweeps surface the failure modes your rubric never mentioned. This guide walks through all three, with a paste-ready judge implementation and the validation step most teams skip.

Why “a classifier per failure mode” breaks down

Training a supervised classifier for each failure type feels rigorous. In production it fails four ways:

  • It only catches what you already named. A classifier for “refusal” catches refusals. It’s blind to the failure mode you haven’t thought of yet, which is exactly where the risk lives. Anthropic built Clio on this premise: benchmarks test predefined behaviors, but real-world risk shows up as patterns that only emerge from usage.
  • Concept drift eats it. A classifier encodes the traffic distribution at build time. Production traffic is non-stationary; prompts, users, and your own system prompt all move. You end up on a detect-drift, relabel, retrain, redeploy treadmill, and you pay for it per failure type.
  • The positive class is tiny. Problematic conversations are a minority of traffic. Rare-event labeling is precisely where trained classifiers are most brittle and where you need the most hand-labeled data.
  • Labeling is slow and expensive. Human annotation being “exceptionally slow and costly” is the founding motivation of the entire LLM-judge literature (Zheng et al., 2023).

Classifiers aren’t bad. The problem is that a bespoke per-mode classifier doesn’t amortize. You want methods that need no per-mode labels, generalize to modes you didn’t specify, and degrade gracefully when the distribution shifts. That points at a funnel.

The funnel: three layers, no per-mode training

                every conversation
                        │
         ┌──────────────▼───────────────┐
         │  1. Deterministic checks      │  ~free, milliseconds
         │  (errors, empty, bad schema)  │  catch the obvious, 100%
         └──────────────┬───────────────┘
                        │ ambiguous middle
         ┌──────────────▼───────────────┐
         │  2. LLM-as-a-judge            │  1 call/convo
         │  (all categories at once)     │  label the gray zone
         └──────────────┬───────────────┘
                        │ periodic batch
         ┌──────────────▼───────────────┐
         │  3. Embedding + clustering    │  cheap, offline
         │  (find UNNAMED failure modes) │  the unknown-unknowns
         └───────────────────────────────┘

Each layer does a job the others can’t, and none of them require training a classifier. Here’s how they divide the work:

Layer Cost Coverage Catches Misses
1. Deterministic signals ~free, milliseconds 100% of traffic Errors, empty/truncated output, bad schema, tool failures, rage-repeat Anything you can’t write a rule for
2. LLM-as-a-judge 1 LLM call per conversation (sample the cheap tier) The ambiguous middle Every category you’ve defined, in one pass Failure modes you never named
3. Embedding + clustering Cheap offline batch Periodic sweep over history Unknown-unknowns, new patterns in the noise Nothing labeled in real time

Layer 1: deterministic signals (the cheap pre-filter)

Some problems are detectable with a rule, at effectively zero cost and zero latency. Run these on every conversation before you spend a single LLM call:

  • User repetition and rephrasing (“rage prompting”), the same intent restated 3+ times.
  • Escalation cues: ALL-CAPS bursts, “this is broken”, “talk to a human”.
  • Tool errors: non-200 codes, exceptions, timeouts on tool calls.
  • Empty or truncated responses: zero-length or cut-off completions.
  • Malformed structured output: JSON that doesn’t parse or fails your schema.
  • Broken prompt caching: a long multi-turn trace where caching is on but the cache-hit rate is far below what it should be, which is a strong signal of a token-wasting bug.
  • Latency spikes and refusal boilerplate (“I’m not able to help with that”).

These are the highest-precision, most interpretable checks you’ll have, and they’re basically free. Their limit is obvious. They only catch what you can write a rule for. So don’t stop here, use them to route. Anything a signal fires on goes straight to the “flag” pile, and everything ambiguous falls through to Layer 2.

Layer 2: LLM-as-a-judge (the ambiguous middle)

For everything the rules can’t decide, prompt a capable LLM to read the conversation and label it. This is the workhorse. Done well, LLM judges hit over 80% agreement with human annotators, which is the same level humans agree with each other (Zheng et al., 2023). The trick is judging all your categories in one pass, so adding a new category costs a line of prompt instead of a new training run.

Best practice, distilled from the judge literature:

  • Decompose by category, each with a one-line criterion and an example.
  • Reason before labeling (chain-of-thought). G-Eval showed CoT plus form-filling lifts correlation with humans substantially (Liu et al., 2023).
  • Emit structured JSON and enforce it with structured outputs.
  • Calibrate against a human-labeled seed set (see validation, below).

Name the known biases honestly so you can design around them. Judges show position bias (GPT-4 was only ~65% consistent when the order of two answers was swapped, Zheng et al.), verbosity bias (they favor longer answers even when the longer one isn’t better, Ye et al., 2024), and self-preference (they rate their own model family higher). Two defenses matter here. Put an explicit “ignore response length” instruction in the prompt, and use a different model family as the judge than the one running your agent. If you can’t call a frontier model per conversation, open judges like Prometheus 2 self-host and land around 0.6 to 0.7 Pearson correlation with GPT-4.

Here’s a paste-ready judge prompt and output schema, tool-agnostic:

You are a strict QA auditor for an AI assistant. Judge ONLY the content
shown. IGNORE response length; longer is not better.

For the conversation below, decide whether each category is PRESENT:
- frustration: user shows irritation, repeats/rephrases, escalates
- refusal: assistant declines a reasonable in-scope request
- jailbreak_attempt: user tries to bypass safety/system instructions
- tool_error: a tool call failed or returned an error the assistant ignored
- empty_or_silent_wrong: empty, truncated, or confidently wrong answer
- off_topic: assistant drifts from the user's actual request
- policy_violation: content violates your usage policy

First write 2-3 sentences of reasoning. Then output JSON only.
{
  "reasoning": "string",
  "problematic": true,
  "categories": {
    "frustration": false,
    "refusal": false,
    "jailbreak_attempt": false,
    "tool_error": true,
    "empty_or_silent_wrong": false,
    "off_topic": false,
    "policy_violation": false
  },
  "severity": "low | med | high",
  "confidence": 0.0
}

problematic is just the OR of the category booleans. Run the judge in a batch over yesterday’s logs, or stream it on conversation-close. Auto-escalate anything with severity: high or any deterministic signal from Layer 1. Sample 5 to 10% of the low and medium verdicts for a drift check, and route low-confidence cases to humans more aggressively. If you’re over-flagging, tighten the category wording rather than filtering the output after the fact.

Layer 3: clustering for the unknown-unknowns

Layers 1 and 2 only find problems you thought to describe. To catch the ones you didn’t, periodically embed your conversations, cluster them by meaning, and inspect the anomalous or low-quality clusters. This is the Clio pattern: summarize, embed, cluster, analyze, with no predefined labels (Clio, 2024). It won’t hand you a clean label, but it surfaces structure, a knot of sessions all failing the same new way, which becomes your next judge category. Run it as a cheap offline batch, well outside the request path.

The validation step nobody skips twice

An LLM judge you haven’t validated is a random number generator with good grammar. Before you trust a single verdict:

  1. Build a golden set of 100 to 200 conversations, oversampling the flagged ones.
  2. Have two humans label them independently.
  3. Check inter-annotator agreement first. If your two humans disagree (Cohen’s κ below ~0.6), the problem is your rubric rather than your judge, so fix the definitions before blaming the model (Es et al., 2024).
  4. Now run the judge and compute precision / recall / F1 plus Cohen’s κ against the humans, per category.
  5. Ship when judge-vs-human agreement is about as high as human-vs-human. Re-sample monthly, and any time you change the model, the system prompt, or the judge prompt.

How we do this at Latitude

The funnel above is the architecture, but wiring it up yourself for every category is exactly the work most teams don’t finish. It’s also, more or less, what Latitude does out of the box, so it’s worth being concrete about where the general pattern meets a real implementation, including the seams.

Latitude’s version of this is called flaggers, and the settings screen groups them along the funnel’s fault line, deterministic versus LLM-based. That grouping is the most important design decision to copy:

  • Response validity is the deterministic Layer 1: Empty response, Tool call errors, Output schema validation. Each one says “Free · Runs on 100% of eligible traces”, and, importantly, “without calling an LLM.” No sampling, no cost, every trace.
  • Cost & efficiency is also deterministic and free at 100%. Low cache hit rate is the interesting one here. It flags large multi-turn traces where caching is active but under 30% of input tokens were served from cache. That check exists because a real customer’s randomized tool-payload ordering silently broke their prompt caching and produced a roughly 7× token overcharge, and a pure-rule check caught it with zero LLM spend.
  • Agent behavior is the LLM Layer 2, watching the agent’s own output: Refusal, Laziness, Forgetting, Thrashing (the agent cycling between tools without progress). Each shows “30 credits per scan · runs on 10% of eligible traces” by default, with a slider.
  • User-side signals is the LLM layer watching user behavior: User Frustration, Jailbreaking, NSFW. Same 10%-default, credit-metered sampling.

That split is the whole point. The deterministic groups run free on 100% of traffic, and the LLM groups sample (10% by default) because judging every trace with a model costs money. You can dial any LLM flagger up toward 100% when you’d rather pay for coverage than miss the traces that didn’t get sampled. It’s a real tradeoff, and the UI puts the cost right next to the slider so you can see it. There’s also a set of use-case presets (Support agent, Coding agent, Sales agent, Tool-workflow, Knowledge-base, Structured-extraction, Safety agent) that toggle the sensible subset for you, so you’re not reasoning about ten switches from a cold start.

Here’s the settings screen itself. The deterministic groups run free on everything, and the LLM-based groups carry the per-scan cost and sampling slider.

The deterministic Layer 1, free and always on:

Latitude flaggers settings: use-case presets at the top, then a Response validity group of deterministic checks (Empty response, Tool call errors, Output schema validation), each marked Free and running on 100% of eligible traces.

Latitude flaggers settings, Cost and efficiency group: a single deterministic Low cache hit rate check that flags large multi-turn traces where under 30% of input tokens came from cache, marked Free and running on 100% of eligible traces.

And the LLM-based Layer 2, sampling at 10% by default with the per-scan cost shown inline:

Latitude flaggers settings, Agent behavior group: Refusal, Laziness, Forgetting, and Thrashing, each an LLM-based check at 10% sampling and 30 credits per scan.

Latitude flaggers settings, User-side signals group: User Frustration, Jailbreaking, and NSFW, each an LLM-based check at 10% sampling and 30 credits per scan.

(User Frustration is deep enough on its own that we wrote a separate guide to detecting it. Most frustration is silent and polite, which is exactly why the deterministic rules miss it and it lives in the LLM group.)

Semantic search is the part we’d tell you to lean on hardest. Alongside the flaggers, Latitude embeds every user and assistant message (Voyage’s voyage-4-large, content-addressed by hash so identical text is embedded once) and lets you search by meaning across 100% of traffic, no sampling. Long conversations are chunked into individual turns to lift accuracy on meta-meaning queries. As Gerard on our team puts it, “think of semantic search as a child reading the conversations”: it reads what a turn means, rather than the keywords. Quote a string for exact match, leave it bare for semantic, and combine them, so user frustrated "billing" finds semantic frustration only in traces that literally mention billing.

Layer 3 is Behaviours. Latitude categorizes 100% of traces by user intent and assistant outcome using clustered embedding vectors, and the clustering isn’t static. A live taxonomy pipeline does online assignment (a session joins the nearest active cluster, or lands in a noise floor) plus offline “gardening” that births new clusters out of that noise, merges near-duplicates, and retires dead ones. That’s the unknown-unknowns layer running continuously instead of as an occasional sweep.

Here’s what that looks like in practice. A semantic search for “user is frustrated” surfaces a conversation the funnel flagged, with the user frustration signal attached and the agent’s transfer_to_human_agents call right below it:

Latitude Sessions view: a semantic search for ‘user is frustrated’ surfaces a conversation where the user snaps at a support agent, auto-flagged with a ‘user frustration’ signal.

The deterministic flaggers behind that view run on 100% of traffic, so nothing has to be pre-selected to get the free checks. Every conversation is screened as it lands, and the LLM layers sample on top of that.

It’s worth naming the seams honestly. The flaggers are still rule- and rubric-bounded, so they only catch the categories you’ve defined, while semantic search and Behaviours are what catch the modes you haven’t named. Internally we actually consider the frustration flagger semi-superseded by the semantic and Behaviours layer, because meaning-based retrieval catches frustration moments that don’t trip a rule at all. If you build this funnel yourself, that’s the tradeoff to plan for: the deterministic and judge layers are only as good as your category list, so you need the clustering layer running underneath them. If you don’t want to build it, that’s roughly what you get out of the box.

Where all of this lands is the difference between flagging and fixing. Once a failure mode is a named, tracked issue rather than a scattered set of flagged conversations, you can turn it into a regression dataset, auto-generate an eval from it, and, with the coding agent connected over Latitude’s MCP server, go from a detected issue to an opened PR without a human writing the first draft. A concrete one from our own dogfooding: a Thrashing flag fired on an agent that called get_weather three times in a row with the same empty arguments, each time getting “location is required” back and retrying identically. That became a tracked issue with sample traces attached, then a GitHub issue assigned to an engineer, then the fix. Flagging is the first layer of that loop rather than the whole thing. But you can’t close a loop you never opened, which is why getting the flagging layer right is worth this much attention.

FAQ

Won’t the judge hallucinate its judgments? It can, which is why you calibrate it against a human-labeled golden set, measure Cohen’s κ, and re-sample monthly. An unvalidated judge is not trustworthy; a validated one reaches human-level agreement.

Do I need a GPT-4-class model to judge? Frontier models are the most reliable judges, but open models like Prometheus 2 self-host and correlate 0.6 to 0.7 with GPT-4. Whatever you pick, use a different model family than your production agent to avoid self-preference bias.

How do I catch failure modes that aren’t in my rubric? That’s Layer 3: embed and cluster your conversations periodically, inspect the anomalous clusters, and promote new patterns into judge categories. Treat it as an ongoing loop rather than a one-time setup.

Isn’t judging every conversation too expensive? That’s what the funnel is for. Deterministic signals are free and handle the obvious cases; you only spend LLM calls on the ambiguous middle, you can sample the low/med tier, and you can batch it overnight against a cheaper model.

How is this different from a moderation or safety endpoint? Moderation endpoints score a fixed set of safety categories. “Problematic” is broader, product-specific, and evolving. A confused user or a silently-wrong answer isn’t unsafe, but it’s still a failure you want flagged.

How do I stop over-flagging? Tighten the category definitions in the judge prompt, add the explicit “ignore length” instruction, and gate on severity and confidence, instead of papering over it with a post-filter. Watch especially for a category that keeps firing on things outside your agent’s control. A support bot will see “user frustration” that’s about the product rather than the agent, and it’ll pile up as noise. The fix is to give the judge context about what your agent is and isn’t responsible for, and to turn off any category you find yourself ignoring over and over.

Sources: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023, arXiv:2306.05685); G-Eval (Liu et al., 2023, arXiv:2303.16634); Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (Ye et al., 2024, arXiv:2410.02736); Clio: Privacy-Preserving Insights into Real-World AI Use (Anthropic, 2024, arXiv:2412.13678); RAGAS / evaluation-agreement work (Es et al., 2024, arXiv:2406.12624); Prometheus 2 (prometheus-eval).