How to Cluster AI Agent Conversations by Topic

To cluster AI agent conversations automatically, embed each completed session, group nearby vectors with a noise-aware clustering method, and label each cluster from representative sessions. Keep uncertain sessions unassigned and update the taxonomy as new traffic changes what users ask.

The hard part is maintaining useful topics. A one-time clustering notebook can produce attractive labels while mixing different user jobs, hiding small topics, and going stale after the next release.

What counts as a conversation topic?

A topic is a recurring user job or situation shared by several sessions. “Resetting an account password,” “checking an order,” and “asking for a refund after delivery” are useful topics. Words such as “account” or “order” are weak topics because they do not describe what the user was trying to do.

Separate topic from outcome. Two sessions can share the topic “change a subscription” while one resolves successfully and the other ends in a clarification loop. Keeping topic and outcome as separate dimensions lets you ask which topics generate frustration or abandonment.

What should you cluster: messages, turns, or full sessions?

Unit	What it captures	Where it fails	Best use
Single message	Local wording and intent	Loses multi-turn resolution and context	Search and routing of short requests
User/assistant turn	One local exchange	Splits a task across several records	Finding moments inside long sessions
Full session	The complete user job and outcome	Long sessions may contain several jobs	Product-level conversation topics
Task segment	One job within a session	Requires a reliable segmentation rule	Agents that handle several independent tasks per session

Use full sessions as the default topic unit. Preserve turn-level evidence so reviewers can see why a session joined a cluster. If agents routinely handle several unrelated jobs in one session, segment on an explicit task boundary or allow secondary topic evidence. Do not split on an arbitrary token count and assume the chunks are separate intents.

How should you represent a session?

Build a compact text representation before embedding. Include the user’s goal, the assistant’s meaningful actions, tool names when they define the workflow, and the final outcome. Remove boilerplate system prompts, repeated tool schemas, secrets, and rendering noise.

One practical representation is:

User goal: change delivery address after ordering
Agent actions: looked up order; requested verification; called update_address
Outcome: tool rejected request because shipment was locked
User response: asked for human support

This gives the embedding model the semantic shape of the session without letting a large tool schema dominate the vector.

Sentence-BERT showed why independent text embeddings are useful for clustering. In its 10,000-sentence example, pairwise BERT comparison required about 65 hours, while SBERT produced embeddings in about 5 seconds and cosine comparisons in about 0.01 seconds on the stated hardware.¹ Those figures are from a 2019 research setup. They illustrate the computational advantage of reusable embeddings rather than a current production benchmark.

Which clustering method should you use?

The BERTopic pipeline is a useful reference architecture. It creates document embeddings, optionally reduces dimensionality, clusters the vectors, and then generates interpretable topic representations with class-based TF-IDF.²

Method	Advantage	Limitation	Use it when
k-means	Fast and simple; supports centroid assignment	Forces every session into one of K clusters	You know a stable topic count and traffic is clean
HDBSCAN	Finds varying-density groups and leaves outliers as noise	Batch operation and parameter tuning are heavier	You are discovering topics and want honest uncertainty
Agglomerative clustering	Exposes a hierarchy at several cut levels	Expensive on large corpora	You need parent and child topics from a manageable batch
Online centroid assignment	Cheap assignment for each new session	Existing centroids cannot discover every new topic	You already have active clusters and need live updates

HDBSCAN is useful during discovery because unrelated sessions can remain outliers. The peer-reviewed software implementation describes hierarchical density-based clustering,³ and BERTopic uses it so documents that do not belong together are not forced into a topic.²

Do not tune for maximum coverage. A noise pool is a quality control. If every session joins a cluster, weak assignments can make the taxonomy look cleaner while making it less useful.

How do you label clusters without inventing vague names?

Generate labels after clustering. For each cluster:

1. Select sessions near the cluster center and several boundary cases.
2. Extract terms or phrases that distinguish this cluster from the rest.
3. Ask a model or reviewer for a short job-shaped label and one-sentence definition.
4. Check the label against random members and neighboring clusters.
5. Store positive and negative examples with the definition.

BERTopic’s class-based TF-IDF treats the documents in each cluster as a class and finds terms that distinguish that class from other clusters.² An LLM can improve the readability of the final label, but it should receive representative evidence. Asking a model to summarize the whole corpus in one prompt skips the grouping step and makes topic boundaries hard to audit.

Prefer “refund for a delivered item” over “refunds.” Prefer “agent cannot update a locked shipment” over “order issue.” The definition should tell a reviewer which sessions belong and which adjacent sessions should stay out.

How do you know the topics are useful?

Topic quality requires more than a coherence score. Review four properties:

Property	Check	Failure signal
Coherence	Do random sessions share the stated user job?	Label sounds plausible but members differ
Separation	Are neighboring clusters meaningfully different?	Two clusters describe the same job with different words
Stability	Does the topic persist across resamples or time windows?	Cluster vanishes when a small batch changes
Operational value	Does the topic support a product or quality decision?	Label is coherent but too broad to act on

Sample from the center, edge, and noise pool. Central examples can make any cluster look good. Boundary cases reveal whether two topics should merge or one broad topic should split.

Track coverage as a diagnostic instead of a target. A rising noise share may mean new user behavior, embedding drift, poor preprocessing, or a threshold that is too strict.

How do you maintain a live topic taxonomy?

Static topics decay because language and usage change. A systematic review of text-stream concept drift selected 48 papers published from 2018 through August 2024 and found that streaming methods must handle changing patterns, arrival speed, limited memory, and retention constraints.⁴

Use two loops:

1. Online assignment. Compare each new session with active cluster centroids. Assign it when similarity clears a validated threshold. Otherwise, send it to noise.
2. Offline gardening. Periodically cluster accumulated noise, create a topic when a coherent group has enough support, merge near-duplicates, split broad clusters, and retire inactive topics.

Keep stable topic IDs through label edits. Store a taxonomy version and assignment time. When clusters merge, preserve redirects from old IDs so historical trends remain interpretable.

Review topics after major releases. A new feature can create a real topic, while changed UI wording can shift embeddings without changing the underlying job.

How should topics connect to outcomes?

Topic counts tell you what users attempt. Join the assignment to outcomes so the taxonomy explains product quality.

For each topic, track:

• session count and share of traffic
• distinct users or tenants
• resolution and abandonment signals
• tool errors and repeated tool calls
• latency or cost when it affects the task
• user feedback
• trend by release, model, and language

This keeps “top topics” from becoming a popularity chart. A small topic with a 60% unresolved rate may deserve attention before the largest successful topic. That percentage only illustrates the prioritization logic. It is not a Latitude result.

How does Latitude build Behaviors?

Latitude’s Behaviors pipeline uses embeddings for topic discovery and keeps a live taxonomy. Manuel, one of the engineers working on the system, put the core plainly: “Behaviors part = embeddings and thats it.” Gerard described the product goal as categorizing “100% of traces by user intent and assistant outcomes using clustered embedding vectors.”

The implementation has two operating paths:

1. New sessions are assigned online to active clusters using centroid similarity.
2. Sessions that do not fit enter a noise floor.
3. Offline gardening creates topics from coherent noise, merges near-duplicates, and deprecates inactive clusters.
4. Centroids live in Postgres, while observations live in ClickHouse.

That split supports quick assignment and longer-horizon analysis. It also keeps uncertainty visible. Latitude does not need to force every new session into the closest existing topic.

Behaviors differs from semantic search. Search embeds user and assistant turns and retrieves sessions relevant to a query. Behaviors groups sessions into a maintained topic taxonomy. Search answers a question you already have. Behaviors can expose recurring usage you had not named yet.

Once a topic exists, teams can compare its trend and outcomes over time. A failure inside one topic can then become a Signal, which carries a lifecycle and regression monitoring.

In this example, Latitude groups sessions into topics such as Flight Booking Assistance and links the selected topic to 25 sessions, its recent activity, outcome moments, and the underlying conversations. Read how to find out how users use an AI agent and how to track recurring failures for those adjacent workflows.

What is a minimal implementation?

Start with one week of completed sessions and this pipeline:

1. Remove boilerplate and secrets.
2. Build one concise representation per session.
3. Generate embeddings and store the model version.
4. Run UMAP plus HDBSCAN, or begin with HDBSCAN on manageable data.
5. Label clusters from central and boundary examples.
6. Review a stratified sample and keep poor fits as noise.
7. Publish topic IDs, definitions, and examples.
8. Assign new sessions online and garden the noise on a fixed cadence.

Do not automate taxonomy changes without an audit trail. Store the prior label, merge or split reason, reviewer, and affected IDs.

FAQ

Can an LLM cluster conversations without embeddings?

An LLM can label small batches, but direct all-pairs comparison becomes expensive and inconsistent as the corpus grows. Embeddings create reusable vectors for similarity and clustering; an LLM can then label the resulting groups.

Should each conversation have exactly one topic?

Use one primary topic when product reporting needs clean counts. Preserve secondary evidence or task segments for genuinely multi-intent sessions. Forced multi-label output can become noisy without clear boundaries.

How many topics should I choose?

Avoid choosing a fixed count before inspecting the data unless the taxonomy is already known. Density-based clustering can discover a variable number and leave uncertain sessions as noise. Choose granularity through human review and operational usefulness.

What should happen to unclustered conversations?

Keep them in a noise pool. Review samples and periodically recluster the pool. It contains genuine outliers, weak representations, and candidates for new topics.

How often should topics be updated?

Base the cadence on traffic and product change. Assign sessions continuously, then garden often enough that the noise pool and duplicate topics do not distort decisions. Trigger an extra review after a major release or distribution shift.

Are automatically generated topics ground truth?

No. Clustering provides hypotheses about recurring behavior. Human review, stable definitions, and outcome analysis determine whether a topic is trustworthy enough for decisions.

References

Footnotes

1. Nils Reimers and Iryna Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, 2019. ↩

2. Maarten Grootendorst, BERTopic: Neural topic modeling with a class-based TF-IDF procedure, 2022. ↩ ↩² ↩³

3. Leland McInnes, John Healy, and Steve Astels, hdbscan: Hierarchical density based clustering, Journal of Open Source Software 2(11), 2017. ↩

4. Cristiano Mesquita Garcia et al., Concept Drift Adaptation in Text Stream Mining Settings: A Systematic Review, review coverage through August 2024. ↩