Start free

LlamaIndex integration

Observability for LlamaIndex applications

Trace LlamaIndex RAG pipelines and agents in production. Inspect retrieval, model, and tool execution, detect recurring failures, and turn real traces into evals.

Start tracing LlamaIndexView setup docs

ADD TRACING TO LLAMAINDEX

import os
import llama_index

from latitude_telemetry import Latitude, capture
from my_app.rag import query_engine

latitude = Latitude(
    api_key=os.environ["LATITUDE_API_KEY"],
    project=os.environ["LATITUDE_PROJECT_SLUG"],
    # One entry. Retrieval, embeddings, model calls,
    # tools, and workflow steps all come from it.
    instrumentations={"llamaindex": llama_index},
)

response = capture(
    "support-rag-query",
    lambda: query_engine.query(user_question),
    {"session_id": conversation_id, "user_id": user_id},
)

latitude.flush()  # needed in scripts and jobs that exit

Works with Latitude's LlamaIndex instrumentation or LlamaIndex's native instrumentation exported over OpenTelemetry. Trace RAG pipelines, agents, Workflows, and querying, plus indexing jobs you wrap.

What your LlamaIndex traces turn into

Complete RAG and agent executions you can inspect end to end, with recurring failures already grouped for you.

  • Complete query sessionsGroup multi-turn questions, retrieval, synthesis, and tool use by session id.
  • Retrieval, models, and toolsInspect query transforms, retrievers, rerankers, model calls, agent steps, and tool calls as nested spans, with the retrieved nodes and their scores on the retriever span.
  • Errors and weak answersSee failed spans, tool exceptions, empty responses, and recurring answer-quality failures in Signals.
  • Cost, latency, and your metadataFilter by model, token cost, duration, release, environment, and any index version or tenant metadata you attach.

Three steps to your first LlamaIndex trace

Initialize the SDK and wrap the query, workflow, or agent entrypoint that handles one request.

1

Install the SDK

One package. Python 3.11+. It installs alongside your existing llama-index version.

2

Instrument LlamaIndex

Pass the LlamaIndex module to Latitude at startup and execute the production query inside capture(). Your index and query engine stay unchanged.

3

Run one query

Open Traces. Inspect what the retriever returned, how context reached the model, and where the final answer was produced.

Send your first trace in minutes. Scripts, notebooks, indexing jobs, and CLIs should call latitude.flush() before exit. Long-running servers export in the background. Python integration guide →

Debug LlamaIndex failures in production

The retrieval, indexing, synthesis, and agent failures that are hard to diagnose from the final answer alone.

Retrieval returns the wrong context

Semantically similar chunks rank above the passage that contains the answer. Inspect the retrieved nodes and their scores on the retriever span, compare them with the reranker output, then group affected sessions into a Signal.

RAG traces →

Chunking and context overflow

Critical context is split across nodes, or too many chunks dilute the prompt and increase cost. Compare retrieval spans, token usage, and answer outcomes across sessions.

Search →

Index and embedding drift

A deploy queries an old index, mixes embedding configurations, or serves before ingestion finishes. Filter traces by release and index metadata to isolate the change.

Signals →

Synthesis and agent regressions

The retriever finds the right evidence, and then the model ignores it, a tool fails, or a workflow path changes after a prompt or model release. Resolved Signals reopen when the failures return.

Signal lifecycle →

Find recurring failures hidden across LlamaIndex sessions

Reading traces one at a time finds the failure you went looking for. Latitude surfaces the one costing you the most sessions.

Search

Describe it in plain language

Find sessions by meaning, exact text, or metadata. The query above finds traces that never use those words.

Search →

Behaviors

Grouped by what users tried to do

Whole sessions clustered by intent and outcome, so patterns surface without you writing the query.

Behaviors →

Signals

One named, tracked problem

Repeated failures become one Signal with a trend, affected sessions, and a lifecycle. Flaggers feed it automatically.

Signals →

From production failure to verified fix

Turn a recurring LlamaIndex failure into a regression test, an opened PR, and a verified fix.

DetectLatitude groups sessions where the current policy is missing from retrieved context into a Signal.
TestTurn affected traces into a regression eval.
FixAgent Dispatch sends the failure, sample traces, retrieved context, and release metadata to Claude Code or Cursor. The coding agent writes the smallest retrieval or indexing fix and a regression test, then opens a PR.
VerifyLatitude scores new traffic and alerts you if the failure returns.

Latitude detects and verifies. Your coding agent writes the fix. You review the PR.

Fits the LlamaIndex stack you already have

  • Python 3.11+
  • Latitude SDK or OTLP ingest
  • RAG, agents, and Workflows
  • Any vector store, embedding model, or LLM provider
  • Cloud or self-hosted (MIT)

Using LlamaTrace, Phoenix, or another observability backend?

Keep it if it serves another workflow. LlamaIndex instrumentation can feed OpenTelemetry backends, and an OpenTelemetry provider can export to more than one destination.

Latitude does not import historical traces from another backend. Search, Behaviors, and Signals begin building from the traces you send after setup.

When do you need more than LlamaIndex logs and traces?

A trace explains one query. Latitude shows which retrieval, synthesis, and agent failures keep affecting production sessions.

CapabilityApplication logsBasic tracing / LlamaTraceLatitude
Complete agent sessionsLimitedYesYes
Semantic search across sessionsNoVariesYes
Recurring failure detectionManualVariesSignals
Production traces into evalsNoVariesYes
Dispatch and verificationNoNoYes

MIT

Open source, 4.6k stars

20K

Free credits a month, 30-day retention

Unlimited

Seats on every plan, no per-user pricing

GDPR

Compliant, data in AWS eu-central-1

PricingSelf-hostingSecuritySource on GitHub

LlamaIndex observability FAQ

How do I add observability to a LlamaIndex application?

Install latitude-telemetry, initialize Latitude with instrumentations={"llamaindex": llama_index}, and run the query engine, workflow, or agent entrypoint inside capture(). One entry covers the whole pipeline: retrieval, embeddings, model calls, tools, and workflow steps all arrive from the LlamaIndex instrumentation, so you do not need to register your model client separately. Add session_id and user_id for multi-turn applications, and call latitude.flush() in short-lived scripts and jobs.

What parts of a LlamaIndex RAG pipeline can I trace?

Querying is traced end to end: query transforms, retrievers with the nodes they returned and their scores, embedding calls, rerankers and other node postprocessors, response synthesis, model calls with messages and prompt and completion token counts, agent steps, tool calls with their arguments and results, and each step of a custom Workflow. Indexing is traced too, but ingestion that runs outside a capture() arrives as its own trace rather than inside a query session, so wrap the indexing job as well if you want it grouped. The exact detail depends on the components and instrumentation version. Add index version, release, and tenant metadata so production failures can be compared safely.

Can Latitude help separate retrieval failures from synthesis failures?

Yes. Inspect the retrieved nodes and context before the model span. Wrong or missing nodes point toward retrieval, chunking, filtering, embedding, or index lifecycle. Relevant nodes paired with a wrong answer point toward synthesis, prompting, model behavior, or output validation.

Can I send LlamaIndex OpenTelemetry traces directly to Latitude?

Yes. Configure LlamaIndex's OpenTelemetry integration with an OTLP HTTP exporter for https://ingest.latitude.so/v1/traces and include the Authorization and X-Latitude-Project headers. You can also use Latitude's Python SDK and its LlamaIndex instrumentation, which sets the exporter up for you.

Does Latitude sample LlamaIndex traces?

Latitude captures 100% of the traces it ingests within your retention window. LLM-based flaggers and evaluations can sample at analysis time, starting around 10% by default, and that coverage can be increased.

Trace your first LlamaIndex query

Connect your LlamaIndex application, run one production-like query or agent session, and inspect retrieval through final response in Latitude.

Start tracing LlamaIndexRead the integration guide

No credit card required.