LLM Observability and Tracing: Reconstructing Why an AI Request Got This Result

A refund assistant receives a user's refund request. The interface returns HTTP 200, there is no exception stack in the logs, and all calls appear to have succeeded. This can still be a serious failure: the assistant may have retrieved a version of the return policy that has been deprecated and answered accordingly; it may have passed the wrong parameter when passing the order number to the refund tool; it may have automatically retried after the first call timed out, when in fact the first call had already created the refund and the retry created another one; or it may have smoothly promised the user an action it had no permission to perform. In these scenarios, all metrics at the transport layer are green, but the user got the wrong result or the wrong promise.

The root of the problem lies in the semantic boundary of HTTP 200. A 200 only confirms that the request was received and got a response at the transport level; it does not promise that the response is semantically correct, that the evidence is reliable, that the permission is valid, or that the business action succeeded. For an AI system composed of retrieval, generation, tool calls, and policy decisions, failures often occur precisely outside the transport layer: the model received the wrong input, retrieval returned outdated evidence, the tool was passed the wrong parameter, a retry introduced duplicate side effects, or the generated content did not match the permissions. These failures do not produce exception stacks, and traditional log-centric observability approaches are almost powerless against them.

Traditional logs are good at recording machine anomalies: process crashes, null pointers, and connection timeouts all leave traces. But LLM-driven systems also need to record three categories of things that traditional logs do not cover: semantic failures (answers that are fluent but factually wrong), evidence paths (which retrieved result and which version a conclusion is based on), and action outcomes (what actually happened in external systems after a tool call). Without these records, engineers can only guess after an incident occurs.

Observability has a boundary here that needs to be clarified: it is not “seeing the real thinking inside the model.” The activations and intermediate states inside a model cannot be reliably read out, and any claim of directly observing model thought exceeds what the tools can actually do. What observability reconstructs is the causal chain outside the model: what versioned input the model actually received, what calls were actually made, what observations external systems gave for those calls, and what side effects were ultimately produced. With this chain, engineers can attribute a wrong result to a specific link: was the input itself wrong, or was the retrieved evidence wrong; did the generation stage deviate from the evidence, or were the tool call parameters wrong; were the policy rules poorly designed, or did the environment data change between two requests.

Whether this set of records is sufficient can be tested with one core test: can on-call personnel, without reproducing the user's sensitive original text, answer four questions based solely on the trace—which step the failure occurred at, which version that step used, what evidence the conclusion at the time was based on, and what state changes this operation produced. If they can answer, then this causal chain is locatable, replayable, and verifiable; if they cannot, then the system is invisible at key links, no matter how frequent its 200 responses are.

From this, we can give the inputs and outputs of the AI observability layer. The inputs are the event stream collected along the request chain: versioned requests, retrieval results, model inputs and outputs, tool calls, policy decisions, and the side effects of each step. The output is a causal timeline that can be located, replayed, and verified, which decomposes “the user got this result” into “which step, with which version, based on what evidence, and what state was changed.” The trace records the actual inputs, actions, and state changes that occurred; it never claims to read model thought. At the same time, the sensitive original text is not something that must be saved by default—which fields to record and to what extent to redact are decided by privacy policy, not an inevitable cost of observability.

Many teams fall into the same predicament after an incident: the logging system clearly stores massive numbers of records, yet when it comes to actually locating the problem they still can only guess. The reason is not that there are too few logs, but that logs are only one of four observability signals, and each answers a different question; no single one can cover the complete diagnostic chain.

Logs record discrete events and answer the question “what happened at a particular step.” For example, if the refund tool returns permission_denied, this event will leave a clear record in the log, and the on-call engineer can confirm from it that the permission was indeed denied. The value of logs is that they faithfully preserve the specific result of a step at a given moment, but logs alone cannot tell you whether this is a large-scale occurrence or an isolated incident.

Metrics aggregate discrete events into time series and answer the question “whether scale and trends are abnormal.” p95 latency and unauthorized interception rate are typical metrics: they do not care about the details of individual requests, but expose the overall direction of the system. When “rising error commitment rate” appears on the dashboard, engineers know that some kind of error is spreading, but this curve alone cannot tell which requests or which version is at fault.

Tracing records spans with parent-child structure and answers the question “how a single request flows across components.” A refund request is passed along a chain such as retrieval, model, tool, retry, and the trace strings this entire path into an ordered, hierarchical relationship, allowing you to see where the request branches, where it waits, and where it fails. Only after metrics find the abnormal scale can traces then find the affected requests and versions.

Evaluation labels judge quality at the sample level and answer the question “whether the result is good or bad, and what type of failure it is.” They annotate semantic judgments such as eligibility errors and unsupported citations, and are the only signal that can confirm “wrong answer content” rather than just “execution errors.”

The four play relay roles in a single diagnosis: metrics detect “rising error commitment rate,” traces find the affected requests and versions, logs explain exactly what was returned, and evaluation labels confirm that this is a semantic failure rather than an execution failure. Missing any one layer leaves a blind spot—with only traces, you do not know whether the problem is widespread; with only metrics, you do not know how to reproduce it. Conversely, understand the inputs and outputs of these four types of signals: the inputs are discrete events, aggregated trends, cross-component paths, and sample quality judgments, and the outputs are logs, metrics, traces, and evaluation labels. Logs explain what happened at a step, metrics detect scale and trends, traces connect a single request, and labels judge whether the result is good or bad; the four are linked together through trace ID and version so that they can corroborate one another. Keeping only one type of signal limits diagnostic ability to the one question it is good at, and during an incident you can naturally only guess.

A single user query often triggers ten model and tool calls in the background, across different services, processes, and even different machines. The core problem trace solves is: when these calls are scattered, how to keep them in the same causal chain and later reconstruct the order and attribution relationships between calls.

The smallest structural unit of tracing is the span. Each span corresponds to an operation with clear boundaries: a retrieval, a model inference, a tool call, a retry. The root of the whole tree is called the trace; it corresponds to the user's overall task. Relationships between spans are expressed using a parent-child structure: under the root span are the retrieval span and the model span, under the model span is the tool span, and the retry span after a tool timeout has the tool span as its parent. Figure 1 shows the parent-child relationships among these types of spans—retrieval, model, tool, and retry—in the refund assistant trace.

Just drawing the hierarchy is not enough; diagnosis requires three kinds of information on spans: parent-child relationships indicate call ownership, event times indicate the sequence, and idempotency keys are used to identify duplicates. The reason why incidents like duplicate refund creation can be diagnosed is precisely time and idempotency keys: the first tool call and the retry call have the same idempotency key. When two side-effect records appear in the business database but there is only one expected operation, the time difference between the two and the idempotency key directly expose the duplication. Without the idempotency key, side effects from retries cannot be determined to be two records of one operation or two independent operations.

To fully reconstruct the trace, span data requires each span to record start and end times, a reference to the parent span, status, and attributes. But the structure itself does not automatically cross processes: trace context must be explicitly propagated across HTTP, queues, and background tasks. A request enters a service via HTTP, is placed into a message queue, and is then consumed by a background worker—if context is lost at any hop, subsequent spans will break off the tree and become orphan points without a parent, and the entire causal chain will be disconnected.

There is also an easily overlooked evidence rule: when the model-generated text claims "I called a tool," that statement itself is not evidence. The model's output may describe a call that never happened, or omit a call that did happen. What truly constitutes evidence of a call is the tool span and the state change in the business database. From this we can summarize the input and output of the tracing structure: the input is the user task and calls across HTTP, queues, and background tasks; the output is a span tree with the root trace as the vertex and connected by parent-child relationships; each span records time, parent, version, status, and actual side effects, and context is responsible for cross-boundary propagation. The tool span and business state constitute the evidence that a call occurred; the model's after-the-fact claim of what it did cannot replace system records.

The same input succeeding yesterday and failing today is the most common form of AI system incident. To answer “what changed,” what needs to be compared is not the text of the two outputs, but a set of reproducible version evidence. The goal of logging is not to save as much content as possible, but to save a minimal set of fields sufficient to compare yesterday's and today's differences item by item after an incident.

These fields can be grouped into several families by diagnostic purpose:

Correlation fields allow all records for the same request to be linked across components, and also allow duplicate records to be deduplicated. Version fields support comparing differences before and after a release, and also determine which version to roll back to. Input evidence fields are used to reconstruct the information actually visible to the model at the time—the model received the masked intent, the selected candidate document IDs, and their spans within the documents. Behavior fields record tool names, parameter summaries, and idempotency keys, for auditing external side effects. Result fields connect quality evaluation to specific requests, rather than just staring at status codes.

In addition, you need to record sampling parameters, retrieval queries, index/embedding/reranking versions, tool schema and response status, policy rules, token counts, costs, and final acceptance results. Each category of fields serves the comparison question of “which item differs between yesterday and today”: a different release number means the deployment changed, a different prompt hash means the instructions changed, a different index version means the evidence base changed, different retrieval queries and candidate document IDs mean the information the model saw changed, and different tool parameters and idempotency keys mean the side-effect path changed.

There is an important trade-off in how to record: save content hashes or controlled references, rather than copying raw sensitive text into every logging system. Hashes and stable IDs can also support locating differences—identical hashes mean the content has not changed, different hashes point to the specific change—while avoiding the spread of original text across multiple systems. Record too little and the incident cannot be reproduced; record all original text and the privacy surface expands. The balance between the two is: store version evidence and references, not unnecessary original text.

There is also a clear exclusion: do not record free-form chain of thought as factual truth. The model's free-form chain of thought may not faithfully reflect what it actually did, may contain sensitive data, and storing it token by token is costly. What needs to be retained is only a concise decision summary and actually observable action records. In summary, the inputs to reproduction fields are the respective versions of the application, model, prompt, parameters, index, candidates, tools, policy, and acceptance; the output is minimal comparable version evidence plus controlled content references. Hashes and stable IDs support locating differences while avoiding copying original text into every logging system; free-form chain of thought is not treated as factual evidence.

A refund request took 1.42 s end-to-end, exceeding its SLO. To locate where the slowness is, first answer a question that is easy to get wrong: Does the duration of the root trace equal the sum of all span durations? Not necessarily.

In the example trace, retrieval took 180ms, model invocation took 620ms, tool invocation took 90ms, backoff and retry took 310ms, and orchestration itself took 220ms. If all these stages execute serially, the total is exactly 180 + 620 + 90 + 310 + 220 = 1420ms. But spans are not always serial: if retrieval and safety classification execute in parallel, the two spans' times overlap, and the duration of the root trace is determined only by the critical path—that is, the longest dependency chain that determines the final completion time—not by the sum of all child span durations. The number obtained by adding all spans is work, not end-to-end time. Therefore the key formula for latency analysis is:

Tcritical = sum of durations of serial stages + duration of the longest dependency path in parallel stages

where Tcritical is the end-to-end wall-clock latency. Serial stages are added directly; for parallel stages, take the longest path that determines the completion time; the sum of the two is the actual duration of the root trace.

With this decomposition, you can compare the duration of each stage this time with the historical p95 one by one:

Retrieval is below the historical level, full model generation is also below p95, and only the first token is slightly slow. The real anomaly is in tools and retries: the tool call itself at 90ms plus backoff and retry at 310ms totals 400ms, while the historical p95 is only 110ms. The total duration of 1420ms exceeds the SLO, mainly due to tool retries, not model generation.

This is exactly where looking only at average total latency goes wrong: average total latency attributes 1420ms to "this request was slow"; if you then assume without further thought that the slowness is in the model, you will blame the model for a tool timeout. To avoid this misjudgment, latency must be broken down to the granularity of TTFT (time to first token), per-token time, retrieval, tools, queueing, and retries, and you must observe p50, p95, and p99 rather than only the average—the experience of long-tail users will not be represented by the average, and only p99 can expose the slowest batch of requests.

Summarize the inputs and outputs of the latency case: the inputs are each span's start and end times, dependency relationships, and the durations of queueing, compute, network, and retries; the output is the critical path wall-clock latency Tcritical. Serial stages are added; for parallel stages, take the longest dependency path that determines completion; the sum of all span durations is work, not equal to end-to-end time. The main cause of the 1.42 s is tool retries, not average model generation.

After the like rate drops, the real problem is not "quality got worse," but how to know which tasks, which versions, and which types of failures the deterioration occurs in. A lone satisfaction curve cannot answer this question; quality metrics must be able to expand in two directions: downward to specific traces, and sideways into meaningful slices.

The way to connect them is to asynchronously write quality judgments back onto traces. Offline scorers or delayed human labels can attach semantic labels such as eligibility correct, citation supported, unauthorized access, tool completed, and human overridden to each request. Because these labels are attached to specific requests, aggregation can then slice by task, language, risk, model, prompt, index, and customer group, and every point on the dashboard can drill down to desensitized samples and actual execution paths. Metrics therefore are no longer a lump of averages but a clue that can be traced to evidence.

Online agent metrics are also valuable, but you need to be clear about their nature: signals such as refusal rate, transfer-to-human rate, user re-asking, tool undo, and citation click respond quickly but are not ground truth. Satisfaction may favor responses that confidently over-promise; a drop in transfer-to-human rate could also be because the escalation rule itself is broken, and users cannot even click the human entry point. Therefore every agent metric needs to come with possible alternative explanations and be validated by sampled human review or final business status; otherwise the green on the dashboard only masks real problems.

The full path from metrics to root cause can unfold like this: monitoring finds that "the citation failure rate rose from 2% to 7%"—this is only a signal and proves nothing. Drill down from this point to specific traces and find that the failed requests all come from index version index p17, and that the corresponding retrieval span retrieved no documents; only then does a testable root-cause candidate form: the new index lost documents during retrieval. Signal, slices, and trace evidence together make the hypothesis hold.

From this we can summarize the inputs and outputs of semantic monitoring: the inputs are asynchronously written-back labels for eligibility, citation, unauthorized access, tool completion, and human override; the outputs are quality trends sliced by task, language, risk, model, prompt, index, and group. Agent metrics only provide root-cause clues; likes, transfer-to-human, and re-asking can all have alternative explanations; you must drill down to desensitized traces and validate with human review or final business status before a quality conclusion can be considered established.

Ten million traffic traces per day cannot all be stored; the storage budget of an observability system forces us to sample. But if sampling is done carelessly, it causes two kinds of damage at the same time: either missing incidents, or making the saved samples unable to be used to estimate the true proportion.

A common question is: if we randomly retain only 1%, won't we happen to miss the incident? The answer is that purely uniform sampling can indeed miss rare events, so we cannot treat all traffic equally. A reasonable approach is stratification: use low-rate head sampling for regular successful traces, and increase tail sampling rates for error, high-risk, very slow, and new-version traffic. The tail rule has a temporal constraint—whether a trace belongs to “error or very slow” can only be determined after it completes, so the tail sampling decision must be made after trace completion, not intercepted in advance. More importantly, the probability that each trace is included must be recorded: when estimating the overall failure rate, we need to weight by this probability; otherwise, the proportion of errors in the sample cannot be used directly as the overall proportion. If only failed requests are saved, the failure proportion in the sample is always 100%, and we cannot estimate the real failure rate in production from it.

The weighted rate estimate based on inclusion probability can be expressed as follows. Let the outcome of each retained trace i be yᵢ (1 for failure, 0 for success), and its retention probability be pᵢ; then the weighted overall rate estimate is:

RateWeighted = Σ(yᵢ / pᵢ) ÷ Σ(1 / pᵢ)

The numerator Σ(yᵢ / pᵢ) means: a failed trace retained with probability pᵢ represents 1/pᵢ traces of the same type in the population; including it in the numerator is equivalent to scaling each observed failure back to the size of the population. Likewise, the denominator Σ(1 / pᵢ) scales all retained samples back to the total number of requests. Dividing the two gives the estimate of the overall failure rate. This Horvitz–Thompson-style weighting makes tail sampling legitimate—as long as the inclusion probability of each trace is recorded truthfully, even severe sampling bias can be corrected in estimation.

The cardinality problem threatens another direction: metric labels. Time-series systems require a limited number of label combinations. Directly using user ID, full prompt, or document ID as metric labels makes the time-series cardinality explode, while also spreading sensitive information to every monitoring system. The correct approach is to keep high-cardinality fields in controlled trace storage, and use only finite enumerations such as task type, language, and risk level as metric slice labels. Additionally, extremely low-frequency catastrophic events cannot rely on ordinary sampling. No matter how high the sampling ratio is, a failure that occurs once a year may fall outside the sample, so such events need a dedicated security event channel for separate reporting.

There is another hidden source of sampling bias: sampling only requests where users clicked “dislike” for quality analysis will miss errors that users are completely unaware of, especially fluent hallucinations—users think the answer is good, but the evidence is actually fabricated. Therefore, in addition to complaint-driven sampling, random audit samples must be retained. Summarizing the inputs and outputs of sampling-based estimation: inputs are each trace’s outcome yᵢ and retention probability pᵢ, output is the Horvitz–Thompson-style weighted rate RateWeighted; tail sampling can retain more error and high-risk traffic, but the inclusion probability must be recorded; saving only failure samples cannot directly estimate production failure rate, and extremely low-frequency disasters go through a separate security event channel.

From a debugging perspective, saving prompts and attachments in full is certainly the most convenient, but this cannot become the default acceptable approach. Prompts, attachments, and tool parameters may contain PII, health information, trade secrets, and credentials; copying them into the logging system simultaneously expands the range of accessors, data residency, retention period, and leakage surface: how many people can see the logs, which jurisdiction they fall under, how long they are retained, and from which entry points they might leak—all expand accordingly.

The correct approach to privacy governance is field-by-field decision-making, not all-or-nothing. First define necessary fields according to diagnostic purpose: if an order hash and error code are sufficient to reproduce an incident, then there is no need to save names and original text; if content review is truly required, place the encrypted original text in a separate controlled store with short-term authorized access, rather than piling it into general logs. Each observability field must go through a decision—masking, hashing, encrypted isolation, or simply not collecting—and the inputs to that decision include the field's diagnostic purpose, sensitivity level, accessors, region, and retention period.

Discipline at the collection stage is more reliable than after-the-fact cleaning: fields must be classified before collection, and original text and keys are off by default; masking should be performed in a structured manner at the entry point, because later regex cleaning can easily miss sensitive values nested deep within tool parameter structures. Access should be restricted by role, and every view must be recorded, with production and development environments isolated. For lifecycle, define retention periods, deletion processes, region requirements, and handling of user deletion requests; deletion must cover derived indexes—deleting only the original records but not the copies in the index is as good as not deleting.

There is also an engineering test that is easy to overlook: the trace pipeline itself must undergo failure-path testing to confirm that when the pipeline errors, it does not write unmasked payloads into fallback logs. During failures, the most dangerous is often the fallback log, because fallback logic often abandons masking in order to "not lose data".

Over-masking can also damage evidence. If all numbers are erased indiscriminately, failures such as amount errors can no longer be diagnosed. The correct granularity is to preserve type, range, or stable identifiers: do not save the specific amount, but save the type and value range of the amount field; do not save names, but save stable user identifiers. This way, diagnostic needs and privacy risks can be weighed separately for each field.

To summarize the inputs and outputs of privacy governance: inputs are each observability field's diagnostic purpose, sensitivity level, accessors, region, and retention period; outputs are the specific decision to mask, hash, isolate encrypted original text, or not collect. If diagnosis can be accomplished with type, range, and stable identifiers, do not copy names and credentials; if content review is needed, use short-term authorized controlled storage; deletion must cover derived indexes and fallback logs, so that privacy minimization and debugging capability can both be achieved.

A dashboard turning red without anyone taking action is not observability; it is only monitoring display. The observability loop must turn signals into action, and then feed the results of action back as new evidence.

The first link in the loop is the completeness of the alerts themselves. Every alert should be bound to a clear set of elements: the SLO target it serves, trigger threshold, duration window, owner, diagnostic query, remediation steps, and rollback version. An alert with no owner and no remediation path only trains people to ignore it. When it comes to how alerts are triggered, burn-rate alerts deserve a separate explanation: they measure the rate at which the error budget is being consumed, and can distinguish incidental noise from ongoing deterioration that is rapidly exhausting the budget. At the same time, every release marker must appear on the timeline; otherwise, it is impossible to determine which deployment a failure started from.

For example, when the refund over-authorization rate crosses the threshold, the response should proceed along two tracks. First, stop the bleeding: freeze the automated refund tool or switch back to the old policy, returning the service to a safe state. Then investigate the cause: check whether affected traces are concentrated in a particular prompt version (prompt h83), a particular policy version, or a particular language. Restoring service and fixing the model are two separate steps; do not delay stopping the bleeding in order to study the root cause — stop the bleeding first, then dissect. After confirming the root cause, construct the failure as a minimal regression sample, add it to the evaluation candidate pool, and after review admit it to the next version's frozen evaluation set, so that this failure will be re-examined in every future release.

Observability also has a clear failure boundary: it can only discover problems that are recorded, defined, and viewed. Problems without instrumentation it cannot see; problems without defined metrics it cannot measure; problems whose dashboards nobody watches it cannot alert on. It cannot replace permission isolation, idempotency design, evaluation systems, or incident response drills — these are the responsibility of the system design itself; observability is only responsible for making their failures visible.

To summarize the inputs and outputs of the response loop: inputs are SLO, error budget, alert window, owner, diagnostic query, and rollback version; outputs are stopping the bleeding, rollback, root-cause experiments, and reviewed regression samples. After an alert triggers, first restore safe service, then study long-term fixes; release markers and affected traces are used to verify hypotheses; and defenses such as permissions, idempotency, and incident drills cannot be replaced by observability — they can only be complemented.

String together the previous links: LLM observability goes from a user question to a verifiable fix, traversing a complete causal chain. Each step provides input to the next step; any break in a step leaves the chain stuck where it is.

The chain starts at the request entry point: generate a trace context for the user task and propagate it across HTTP, queues, and background tasks. This step determines whether the ten calls triggered by a single question still belong to the same causal chain. Then, each component records version, evidence, and actions in its own span—which application release was used, which prompt hash, which index version, which candidate document IDs were retrieved, what parameters the tool was called with, and what the idempotency key was. At this point the data is still scattered, so quality labels and system metrics are aggregated by task, language, risk, model, prompt, index, and cohort slices, and anomalies surface as trends.

Trends are only signals. When an anomaly appears, drill down into controlled, sanitized traces to turn signals into testable hypotheses: whether a particular version, a type of evidence path, or a tool call is affected. The hypothesis must then be verified back in production—roll back a version, ablate a parameter, observe whether metrics recover accordingly, while also stopping the loss. Finally, the verified failure samples go through human review and flow back into the evaluation set, becoming regression cases that the next release must pass.

Each link in this chain has clear inputs and outputs: the input to context propagation is the user task, and its output is a span tree that crosses boundaries; the input to span recording is the component's actual execution, and its output is fields for version, evidence, and actions; the input to aggregation is labels, and its output is slice trends; the input to drill-down is anomaly signals, and its output is root-cause hypotheses; the input to rollback and ablation is a hypothesis, and its output is the recovered service; the input to failure feedback is verified failure samples, and its output is regression cases that enter a frozen evaluation set. A question that starts from “where is it slow, where is it wrong” and finally lands on “how to ensure it won’t happen again” is the complete closed loop of LLM observability.