Agent Tracing and GenAI Telemetry

Three assumptions an agent trace breaks

You already know distributed tracing on the request and response model: a span is a network call, one timed step recorded with a parent so the steps reassemble into a tree, the depth of that tree is fixed by your call graph, and the attribute you stare at is duration. An agent run breaks all three at once.

Depth is decided at runtime by the model rather than by your code, so the same endpoint produces a three-span trace for one user and a ninety-span trace for the next. One logical operation runs for minutes rather than milliseconds, so the trace is still open while the user is still waiting. And duration stops being the expensive axis, because tokens are money: a span that took 400 milliseconds can cost more than the span beside it that took 40 seconds.

Here is the shape, with the numbers that matter on every node.

invoke_agent  research-assistant           42.1s   in 61,206   out 3,410
├─ plan  gpt-x                              6.4s   in  8,614   out   380
│  ├─ chat  gpt-x                           3.1s   in  4,102   out   210
│  └─ chat  gpt-x                           3.3s   in  4,512   out   170
├─ execute_tool  search_docs                0.9s   in      0   out     0
├─ retrieval  corpus-v4                     0.3s   in      0   out     0
├─ chat  gpt-x                             11.2s   in 22,140   out   890
├─ execute_tool  run_query                  2.7s   in      0   out     0
├─ chat  gpt-x                             10.9s   in 24,802   out 1,020
└─ chat  gpt-x                              8.5s   in  5,650   out 1,120

Read the depth first. The two chat spans that produced the plan are children of plan, and every tool span is a sibling of plan under invoke_agent. Read the durations second: the children sum to 40.9s while the parent is 42.1s, because the orchestrator's own work between steps lives in the parent and in no child. Read the tokens last, which is the reading nobody trained on HTTP traces performs: 61,206 input tokens crossed this one user request, and duration alone will never show it.

The convention, and where it now lives

OpenTelemetry's GenAI semantic conventions are the vendor-neutral answer to all of this. They define gen_ai.* attributes, span shapes for inference and tool and agent operations, metrics, and a convention for where prompt content goes. That matters because a trace outlives the vendor that first collected it: spans named by a vendor SDK cannot be moved to another backend without re-instrumenting every service, and they cannot be joined to the HTTP spans around them because the two vocabularies disagree about what a span is called.

Two facts about the state of these conventions have to be said plainly, because both of them mislead a careful reader.

Nothing in GenAI semconv is Stable. The spans, events, metrics, and agent-spans documents all carry the Development marker, and the repository has no tagged release at all. Attribute names can still change. Build on the shape, not on the spelling.

As of semantic-conventions v1.42.0 in June 2026, the GenAI conventions moved out of the main semantic-conventions repository into a dedicated GenAI repository. Every gen_ai.* page on the main registry now renders with a Deprecated badge, and that badge means relocated, not abandoned. A learner who greps the old registry for authority concludes the whole vocabulary was retired, which is exactly backwards.

The operation vocabulary is the part you should memorize, because it is what makes one team's trace legible to another: chat, embeddings, retrieval, fetch_response, generate_content, text_completion, execute_tool, create_agent, invoke_agent, invoke_workflow, plan, plus a memory family (create_memory, update_memory, upsert_memory, delete_memory, search_memory, and create and delete for a memory store).

Span names are model-parameterized rather than free text: {gen_ai.operation.name} {gen_ai.request.model} for inference and embeddings, {gen_ai.operation.name} {gen_ai.data_source.id} for retrievals, and the bare operation name for the rest. The response id is deliberately kept out of the name. A name carrying a unique id per call is unbounded cardinality, which is the same cost mistake as an unbounded metric label. Note also that invoke_agent splits into a client variant and an internal variant: the client variant requires a provider name, the internal one does not, because an in-process loop has no provider to name.

Table

Levels as documented in August 2026 on the inference operation. Nothing here is Stable, so treat the spellings as dated and the shape as durable. The conversation-id guidance is the one most often ignored: the spec declines to fabricate a grouping key, and a synthetic UUID or a reused trace id silently defeats the grouping it appears to provide.

Attribute	Level, August 2026	What it is for
gen_ai.operation.name	Required	chat, execute_tool, invoke_agent, plan, and the rest of the vocabulary
gen_ai.provider.name	Required	Which provider served the call, so a gateway failover is visible
gen_ai.request.model	Conditionally required	What you asked for. Rides in the span name
gen_ai.response.model	Recommended	What actually served you, which is often not what you asked for
gen_ai.usage.input_tokens	Recommended	The billable half nobody watches
gen_ai.usage.output_tokens	Recommended	The billable half everybody watches
gen_ai.conversation.id	Conditionally required	Set it only when it is readily available, never fake one
gen_ai.input.messages	Opt-In	The prompt. Registry-annotated as likely to contain sensitive data
gen_ai.output.messages	Opt-In	The completion. Same annotation
gen_ai.system_instructions	Opt-In	The system prompt. Same annotation

Plan and execution are phases, not caller and callee

An agent trace is a tree over a loop rather than a call chain, and that is the structural fact the HTTP model cannot supply. A plan span is the decision phase, where the agent formulates a strategy before executing it. The LLM calls that produce the plan nest under it, because they genuinely happen inside its clock. The tool spans that carry out the plan are typically siblings of plan under the parent invoke_agent, because the plan had already finished when they started.

Get that wrong and every downstream aggregation lies in the same direction, quietly, forever. Getting it right is a five-minute instrumentation decision.

Why a span per model call is not enough

This is the beat that justifies treating agent telemetry as its own subject. The conventions define, per agent invocation, two histograms of counts: one of inference calls and one of tool calls (invoke_agent.inference_calls and invoke_agent.tool_calls in the current spelling). Not counters. Histograms, so you can ask for a p99.

Runaway loops, tool thrash, and a planner that re-queries seven times before committing are all properties of the distribution of child-span counts under one parent. If your unit of analysis is one model call, all three are invisible, because every individual call looks completely normal: normal duration, normal token count, normal status. A p50 of 4 inference calls per run beside a p99 of 60 is a loop that does not always terminate, and no per-call dashboard will ever draw it.

There is no HTTP analogue for this measurement, which is precisely why the mental model does not transfer. In a request and response system the number of downstream calls is a property of the code: it is 3, or it is 3 plus a retry, and you would never build a histogram of it. Here it is a property of the model's output, so it is a random variable, and you watch its tail the way you watch a latency tail.

There is no cost attribute, and that is deliberate

The conventions carry token usage and deliberately carry no price. This is worth understanding rather than working around, because the reasoning generalizes: prices change, spans are immutable, and a dollar figure baked into a span written last March cannot be re-derived when you renegotiate a contract or when a provider drops its rate. A count can be re-priced forever. A price cannot be re-counted.

So cost is a downstream join and the price table is yours to version. The join needs three token classes rather than one, because uncached input, cached input, and output are three different prices. Your provider bills a cache read at a fraction of a fresh input token, and the standard token counters do not split them, so the cached-input count is a number your gateway has to record on the way past.

price table (versioned, with an effective_from date)
  model   class            usd per 1M tokens
  gpt-x   input_uncached    3.00
  gpt-x   input_cached      0.30
  gpt-x   output           15.00

the join, per span
  cost = in_uncached / 1e6 * 3.00
       + in_cached   / 1e6 * 0.30
       + out         / 1e6 * 15.00

the run from the span tree above, with 40,000 of its 61,206
input tokens served from the prompt cache
  uncached   21,206 / 1e6 *  3.00  = $0.0636
  cached     40,000 / 1e6 *  0.30  = $0.0120
  output      3,410 / 1e6 * 15.00  = $0.0512
  total                            = $0.1268

the same run, priced with one input class at the uncached rate
  input      61,206 / 1e6 *  3.00  = $0.1836
  output      3,410 / 1e6 * 15.00  = $0.0512
  total                            = $0.2348

One input class reports this run at 85 percent more than it cost. That error does not average out across a fleet, because it is signed: it always overstates, and it overstates most for the teams doing the best prompt-cache work, which is the exact opposite of the incentive you want on a chargeback dashboard.

Payload capture, and the storage seam it hands you

Three attributes carry the payloads: gen_ai.input.messages, gen_ai.output.messages, and gen_ai.system_instructions. All three are Opt-In, and all three are annotated in the registry as likely to contain sensitive information. The spans document is explicit that instrumentations should not capture them by default and that capture should be gated behind an explicit opt-in, "for example" an environment variable. Read that hedge carefully: the variable is illustrative, not normative, so do not design around one portable flag across four languages.

The same three attributes can instead ride a log-based event, gen_ai.client.inference.operation.details, which is one of only two events the conventions currently define (the other reports an evaluation result).

That dual homing is not a footnote, it is the architecture. Metadata-rich spans go to the hot trace store you query all day. The bulky prompt and completion payloads go to a separate pipeline that is cheaper per byte, shorter on retention, and access-controlled, and the two are joined on trace id and span id. A seven-day prompt retention beside a ninety-day trace retention falls out of that split as configuration. Reach for it before you reach for a bespoke redaction processor, which has to understand the shape of every prompt you will ever ship and will be wrong about the first one you did not anticipate.

Two cautions come with it. The events path is in development and not yet available in some languages, and the fallback is span attributes, which puts prompt text straight back into your trace store. And treat the capture flag as production configuration with secret-level review: content capture is off by default, so the risk direction is not somebody forgetting to enable it, it is somebody enabling it to debug an incident on Thursday and nobody turning it off.

On sampling, reason from the cost structure rather than from the spec, which takes no position here. You already know head-based sampling decides at the first span and tail-based sampling buffers the whole trace at the collector before deciding. For an agent, every signal that makes a run worth keeping (it errored, it looped, it burned 400k tokens, an eval marked it bad an hour later) exists only after the run has finished, so a head decision is a coin flip on exactly the traces you need. Agent traces are also low volume and high value next to web traffic, which inverts the calculus that makes head sampling attractive there. That is an argument, and a good one, but it is your argument and not a rule anyone handed you.

Bucket boundaries, the smallest beat with the largest payoff

The conventions specify explicit, non-default bucket boundaries for the two histograms that matter, precisely because these distributions run across three to four orders of magnitude.

gen_ai.client.token.usage       unit {token}, powers of 4
  1, 4, 16, 64, 256, 1024, 4096, 16384, 65536, 262144,
  1048576, 4194304, 16777216, 67108864

gen_ai.client.operation.duration  unit s, powers of 2
  0.01, 0.02, 0.04, 0.08, 0.16, 0.32, 0.64, 1.28, 2.56,
  5.12, 10.24, 20.48, 40.96, 81.92

http.server.request.duration      unit s, the web ladder you inherit by habit
  0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75,
  1, 2.5, 5, 7.5, 10

one 45-second agent run, measured on each
  GenAI ladder  lands in (40.96, 81.92]   a real bucket with two real edges
  web ladder    lands in (10, +Inf)       the overflow bucket, no upper edge

An explicit-bucket histogram estimates a quantile by interpolating inside the bucket the quantile falls into. If your top finite boundary is 10 seconds and your agent runs take 30 to 90 seconds, every single run lands in the overflow bucket, the estimator has no upper edge to interpolate against, and your reported p99 becomes a property of the bucket layout rather than a measurement of the system. The cruel part is that it will also be perfectly stable, so it renders as a healthy flat line while the thing it claims to measure doubles.

One more detail on the token histogram: it requires a token-type attribute separating input from output. Without it you have summed two quantities with different prices and different distributions into a single number that means nothing.

Three conventions, one normalization target

Three instrumentation conventions exist in the wild and you will meet all three. OpenInference, from the Arize and Phoenix ecosystem, does not use gen_ai.* at all: it organizes everything under a span-kind attribute with ten kinds, including RERANKER, GUARDRAIL, and EVALUATOR, which tells you it was designed around evaluation workflows. OpenLLMetry uses an llm. prefix. GenAI semconv is the third.

The ecosystem's answer was not agreement at the instrumentation library, it was normalization at the collector. The OpenTelemetry Collector contrib distribution ships a GenAI normalizer processor (Alpha, traces only) that rewrites attributes from non-OTel GenAI instrumentation into GenAI semconv, with built-in mapping tables for exactly two sources: OpenInference and OpenLLMetry. So the practical stance is to emit the standard where you control the code and translate at the edge where you do not. That mapping table also exposes the trap worth remembering: llm.token_count.prompt and llm.usage.prompt_tokens are two different names for the same quantity, in two different libraries, both using an llm. prefix.

Interview nuance: the answer that shows experience is a trace id that survives into the eval set. A production failure becomes a regression case only if you can find the exact run, read its trajectory, replay its inputs, and attach the trace to the case. If the trace id dies at the edge of the trace store, then "the assistant got worse this week" is an opinion forever, and no amount of dashboard will settle it.

Recap: an agent trace is a tree over a loop, so nest the plan's model calls under plan and keep tool spans as siblings under invoke_agent; measure per invocation, because inference-call and tool-call counts are distributions and loops hide in their tails; join token counts against your own versioned price table with uncached input, cached input, and output priced separately; dual-home prompt payloads to a short-retention, access-controlled channel and treat the capture flag as production config; and set the specified bucket boundaries, because the web ladder's 10-second ceiling turns every agent p99 into a quantization artifact.

Sources: OpenTelemetry GenAI semantic conventions · Gen AI registry, now relocated · Inside the LLM Call: GenAI observability · How we built our multi-agent research system