Token usage (LLM) is the number of tokens a large language model processes for a request: input tokens for the prompt, system instructions, history and retrieved context, and output tokens for the generated answer. It is the main driver of LLM cost and a strong factor in latency, so it is measured per call, per feature and per tenant.
What is token usage in LLMs?
Language models do not read characters or words but tokens: fragments of text produced by a tokenizer. Depending on the model and language, a token can be a whole short word, part of a longer word or a punctuation mark, and the same sentence can result in different token counts for different models. Token usage is the count of these tokens per request, reported separately as input and output.
Most model providers return token counts in the API response and bill by them, usually with different prices for input and output tokens. Token usage therefore connects three things that teams care about: cost, latency and the limits of the context window. Measuring it is one of the first steps of LLM observability.
Types of tokens
| Type | What it contains | Why it matters |
|---|---|---|
| Input tokens | System instructions, user prompt, conversation history, tool definitions, retrieved documents | Usually the largest share in chat and RAG applications |
| Output tokens | The generated answer, tool call arguments | Generated sequentially, so they drive response time |
| Cached input tokens | Prompt prefixes served from a provider-side cache | Many providers bill them at a reduced rate |
| Reasoning tokens | Internal reasoning of reasoning models, often not visible | Billed as output but not part of the returned text |
Which categories exist and how they are billed differs by provider and model, so always check the provider's documentation.
How to measure token usage with OpenTelemetry
The OpenTelemetry GenAI semantic conventions, currently with status development, define span attributes for token counts on every model call:
gen_ai.usage.input_tokens: input tokens, including cached tokensgen_ai.usage.output_tokens: output tokensgen_ai.usage.cache_read.input_tokens: input tokens served from cache
Alongside them, attributes such as gen_ai.request.model, gen_ai.response.model and gen_ai.provider.name identify where the tokens were spent. The conventions also define client metrics for token consumption; earlier versions used a histogram named gen_ai.client.token.usage, while newer drafts define separate counters in the gen_ai.client.inference.usage.* family. Because the conventions are still evolving, pin the version your instrumentation emits and check the release notes when upgrading.
For cost reporting, enrich the data with business dimensions such as feature, tenant or agent name. Keep an eye on cardinality: user IDs or prompt hashes as metric labels create a very large number of series. Put them on spans instead and aggregate later.
Example: finding a token cost driver
An illustrative case: the token spend of an internal assistant doubles within a week, while the number of requests stays roughly flat. Grouping input tokens per request by feature shows that one chat feature now sends the full conversation history on every turn. Average input tokens per call in that feature grew from about 2,000 to about 9,000 tokens. Output tokens are unchanged.
The team caps the history at the last few turns plus a short summary and moves the static system instructions to the beginning of the prompt so they can be served from the provider's prompt cache. The dashboard then shows input tokens per call falling again and a rising share of cached input tokens.
Best practices for controlling token usage
- Record input and output tokens on every model call, not only in monthly provider invoices.
- Track tokens per request and per successful outcome, not just totals; growth in traffic is fine, growth per request is a signal.
- Trim history, deduplicate retrieved chunks and limit the number of documents in RAG prompts.
- Set a maximum output length that fits the use case.
- Structure prompts with a stable prefix to benefit from provider-side caching.
- Budget agents: limit steps and tokens per run, because loops multiply consumption, as described under AI agent observability.
- Alert on sudden changes in tokens per request after deployments or prompt changes.
Common mistakes
- Estimating instead of measuring. Character-based rules of thumb are often far off, especially for non-English text, code and structured data such as JSON.
- Ignoring hidden tokens. Tool definitions, system instructions and reasoning tokens are easy to overlook but count towards usage and cost.
- Only looking at totals. A flat monthly total can hide one feature whose consumption per request is rising while another is shrinking.
How Obseria helps
Obseria stores GenAI spans and metrics received over OTLP and lets you break token usage down by model, provider, service, feature or tenant with PromQL, OQL or SQL. Token metrics can be combined with alerting and with cloud cost management to see LLM spend next to infrastructure cost, and traces show which step of a request consumed the tokens.
