Time to First Token (TTFT)

What is time to first token (TTFT)?

Time to first token (TTFT) measures how long a user waits before a language model starts answering. With streaming, the response appears token by token, so the first visible output matters more for perceived speed than the time until the last token. TTFT is to LLM applications roughly what time to first byte is to web pages: a measure of initial responsiveness, not of total duration.

LLM latency is usually split into two parts: TTFT, and the time per output token after the first one (also called inter-token latency). Total generation time is roughly TTFT plus the number of output tokens multiplied by the time per output token.

The two parts have different causes and different fixes. TTFT is dominated by everything that happens before generation starts, especially processing the input, while time per output token depends mainly on model size, hardware and load. Looking only at total request duration mixes both effects: a response that took ten seconds may have started quickly and simply been long, or it may have kept the user staring at an empty screen for most of that time.

What drives TTFT

• Queuing at the provider or inference server under load, and rate limiting.
• Network round trips, TLS setup and distance to the model endpoint.
• Prefill: the model must process the entire input before generating. Long system prompts, chat history and retrieved documents increase it.
• Model size and hardware: larger models and busier accelerators process input more slowly.
• Reasoning: models that think before answering may emit their first visible token late, depending on whether reasoning output is streamed.
• Application work before the call: retrieval, guardrail checks and tool calls all happen before the user sees anything.

How to measure TTFT

On the client side, start a timer when the request is sent and stop it when the first content chunk arrives in the stream. Measure it on the server that calls the model and, ideally, also in the user interface, because end-user TTFT includes your own pre-processing. Report percentiles such as p50, p95 and p99, not averages, and break them down by model, prompt version and input length.

The OpenTelemetry GenAI semantic conventions define the histogram gen_ai.server.time_to_first_token (unit seconds) for model servers, together with gen_ai.server.time_per_output_token; both are marked as development. On the client side, many instrumentations record TTFT as a span attribute or event alongside gen_ai.client.operation.duration.

Example: a slow chat assistant

An illustrative case: a chat assistant's p95 TTFT rises from about 1.2 to 3.5 seconds after a release, while time per output token is unchanged. The trace shows that retrieval now adds twelve documents instead of four to the prompt, roughly tripling input tokens. Since prefill time grows with input length, TTFT grows too. Limiting the context to the most relevant documents brings TTFT back down without a measurable quality loss in the team's evals.

How to reduce TTFT

• Stream responses so users see output as soon as it exists.
• Shorten the prompt: trim history, retrieve fewer but better documents, remove unused instructions.
• Use prompt caching where the provider supports it, so a stable prefix does not have to be processed again.
• Choose the right model: a smaller model for simple steps, the larger model only where needed.
• Parallelize pre-processing such as retrieval and guardrail checks, and keep connections warm.
• Pick a nearby region and monitor provider queuing and rate limits.

Check every optimization with evals before rolling it out. A shorter context or a smaller model lowers TTFT but can reduce answer quality.

TTFT as a service level indicator

For interactive AI features, TTFT is a good candidate for an SLI, for example "95 percent of chat requests receive the first token within an example target of 2 seconds". Combine it with error rate and a quality metric, since a fast but wrong answer does not help. See what is an SLO for how to turn this into objectives and alerts.

How Obseria helps

Obseria's AI observability captures LLM calls as OpenTelemetry traces with latency, token usage and cost per model and prompt version, so you can see whether TTFT changes come from the provider, from longer prompts or from your own retrieval and guardrail steps.