Definition
Time per Output Token, commonly abbreviated TPOT, measures the average time associated with producing each generated token after the first output token. It focuses on the autoregressive decoding phase, during which the inference stack produces output tokens conditioned on the preceding sequence.
A common calculation divides the time from the first token to the final token by the number of output-token intervals. Measurement definitions should state whether they include network delivery, buffering, or only server-side generation.
Simple example
A streaming request produces its first token after 700 milliseconds. The remaining 100 tokens arrive over five seconds. That is about 50 milliseconds per output token, or roughly 20 output tokens per second.
Another deployment might produce its first token in 300 milliseconds but take eight seconds to produce the remaining 100 tokens. Its Time to First Token is lower, but its TPOT is higher.
Why it matters
TPOT affects how quickly a streamed response continues once it starts. It helps distinguish ongoing generation latency from queueing and prompt-processing delays that often contribute substantially to Time to First Token.
The metric is influenced by model size, hardware, numerical precision, KV-cache behavior, batch scheduling, and concurrent load. Track distributions and output lengths rather than one isolated average.
One important nuance
TPOT and Time to First Token describe different phases of latency. A scheduling change can admit requests sooner while delaying tokens for requests already running. Average TPOT can also hide pauses and uneven token delivery. For a useful comparison across serving systems, keep the timing boundary consistent and look at TPOT alongside TTFT, total latency, throughput, and workload characteristics.