# Inference latency: what it measures & why it varies

DevFeed: [Inference latency: what it measures & why it varies](<https://devfeed.tech/articles/inference-latency-what-it-measures-why-it-varies-4812.md>)

Original publisher: [Read original article](<https://redis.io/blog/inference-latency-what-it-measures-why-it-changes/>)

Author: Jeff Mills

Published: 2026-08-06T00:00:00Z

Content type: tutorial

Language: en

Sources: [Redis Blog](<https://devfeed.tech/sources/redis-blog.md>)

Topics: [Inference Performance](<https://devfeed.tech/topics/inference-performance.md>), [Low-Latency Inference](<https://devfeed.tech/topics/low-latency-inference.md>), [Language models](<https://devfeed.tech/topics/language-models.md>)

Tags: [agent](<https://devfeed.tech/tags/agent.md>), [benchmarks](<https://devfeed.tech/tags/benchmarks.md>), [gpu](<https://devfeed.tech/tags/gpu.md>), [inference](<https://devfeed.tech/tags/inference.md>), [latency](<https://devfeed.tech/tags/latency.md>), [llm](<https://devfeed.tech/tags/llm.md>), [memory](<https://devfeed.tech/tags/memory.md>), [metrics](<https://devfeed.tech/tags/metrics.md>), [model](<https://devfeed.tech/tags/model.md>), [production](<https://devfeed.tech/tags/production.md>), [redis](<https://devfeed.tech/tags/redis.md>), [retrieval](<https://devfeed.tech/tags/retrieval.md>), [tech-de](<https://devfeed.tech/tags/tech-de.md>)

## AI overview

A guide to LLM inference latency, distinguishing first-token time, per-token time, total response time, and multi-step agent run time. It contrasts latency with throughput and explains how prompt prefill, token decoding, queueing, and tail latency affect user experience.

## Source excerpt

Ask an engineer what their LLM app's inference latency is, and the honest answer is "which one?" The time to the first visible token, the time to the finished response, and the time an agent spends across a chain of calls are three different numbers. ...