What is latency in AI applications?
Latency is the time between sending a request and getting the answer. In AI products it shapes the user experience directly.
What is latency?
Latency is the time between sending a request to a system and receiving the response. On websites it is known as page load time. Language models deliver their answers in pieces, so latency is tracked with two separate measures.
How is AI latency measured?
- Time to first token (TTFT): how long until the first word appears after you send the request. It decides whether the system feels responsive.
- Generation speed: how many tokens the model produces per second. It decides how long the full answer takes.
Total time is roughly the time to first token plus the answer length divided by generation speed. A user asking for a long report naturally waits longer than one asking a short question.
What increases latency?
Large models run slower than small ones. Long documents stretch the time to first token and long answers stretch the total. Reasoning models spend time thinking before they answer. Server load and the distance between user and server add to the wait. In agent systems every tool call is another wait, and on multi step tasks those waits stack up.
How to reduce latency
Streaming the answer, printing words as they are generated, cuts the perceived wait considerably. Using a smaller model for simple tasks, shortening prompts, caching prompts that repeat and running independent steps in parallel also help. In real time products such as voice assistants and live chat, latency is treated as seriously as accuracy.
For the cost side, see the article on inference.