What is inference in AI?

An AI model's life has two stages. In training, the model learns from large amounts of data over a long time and at high cost. In inference, it uses what it learned: you ask something and the model produces an answer. Training happens once or at intervals; inference happens every time the model is used.

If a chef's years of training in the kitchen are training, cooking each order that comes in is inference.

For language models inference has two parts. First the model processes all of the text you sent in one pass. Then it generates the answer one token at a time, computing again for each new token. That is why long answers take longer and why output tokens are priced higher than input tokens.

A model is trained once but used millions of times. For a widely used AI product, total inference cost can eventually exceed the cost of training, which is why companies invest heavily in making it cheaper.

The main techniques are running the model at lower numerical precision (quantization), transferring a large model's knowledge into a smaller one (distillation), caching the computation for prompt parts that repeat (prompt caching) and using chips designed specifically for inference. Architectures such as MoE also cut cost by running only part of the model at each step.

Most of an AI product's monthly bill comes from inference. Not every job needs the largest model: simple tasks such as classification and summarizing can be done far more cheaply with small language models. How fast answers arrive is also an inference question, covered separately in the article on latency.