AI inference is the operational phase where a trained model processes new input to generate outputs, distinct from the training phase where a model's internal weights are adjusted using massive datasets. During inference, the model's weights are frozen, meaning no new learning occurs during the interaction. The technical process involves a "forward pass," where input is transformed through the model's layers to produce a probability distribution over potential next tokens. In large language models, this occurs autoregressively, where each generated token is fed back into the model as part of the input to determine the subsequent token, a process that contributes to increased latency for longer responses. Key performance metrics for inference include latency, throughput, and cost per token. To optimize these production environments, engineers employ specific techniques: quantization, which uses lower-precision numerical formats to increase speed; batching, to process multiple requests simultaneously; and caching, to store frequent computations. While training is a periodic, resource-intensive event, inference is a continuous, high-volume computational task that often represents the primary ongoing cost for AI-driven products. Consequently, developers frequently trade a degree of model intelligence for speed by utilizing smaller, compressed models to ensure low-latency user experiences.