Large Language Models (LLMs) function as neural networks designed to predict the next token in a sequence through iterative loops. Rather than possessing inherent understanding, these models use statistical probability to determine the most likely next unit of text—such as a word or punctuation—based on the preceding prompt and generated outputs. The complexity of a model is measured in parameters; for instance, a 7B model contains 7 billion numeric weights. While larger models generally offer superior reasoning, they require significantly more VRAM and compute power. To mitigate hardware constraints, quantization reduces weight precision (e.g., from 16-bit to 4-bit), which can shrink a model's footprint by approximately 4x. Text generation is controlled by decoding strategies: greedy search selects the single most likely token, while sampling introduces variability via a temperature parameter. The context window defines the maximum token capacity the model can hold in memory, including system prompts and history; exceeding this hard ceiling results in dropped tokens or errors. Performance is measured via throughput, specifically the "eval rate" in tokens per second. Engineers can calculate this using Ollama's API fields by dividing the eval_count by the eval_duration. Ultimately, selecting a model involves balancing parameter count, quantization levels, and context requirements against available VRAM and latency needs.
Read original
dev.to