What are large language models (LLMs) and how do they work?
LLMs are probabilistic models with billions of parameters, trained on internet text. A next-word prediction game yielding uncanny capabilities at scale.
What do they do?
An LLM produces the most probable continuation for a given text (prompt). Simple on paper; but at sufficient scale, this yields near-human performance on coding, translation, summarization, reasoning, and even tool use.
Architecture: Transformer
The Transformer (Vaswani et al., 2017) broke the sequential bottleneck of RNNs/LSTMs. Self-attention lets every token attend to every other token in parallel, enabling training across thousands of GPUs.
Training stages
Pre-training: Masked language modeling on trillions of tokens — learns statistical structure of language, world knowledge, reasoning patterns. Cost: millions of GPU-hours.
Fine-tuning / RLHF: Reinforcement learning from human feedback aligns the model for instruction following, safety, helpfulness. This stage defines the model's "personality" and guardrails.
Limits and risks
LLMs are reasoning engines, not knowledge bases. Hallucination, training-data bias, staleness, and cost are real. RAG and tool use partially mitigate these.