Behind the anthropomorphic vocabulary are probabilities, vectors, and learned weighted averages
A scientist who has spent decades studying nonlinear systems, modelling, statistics, or linguistics can open a modern paper on large language models and feel that an entirely new branch of mathematics has appeared. Query, key, value, attention, heads, embeddings, transformers. The impression is misleading. As Joseph L. Breeden argues in his recent paper The Simple Mathematics of Large Language Models, much of the barrier to understanding LLMs comes not from the mathematics itself, but from the language used to describe it.
Underneath the terminology are familiar tools: linear algebra, conditional probabilities, weighted averages, multinomial logistic regression, maximum likelihood, and gradient descent. This does not make LLMs simple systems. Their complexity comes from scale, from repeating these operations across many layers, and from fitting vast numbers of parameters to enormous datasets.
At the core is a conditional probability distribution
An autoregressive LLM can initially be described without saying anything about intelligence. It receives a sequence of tokens and estimates a conditional probability distribution for the next token:
P(xₜ | x₁, x₂, …, xₜ₋₁).
It selects or samples a token, adds it to the context, and repeats. Through the probability chain rule, successive next-token distributions define a probability distribution over complete sequences.
The hard part is that the number of possible contexts grows combinatorially. We cannot construct a lookup table containing every possible sequence and its empirical next-token distribution. The model therefore has to learn which different contexts are sufficiently similar for information about one to help predict another. Much of the Transformer architecture can be understood as machinery for solving this generalisation problem.
Turning language into vectors
Words do not enter the model as linguistic concepts. Text is first tokenised, and each token is mapped to a vector. These embeddings place linguistic elements in a high-dimensional space where relationships between them can be represented numerically.
For a statistician, principal component analysis provides a useful, though imperfect, analogy. Both approaches represent large amounts of correlated information through latent numerical dimensions. Unlike classical PCA, however, embedding coordinates generally have no clean independent interpretation and are learned jointly with the rest of the model.
The underlying intuition also has deep linguistic roots. Distributional semantics begins from the observation that linguistic meaning is partly reflected in context. A word appearing near “deposit”, “loan”, and “interest rate” develops a different contextual representation from the same surface form appearing in another semantic environment. Modern contextual embeddings go far beyond word2vec, but the principle that linguistic regularities can be learned from patterns of co-occurrence remains fundamental.
What “attention” actually does
Attention may be the most misleading term in the architecture because it invites comparison with a psychological process. Mathematically, its core is considerably less mysterious: it computes learned weighted combinations of vectors.
For each position, learned linear transformations create three representations conventionally called query, key, and value. Breeden suggests stripping away the metaphor and thinking of them simply as different projection matrices. The first two projections determine a compatibility score between positions. Softmax converts the scores into positive, normalised weights. Those weights are then used to construct a weighted combination of the third set of projected vectors.
For a statistician, softmax is familiar. It is the same mapping used to turn logits into class probabilities in multinomial logistic regression. The final output stage of an autoregressive language model can likewise be viewed as an enormous multinomial classification problem whose categories are the possible next tokens.
So-called attention heads are multiple sets of these transformations operating in parallel. Some heads end up responding strongly to positional patterns, syntactic dependencies, or coreference. Their specialisation, however, is not designed in advance. It emerges during training, and many heads do not admit a simple linguistic interpretation.
Training is parameter estimation at extraordinary scale
The fundamental training objective of an LLM is equally familiar from statistics. The model adjusts its millions or billions of parameters to increase the probability assigned to the actual next tokens in the training corpus. The cross-entropy loss used for this purpose is, in this setting, simply negative log-likelihood.
Scale changes the computational problem dramatically, but not the underlying statistical principle. Gradient-based optimisation searches a huge parameter space for models that make the observed sequences more probable.
After pretraining, additional stages may be used. Supervised fine-tuning can teach models to respond to instructions in desired formats, while human-feedback methods can further alter their behaviour. This distinction matters because the conversational behaviour people associate with a chatbot is not identical to the underlying next-token model produced during pretraining.
What has been learned, and what should not be assumed
The scientifically remarkable fact is that a system built from relatively familiar mathematical components can display extremely complex behaviour when trained at sufficient scale. But behavioural complexity does not automatically justify anthropomorphic conclusions.
Following the statistical perspective also developed by Cosma Shalizi, Breeden notes that a model operating over a finite context window can, in a precise sense, be treated as a very high-order finite-context Markov model. This does not reduce a Transformer to an n-gram table. Its learned internal representations allow it to generalise across contexts in ways that ordinary n-gram models cannot.
There is also an active scientific debate about what should be called “reasoning”. Strong performance on reasoning benchmarks may reflect the learning of useful abstract structures, sophisticated pattern exploitation, or different mixtures of the two depending on the task. Surface perturbations can sometimes cause striking failures. Hallucinations remain intrinsic to unconstrained probabilistic generation, and linguistic competence by itself does not provide grounding in the physical or social world.
This is why an LLM is often most useful as one component of a larger system. Databases can supply authoritative facts, retrieval systems can ground answers in documents, symbolic programs can perform exact calculations, and theorem provers can check formal deductions. The language model can serve as an interface and orchestrator without being expected to embody every kind of computation internally.
Demystifying LLMs is part of building open AI
The terminology matters because scientific language determines who can participate in a field. If LLMs are presented as mysterious machines that “pay attention”, “think”, and “understand”, researchers from statistics, linguistics, mathematics, and complex-systems modelling may reasonably conclude that the field depends on unfamiliar conceptual machinery.
Once the same system is described in terms of conditional distributions, learned projections, weighted averages, nonlinear transformations, likelihood optimisation, and distributional semantics, the intellectual landscape changes. LLMs remain extraordinary engineering achievements, but they become analysable objects rather than technological magic.
That matters especially for open models and for languages such as Greek. Building better language technologies requires more than machine-learning engineers. It requires linguists who understand morphology and meaning, statisticians who can distinguish estimation from interpretation, domain scientists who can design rigorous evaluations, and open-source communities capable of reproducing and auditing the systems.
Understanding what an LLM really is therefore has consequences beyond pedagogy. What can be understood can be tested. What can be tested can be reproduced. What can be reproduced can be improved openly rather than accepted as an opaque service supplied by a handful of platforms.