The genealogy of language models

What if prediction is the only task?

Up until now our discussion has been informed by the original Transformer paper and the task it set out to solve: translation. Nonetheless, the mechanics of attention and learning are not specific to translation; the only thing that made it adept at French or German is that every sentence fed to it was paired with an English one (roughly four and a half million of them for English-German, and thirty-six million for English-French). Data like that, labelled, back then, was tediously assembled by hand and there was no way around it.

Radford et al.CitationRadford, Narasimhan, Salimans, and Sutskever. Improving Language Understanding by Generative Pre-Training. 2018. PDF motivate GPT-1 by the fact that unlabelled text is abundant while labelled text is scarce, and that we ought therefore to arrange for the majority of the learning to happen on the former. It takes two stages:

1. A Transformer decoder is trained to predict the next token over a large corpus of text.
2. For each downstream task, a single linear layer is attached to the pre-trained model and the whole thing is fine-tuned on some labelled data.

With a vocabulary of $40{,}478$ tokens, an untrained model pays a negative log-likelihood of $\ln 40478 \approx 10.6$ nats on every prediction. The pre-trained model reports a token-level perplexity of $18.4$, which corresponds to a loss of $\ln 18.4 \approx 2.9$.NoteThis is entirely off the unlabelled data. Interpreting token-level perplexity plainly, $18.4$ suggests that the model has the same average uncertainty as if, on each prediction, it had to choose uniformly among about eighteen equally plausible tokens. The model has gone from being unable to distinguish forty thousand tokens to being unable to distinguish eighteen of them, and the seven-and-a-half-nat difference was bought entirely off text that was unlabelled and unannotated.

The resulting model improved on the state of the art in nine of the twelve benchmarks the authors evaluated, using one architecture across all of them and essentially no task-specific components.

What was 2018 like?

Before we talk about GPT-1, I believe it is worth catching up on the revelations of the two years between the original Transformer architecture and this paper.

The idea of pre-training then fine-tuning was not itself novel. Dai and LeCitationsDai and Le. Semi-supervised Sequence Learning. 2015. arXivHoward and Ruder. Universal Language Model Fine-tuning for Text Classification. 2018. arXivPeters et al. Deep Contextualized Word Representations. 2018. arXiv had essentially proposed this in 2015, and Howard and Ruder's ULMFiT used fine-tuning for text classification, though both were built on LSTMs. Peters et al. trained bidirectional LSTM language models and then used their internal states as features, which were fed into whatever task-specific architecture the practitioner had already constructed. The representations transferred; the model did not.

The other concept generally attributed is making Transformers decoder-only. For posterity I will note here that Liu et al.CitationLiu et al. Generating Wikipedia by Summarizing Long Sequences. 2018. arXiv had already discarded the encoder for long-form summarization.

To me there seems to be parallel between the perception of GPT-1 & Transformers and what they actually did. Both are touted as these landmark papers in the space (I agree!) but what is revolutionary about them is the construction of the models not the ideas. Both of them took concepts that were commonplace and at the forefront of research, discarded a component previously regarded as indispensable, and showed incredible promise. For the Transformer that component was recurrence. GPT-1 discards labelled data, the encoder, and the hand-designed positional prior.

Every one of these choices had an incumbent, and a choice is only justified against the incumbent it displaced. So our discussion will be grounded in asking "what the alternative was and what keeping it would have cost?"

Goodbye Encoder!

Recall the three applications of multi-head attention§ in the original architecture:

1. encoder self-attention
2. masked decoder self-attention
3. encoder-decoder cross-attention

As I alluded to in the first section§, we would revisit some of these mechanisms. GPT-1 retains only the masked decoder—getting rid of the encoder entirely. Cross-attention§ exists so that the decoder may consult a source sentence, and encoder self-attention exists so that the source sentence has a representation worth consulting. If the task is simply to continue a passage of text, there is no source sentence, and both operations are left without anything to do.

What remains is a stack of blocks containing two sub-layers apiece:NoteGPT-1 remains post-norm.

The paper compresses the entire forward pass into three lines:

where $W_e$ is the token embedding matrix and $W_p$ the position embedding matrix. It is worth observing that the output projection in the final line is $W_e^\top$, the transpose of the input embedding matrix. This is exactly weight tying§.

Au revoir sinusoids

The sinusoids are gone. In their place, $W_p \in \mathbb{R}^{512 \times 768}$ is an ordinary parameter matrix containing one row per position, trained by gradient descent alongside everything else.

We spent a considerable amount of effort in the first section establishing and justifying the sinusoidal construction: displacement by a fixed offset corresponds to a linear transformation whose coefficients depend only on that offset, so that a relationship such as five tokens earlier is in principle recoverable by simple linear operations. Learned embeddings discard all of it; there is no reason for the vector at position $400$ to bear any particular relationship to the vector at position $401$, and whatever relationship they eventually acquire must be discovered from the data.

The sinusoidal construction shares structure across positions, and sharing is valuable precisely when some positions are rare: when the model sees position $9$ ten thousand times for every time it sees position $90$. GPT-1 trains on fixed-length sequences of $512$ tokens. Position $400$ therefore appears just as often as position $3$ (every training sequence contains both). There is less reason to impose a hand-designed rule for how those positions should relate when the model can learn the relationship directly.

Nothing prevents gradient descent from settling on an approximately sinusoidal basis if that is what helps; the learned table is free to rediscover the construction. It can also learn special behavior for particular locations—for example, treating the first position differently from the middle of a sequence. Ultimately, it spends $512 \times 768 = 393{,}216$ parameters on position embeddings, and those embeddings say nothing about positions beyond $512$. There is simply no learned vector for position $513$. Sinusoidal encodings, by contrast, can be evaluated at positions never encountered during training. GPT-1 accepts a hard context window in return.

Counting parameters

The model is composed of twelve layers, $d_{\text{model}} = 768$, twelve attention heads, $d_{\text{ff}} = 3072$, and a context length of $512$.

Within each layerNoteThis ignores biases together with the LayerNorm gain and shift parameters, which amount collectively to a few tens of thousands., attention contributes four square projections, $W^Q$, $W^K$, $W^V$, and $W^O$, giving $4 \times 768^2 = 2{,}359{,}296$, while the feed-forward network contributes two rectangular ones, $2 \times 768 \times 3072 = 4{,}718{,}592$, for a total of $7{,}077{,}888$ parameters per block, or $84{,}934{,}656$ across the twelve. The paper specifies a byte-pair encoding§ with $40{,}000$ merges; once base characters and special tokens are included the released vocabulary contains $40{,}478$ entries, so the embedding matrix accounts for $40{,}478 \times 768 = 31{,}087{,}104$, and the positional encoding table for the $393{,}216$ we just counted. Summing these gives $116{,}414{,}976$.

Something worth noticing in our decomposition is that roughly twenty-seven percent of the model consists of the embedding matrix, which is to say that better than a quarter of GPT-1's capacity is a lookup table. This ratio inverts as models grow, and it is one of the more reliable ways to see at a glance how large a model is.

Written August 22, 2026 at 6:45 PM GMT+5:30

What's changed

Beyond the removal of the encoder and sinusoidal embeddings, I count three further changes with respect to the architecture, none of which removes anything.

Initialization

Weights are drawn from $\mathcal{N}(0, 0.02)$, and the paper is explicit that a scheme this crude suffices because layer normalization is used extensively throughout the model.

The value invites comparison with the schemes and initialization patterns it declines. Xavier initialization would suggest a scale near $1/\sqrt{768} \approx 0.036$ and He initialization near $\sqrt{2/768} \approx 0.051$. So the constant $0.02$ is crudely half the principled value, and deliberately so.

These normative patterns can be rationalized as artifacts of plain feed-forward networks where it is required to hold activation variance constant as signal propagates through a deep stack: it is not a problem here. Every sub-layer output passes through a $\operatorname{LayerNorm}$ that rescales it regardless of what the weights were, so the forward-pass variance argument is immaterial in our setting. Initialization controls two other things.

The first is the scale of the residual branches. With small weights, the attention and feed-forward sub-layers initially make relatively small changes to the representation they receive. The residual connection therefore dominates early on, rather than each randomly initialized block immediately rewriting its input. Layer normalization then keeps the resulting activations at a controlled scale. This gives the twelve-layer stack a fairly conservative starting point: the blocks can learn useful transformations gradually instead of beginning with large arbitrary ones.

The second follows from weight tying. Recall the logits are $h_nW_e^\top$. Initializing $W_e^\top$ with a small standard deviation keeps those initial scores relatively close together, so the softmax begins without a strong preference for any particular token. Hence, the initial loss is near $\ln 40{,}478 \approx 10.6$ nats, an expected uniform baseline.

Activation function

Vaswani et al. used the rectified linear unit (ReLU) function, $\operatorname{ReLU}(x) = \max(0,x)$. Radford et al. substitute it with the Gaussian error linear unit (GELU)CitationHendrycks and Gimpel. Gaussian Error Linear Units. 2016. arXiv, $\operatorname{GELU}(x) = x \cdot \Phi(x)$, where $\Phi$ denotes the standard normal cumulative distribution function.

We can reinterpret ReLU as $x \cdot \mathbb{1}[x > 0]$. GELU replaces the indicator with $\Phi(x) = \mathbb{P}[Z \le x]$ for $Z \sim \mathcal{N}(0,1)$, which is the probability that the input exceeds a random draw from the standard normal distribution. The gate is thereby made soft, and the function, unlike ReLU, becomes smooth at the origin.

A couple of nice things to note. GELU is not monotonic; it descends to a minimum of approximately $-0.170$ near $x \approx -0.752$ before returning toward zero. The more practically relevant concerns the derivative,

which at $x = -1$ evaluates to $-0.083$. ReLU that has been driven negative receives no gradient whatsoever and may remain in that state indefinitely. GELU in the same position continues to receive a small signal, and can therefore recover.

With the initialization of the previous section and a LayerNorm on every sub-layer output, pre-activations sit concentrated in a band of roughly $[−2,2]$.NoteImmediately after LayerNorm, a hidden vector has approximately unit variance across its $768$ coordinates, so $|h|^2 \approx 768$. For a linear projection with independently initialized weights $w_i \sim \mathcal{N}(0,0.02^2)$, a pre-activation $z=w^\top h$ therefore has variance approximately $768(0.02)^2 = 0.307$, or standard deviation $0.554$. Thus $[-2,2]$ spans about $\pm 3.6$ standard deviations at initialization, containing essentially all of the mass under the resulting approximately Gaussian distribution. Looking again at the table, that is exactly the interval over which the two functions differ.

These weights are pre-trained for a hundred epochs and then handed, unchanged, to twelve different downstream tasks. Fine-tuning shifts the input distribution — serialized inputs, freshly initialized delimiter tokens, a different mixture of genres — so a unit that was dead and harmless on books may be exactly the unit the new distribution wants, and it arrives with no gradient by which to come back. Softness is insurance, and its value scales with how many times we intend to reuse the parameters.

The non-monotonicity is worth one extra remark. ReLU maps every negative pre-activation to the same output; it preserves the magnitude of positive evidence and destroys the magnitude of negative evidence. That dip to $-0.170$ is the unit keeping a little of it, so that mild and overwhelming negative evidence remain distinguishable downstream.

RemarkA practical note

The released implementation uses the tanh approximation

$$0.5x\left(1 + \tanh\left[\sqrt{2/\pi}\,(x + 0.044715x^3)\right]\right),$$

which agrees with $x\Phi(x)$ to roughly four decimal places over the range that matters and was faster than the error function on the hardware of the period.

Optimizer

It still uses Adam but the schedule has changed. The 2017 model warmed up linearly over $4000$ steps and thereafter decayed proportionally to $t^{-1/2}$, with its scale determined partly by the factor $d_{\text{model}}^{-1/2}$. Recall, specifically,

GPT-1 sets a maximum learning rate of $2.5 \times 10^{-4}$, increasing it linearly from zero over the first $2000$ updates. From there, rather than following the inverse-square-root decay, it anneals the learning rate to zero using a cosine schedule. If $T$ is the total number of training updates and $w$ the warmup length, the decay can be written as

Written August 24, 2026 at 3:27 PM GMT+5:30

Comparing these two equations, we can see that the first one depends on the current step alone and requires no knowledge of the total, which is exactly what we want when training runs until a validation metric stops improving and we cannot say in advance when that will be. Cosine annealing has no such property: $T$ is found in the denominator, and the schedule cannot be written down until the length of the run has been chosen. The authors had already decided: a hundred passes over a corpus of known fixed size at $32{,}768$ tokens per update.

The learning rate actually arrives at zero. This matters here in a way it does not in a translation model, because these weights are not the deliverable; as we have already seen, they are the input to twelve subsequent optimizations. A checkpoint lifted from a trajectory still moving at an appreciable learning rate is a sample from an oscillation, and which sample we get is luck. Annealing to zero produces a settled point instead.

There is one further line in the setup that is easy to read past. Regularization is applied following Loshchilov and HutterCitationLoshchilov and Hutter. Fixing Weight Decay Regularization in Adam. 2017. arXiv, with $w=0.01$ on every weight that is not a bias or a gain. That paper is the one we now know as AdamW, and the distinction it draws is worth restating: adding an $L_2$ penalty to the loss means the decay term passes through Adam's per-parameter normalization and is therefore scaled by the same $\sqrt{\hat{v}_t}$ as the gradient, so parameters with small historical gradients get decayed harder than parameters with large ones. Decoupling the decay applies a uniform pull toward the origin.

The objective

Having assembled the model, we can state what it is actually asked to do. Over an unlabelled corpus $\mathcal{U}=\{u_1,\ldots,u_n\}$, the objective isNoteThe window $u_{i-k},\ldots,u_{i-1}$ describes a fixed-width context of the kind an $n$-gram model or a windowed feed-forward language model would use, but there is no sliding window inside the $512$ tokens; the implemented model attends over the whole preceding span.

This is the same expression we arrived at in the previous section, once the encoder and the source sentence are removed from the translation loss. All the intermittent details—teacher forcing, casual masking, and right-shifted targets—are all found here too. The distinctive feature about this loss function is the set of things it is summed over.

Word-level embeddings like word2vec and GloVe were the incumbent way of transferring anything at all from unlabelled text, and they transfer exactly one vector per word type: no context, no composition, and nothing above the lexical layer. ELMo transfers more, but freezes it, so the pre-trained computation cannot be adapted to what the target task turns out to need.

Each optimization step supplies $64 \times 512 = 32{,}768$ prediction sub-problems, and pre-training runs for a hundred epochs. The dataset in terms of byte-pair encoding is on the order of $10^9$ tokens; a hundred passes over it is therefore something like $10^{11}$ individual predictions. The largest labelled dataset in the entire evaluation, SNLI, contains roughly $550{,}000$ examples.

Fine-tuning

We now have a model that assigns probabilities to sequences. Downstream tasks want labels.

Given a labelled dataset $\mathcal{C}$ in which each instance is a token sequence $x^1,\ldots,x^m$ with a label $y$, we pass the sequence through the pre-trained stack, take the final block's activation at the last position, $h_l^m$, and feed it into a single new linear layer $W_y$:

Written August 28, 2026 at 1:12 PM GMT+5:30