How a language model is trained.
How a model goes from guessing the next word to acting like an assistant, and where the chat app ends and the model begins.
Here is a small experiment. Open any chatbot and type The capital of France is, with no question mark and no instruction. It will almost certainly say Paris, and probably add a sentence or two. Nothing about that looks remarkable. But finishing a sentence is, in fact, the one skill the whole model was built on.
This lesson walks through how a language model goes from a blank network to the assistant you talk to. There are three stages, and the first is nothing more than that experiment repeated an almost unimaginable number of times.
Stage one: guess the next token, over and over
Training starts with text. A very large amount of it: web pages, books, articles, reference sites, forum threads, code. The text is cut into tokens, the word-pieces from the last lesson (a token is roughly three quarters of an English word). Then the model is given a game. It sees a stretch of tokens with the next one hidden and has to guess what comes next.
At first the guesses are random. Each time the model is wrong, its parameters (the billions of adjustable numbers inside it) are nudged a little in the direction that would have made the right token more likely. Then it sees the next stretch and guesses again. This repeats across the entire pile of text, which takes weeks or months on thousands of specialized chips. Nobody labels anything. The text supplies its own answers, because the hidden token was there all along. This is the self-supervised learning from lesson 3, and this first stage is called pretraining.
It is worth sitting with how simple the objective is. The model is never told to learn grammar or chemistry. It is only ever asked: what comes next?
What it absorbs along the way
Predicting the next token well turns out to require knowing a great deal. To finish The capital of France is, the model needs a fact. To continue a sentence that opens in a legal register, it needs a style. To close a parenthesis that opened forty words earlier, it needs structure. So, as a by-product of the guessing game, the model absorbs grammar, facts, writing styles, the shape of arguments, how code is formatted, how recipes are laid out, and a rough map of which ideas sit near which others.
It also absorbs everything else in the text, with no filter for truth. If an outdated claim is repeated on a thousand pages, the model learns that the claim is likely. If the text carries stereotypes, those become patterns too. The model learns the world as the text describes it, and the text was written by people, with people's mistakes and assumptions. (A later lesson looks at the bias side of this closely.)
One more thing comes from this stage: the knowledge cutoff. Pretraining uses text collected up to a certain date, and then collection stops. Anything that happened after that date is simply not in the model. Ask it about last month's news and, unless the app has a search feature, it has nothing to draw on.
Why the raw model is not yet an assistant
At the end of pretraining you have what is called a base model. It is a remarkable text predictor and a poor assistant. Give it a question and it may continue with more questions, because on the web, questions often appear in lists. Ask it to summarize an article and it might write another article, since that is what tends to follow one. It has no notion that you are a person who wants something. It continues the text in the most probable direction, and that is all.
The base model also has no preference for being accurate, polite or safe. It will complete an insulting sentence as readily as a kind one, because both appear in its training text. Something has to turn a text predictor into something that answers.
Teaching it to be helpful
The second stage is fine-tuning: training the model further on a much smaller, carefully chosen set of examples. These examples are conversations. A request, then a good reply. Summarize this article in three bullet points, followed by three good bullet points. Thousands of these, written or checked by people. The model learns the pattern: when text looks like a request, what follows should look like a helpful answer. Because the examples show instructions being followed, this is often called instruction tuning.
The third stage uses human judgment more directly. The model is given a prompt and produces several answers. People look at them and rank them: this one is clearer, that one is wrong, this one is rude. Those rankings train a second, smaller model whose only job is to predict which answers people would prefer. Then the main model is tuned to produce the kind of answer the second model scores highly. This is reinforcement learning from human feedback, usually shortened to RLHF. Reinforcement learning, from lesson 3, means learning from rewards; here the reward is a prediction of human approval.
RLHF is a large part of why the model declines harmful requests, explains its reasoning and writes in the structured, agreeable tone you probably recognize. It is also why models can be too agreeable. If raters tended to prefer confident, pleasing answers, the model learned to produce confident, pleasing answers.
The system prompt, and the app around the model
When you open a chat app and type your first message, your message is not the first thing the model reads. The app has already placed a block of text in front of it called the system prompt: standing instructions from the company that runs the app. What the assistant is called, today's date, how to format answers, what to refuse, how to handle files. You usually cannot see it, but it shapes every reply. Some apps let you add your own standing instructions, which the app places alongside its own.
This points to a distinction that clears up a lot of confusion. The model is the trained network: a fixed set of numbers that turns text into text. The chat app is the software wrapped around it. When a chatbot searches the web, remembers your name from last week, runs a calculation, reads a PDF or books a meeting, that is the app doing the work and handing the results to the model as more text. The model itself does none of those things. It reads tokens and predicts tokens.
So when one product knows today's news and another does not, the difference is usually not a smarter model. One app has a search feature and the other does not. Keep the two layers separate in your head and a lot of product claims become easier to read.
Try it
Open any chatbot and try two things.
First, type: Finish this sentence and add nothing else: The best way to learn a language is. You are watching the oldest layer of the model at work, pure continuation. Notice how natural the completion feels.
Then ask: What is your knowledge cutoff, and what happens if I ask about something after it? Most assistants will name a rough date and explain that they cannot know later events unless the app gives them search results. That answer is the fine-tuned layer describing the pretrained layer underneath it.
Key terms
- Pretraining
- The first and longest stage of training, where the model learns by predicting the next token across a huge amount of text.
- Base model
- The model as it comes out of pretraining: a strong text predictor that has not yet been trained to follow instructions.
- Fine-tuning
- Further training on a smaller, targeted set of examples, such as request-and-reply pairs, to change how the model behaves.
- RLHF
- Reinforcement learning from human feedback: people rank the model's answers and the model is tuned toward the kind they preferred.
- Knowledge cutoff
- The date after which no text was included in training, so the model knows nothing about later events on its own.
- System prompt
- Standing instructions the app places before your first message, which shape how the model answers.
Check yourself
A language model is a next-token predictor that was later tuned to behave like an assistant; search, memory and tools belong to the app around it.