Lessons0 of 20 done

Module 1: Foundations. Lesson 4 of 20, 8 minutes.

Neural networks, without the math.

You will know what a neuron, a layer and a parameter are, how gradient descent trains billions of them, and what a trained network actually is.

If you have read anything about AI, you have met the phrase "neural network", usually next to a picture of glowing dots joined by lines. The picture is not wrong, but it does not explain anything. What are the dots doing? Why do more of them help? And what is left when the learning stops?

This lesson answers those questions with no equations. Everything here can be understood with a vote, a stack of filters and a walk down a foggy hill.

A weighted vote with a switch

Start with a single artificial neuron. Despite the name, it is not a detailed model of a brain cell. It is a tiny decision-maker that does two things.

First, it takes a vote. Several inputs arrive, each one a number. Each input has a weight, which says how much that input counts. A weight of zero means ignore this input; a large positive weight means it matters a lot and pushes toward "yes"; a negative weight pushes toward "no". The neuron multiplies each input by its weight and adds everything up. That is the weighted vote.

Second, it applies a switch. If the total crosses a threshold, the neuron fires and passes a signal onward; if not, it stays quiet. The switch is what makes this more than arithmetic. Without it, stacking neurons would collapse into one big sum. With it, each neuron is a small detector that lights up when enough of its favorite inputs are present.

One neuron alone can do very little. The power comes from connecting many.

Layers, and what deep means

Arrange a group of neurons side by side, all looking at the same inputs, and you have a layer. Each neuron has its own weights, so each detects something different. Feed that layer's outputs into a second layer, and that one's into a third, and so on until a final layer produces the answer: a probability that an image contains a cat, say, or a score for each possible next word.

The first layer sees the raw input: pixel brightnesses in an image, or the pieces of a sentence. The last layer produces the output. Everything in between is called hidden, because you never look at it directly. A network with many hidden layers is called deep, and that is all "deep learning" means: machine learning done with neural networks that have many layers. Early networks had one or two. Modern ones have dozens or hundreds.

What the layers learn to see

Why does depth help? Because each layer builds on what the one below found, and complicated things are made of simpler things.

Take a network trained to recognize faces. Researchers who look inside such networks find a consistent pattern. Neurons in the first layer respond to tiny, simple features: an edge here, a patch of light next to a patch of dark there. The next layer combines edges into corners and curves. A few layers up, neurons respond to parts: something eye-like, something like the bridge of a nose. Higher still, neurons respond to arrangements of parts, which is to say, faces. Nobody designed this hierarchy. Training found it, because building faces from parts and parts from edges is an efficient way to solve the problem.

The same holds for language, though it is harder to see. Lower layers handle local matters such as grammar and which word pieces belong together. Higher layers handle things that span sentences and paragraphs: who "she" refers to, whether a remark is sarcastic, what the topic is. Depth lets the network climb from letters to words to meaning in stages.

Billions of dials

Every weight in every neuron is a number that training can change. Together with a few other adjustable numbers, such as each neuron's threshold, these are the network's parameters. They are the spam filter's dials from the last lesson, at a different scale.

How different? A small network for a narrow task might have thousands of parameters. The networks behind modern chatbots have billions, and some a great deal more. A billion dials is hard to picture: turning one per second, they would take you more than thirty years. Training turns all of them, many times over, in days or weeks.

The count matters because it sets how much the network can hold. More parameters mean more capacity for patterns, which is why bigger models tend to be more capable, and why they need so much data: billions of dials and a few thousand examples means memorizing, not learning, the overfitting problem from the previous lesson at a larger scale.

Training: downhill in fog

How do you set billions of dials? With the spam filter's nudging, carried out on every parameter at once.

Imagine standing on a hillside in thick fog. Your altitude is how wrong the network currently is: the higher you stand, the bigger its total error on the training examples. You want the bottom of the valley, but you cannot see it. You can feel the slope under your feet, though. So you step a little in whichever direction goes most steeply downhill, feel the slope again, and step again. Repeat for long enough and you end up low without ever seeing where you were going.

That is gradient descent. The gradient is the slope: for every parameter, the training process works out whether nudging it up or down would reduce the error, and by how much, then nudges all of them a small amount in the helpful direction. One nudge barely changes anything. Millions of nudges walk the network from random guesses to useful ones.

The fog is real: nobody can see the whole landscape of possible settings. Following the local slope, step after step, works anyway. It does not always reach the lowest point in existence, but it reaches one low enough to be useful.

Why graphics cards mattered, and what you end up with

Most of the above was understood decades ago. What changed was the ability to do it at scale. Each training step is an enormous number of simple, independent multiplications and additions, which can all be done at the same time. Ordinary processors do a few complicated things in sequence. Graphics cards, or GPUs, were built for video games, where millions of pixels need the same simple calculation every frame: exactly the shape of work neural networks need. On GPUs, networks that would have taken months took days, and networks that had been impossible became routine.

Scale then became a strategy in itself: bigger networks, more data and more GPUs kept producing better results for longer than most people expected.

Here is what you are left with when training stops. A trained neural network is a very large, fixed set of numbers. Give it an input and the numbers turn it into an output. No learning takes place while you use it, and nothing inside it changes from one conversation to the next. Every chatbot reply and every recognized face comes from running an input through frozen weights. When you talk to one of these systems, you are talking to the result of the walk down the hill, not to something still walking.

Try it

Ask a chatbot to be your fog-walker. Type: I am thinking of a number between 1 and 100. Guess a number, and I will only tell you "higher" or "lower". Try to find it in as few guesses as possible. Play it through.

Each "higher" or "lower" is a slope: it tells the guesser which direction reduces the error without revealing where the answer is. That is gradient descent in miniature, one dial instead of billions. When it finds your number, ask: How is what you just did similar to how a neural network is trained? and compare its answer with this lesson.

Key terms

Neuron (artificial)
A small unit that adds up its inputs, each multiplied by a weight, and passes a signal on if the total crosses a threshold.
Weight
The number attached to one input of a neuron that says how much that input counts and in which direction.
Layer
A group of neurons that all look at the same inputs; networks stack many layers, and "deep" means many of them.
Parameters
All the adjustable numbers in a network, mainly its weights; modern language models have billions.
Gradient descent
The training method that nudges every parameter a little in the direction that reduces the network's error, over and over.
GPU
A graphics card, hardware built to do millions of simple calculations at once, which made training large networks practical.

Check yourself

1.What does the word deep in deep learning mean?
2.In the foggy hillside picture of training, what does your altitude stand for?
3.What is a trained neural network, once training has finished?

A neural network is layers of weighted votes, tuned by nudging billions of numbers downhill on error, and once trained it is a fixed set of numbers.