Backprop and the Connectionist Revival

Between 1982 and 1998 a small community brought neural networks back from exile. You will learn how backpropagation trains a network, who really invented it, how networks learned to read handwriting, and why they still were not enough — yet.

By the late 1990s, if you deposited a handwritten cheque at an American bank, the amount might well have been read by a neural network. It had been trained at AT&T Bell Labs by a young French researcher named Yann LeCun, using a method that had made headlines in Nature a decade earlier. Few customers knew, and few computer scientists cared: by then neural networks were again unfashionable. This is the story of the revival between two winters — the people who kept the idea of learning machines alive, the tools they built, and the wall they hit.

Physicists and Boltzmann machines

After Minsky and Papert’s Perceptrons (see The First AI Winter), neural networks survived on the margins. The person who made them respectable again was a physicist. In 1982 John Hopfield of Caltech showed that a network of interconnected units could act as a content-addressable memory: give it a corrupted or partial pattern and it settles, like a ball rolling downhill, into the nearest stored pattern. He described this with an “energy” borrowed from the physics of magnetic materials. The Hopfield network brought physicists, and their mathematics, into the field.

Geoffrey Hinton and Terrence Sejnowski built on that idea with the Boltzmann machine (1983–85, with David Ackley), a network of randomly flipping units that could learn the statistical structure of its inputs, including hidden units that discovered features nobody had specified. It was slow, but it showed something the perceptron debate had seemed to rule out: hidden layers could be trained. In 2024 Hopfield and Hinton shared the Nobel Prize in Physics for these foundations of machine learning with artificial neural networks.

Backpropagation goes public

The tool that changed everything was backpropagation. In 1986 David Rumelhart, Geoffrey Hinton and Ronald Williams published “Learning representations by back-propagating errors” in Nature, and the same year Rumelhart, James McClelland and the PDP Research Group at UC San Diego published Parallel Distributed Processing, two thick volumes that became the manifesto of connectionism — the view that intelligence emerges from many simple connected units rather than from explicit symbols. Step around the loop to see what backpropagation does.

LOOPOne training step

Step 4 is gradient descent, walking downhill on the error surface. Try it below: a small learning rate creeps towards the bottom; too large a rate overshoots and bounces.

The 1986 paper’s great contribution was to show that hidden units trained this way learn useful internal representations — features that no one designed. That answered the XOR objection: a multilayer network could learn XOR and much more.

Debated Who invented backpropagation is a long-running dispute.

Other views
  • The earlier inventors: the general method — computing gradients efficiently by working backwards through a calculation, now called reverse-mode automatic differentiation — was published by the Finnish student Seppo Linnainmaa in his 1970 master’s thesis. Paul Werbos proposed using it to train neural networks in his 1974 Harvard PhD thesis, and applied it explicitly to networks in 1982. Related ideas appear in 1960s control theory (Henry Kelley, Arthur Bryson). David Parker and Yann LeCun also reached it independently around 1985. Jürgen Schmidhuber, among others, argues that the popular story gives these pioneers too little credit.
  • The popularisers’ case: Rumelhart, Hinton and Williams never claimed to be first to compute such gradients. Their supporters say their contribution was to show convincingly, with experiments, that the method makes networks learn meaningful representations — and that this is what launched the revival.

Networks that saw, spoke and read

The new tools were quickly put to work. Unfold the key papers on the shelf.

The revival’s bookshelf1980–1998
Tap a book.

LeCun’s networks were the first widely successful convolutional neural networks, and LeNet-5’s layered design of convolution, pooling and fully connected layers is recognisably the template that AlexNet scaled up fourteen years later. Other groups made networks steer: Dean Pomerleau’s ALVINN at Carnegie Mellon (1989) learned to keep a van on the road by watching a human drive.

The vanishing gradient and LSTM

So why did neural networks not take over in the 1990s? One reason was found by Sepp Hochreiter in his 1991 diploma thesis in Munich, supervised by Jürgen Schmidhuber. When the error signal is passed backwards through many layers — or through many time steps in a recurrent network that reads a sequence — it is multiplied again and again by small numbers and shrinks towards nothing. Early layers barely learn. This is the vanishing-gradient problem, analysed further by Yoshua Bengio, Patrice Simard and Paolo Frasconi in 1994. It meant that deep networks, and networks that needed to remember things from long ago in a sequence, were very hard to train.

Hochreiter and Schmidhuber’s answer, published in 1997, was long short-term memory (LSTM): a recurrent unit with a protected memory cell and “gates” that learn when to store, keep and release information, so the error signal can flow back over long stretches without fading. LSTM was slow to catch on, but two decades later it powered speech recognition and machine translation on billions of phones.

Intelligence without representation

A different rebellion came from robotics. Rodney Brooks at MIT argued that the classical approach — build a detailed internal model of the world, then plan — was why robots like Shakey were so slow and fragile. His subsumption architecture (1986) stacked simple behaviours — avoid obstacles, wander, follow walls — each wired directly from sensors to motors, with higher layers able to override lower ones. His six-legged robot Genghis walked over rough ground with no central world model at all. Brooks put the philosophy in provocative titles: “Elephants Don’t Play Chess” (1990) and “Intelligence without Representation” (1991). His company iRobot, co-founded in 1990, later made the Roomba vacuum cleaner.

This guide’s framing Connectionists and Brooks agreed on one thing: intelligence might emerge from many simple parts interacting with data and the world, rather than from rules written by people. That shared bet is the thread that runs from this page to today.

Quick check

What does the backward pass in backpropagation calculate?

Working backwards with the chain rule assigns each weight its share of the blame, so all weights can be nudged at once.

What does the universal approximation theorem not promise?

It is an existence result. Finding the weights, and doing it with a manageable network, are separate problems.

Why was LSTM designed?

Gated memory cells let error signals travel across many time steps without vanishing.

// Pause and reflect

  1. Backpropagation was invented several times before it caught on. What does it take for an idea to “arrive”, beyond being discovered?
  2. LeNet read millions of cheques yet neural networks stayed unfashionable. Why might a practical success fail to change a research field’s mind?
  3. Brooks argued robots need no internal model of the world. Do today’s AI systems seem to support or undermine that view?

Your notes are saved privately in this browser.

What happened next

With deep networks hard to train and computers still slow, many researchers in the late 1990s turned to methods with cleaner mathematics and better results on the data of the day: support vector machines, Bayesian models and boosted trees. Continue with The Statistical Turn.

Key terms on this page

Sources & further reading