The Deep Learning Revolution

Between 2006 and 2016 neural networks went from unfashionable to unstoppable. You will see the three ingredients that made it happen — algorithms, data and graphics chips — read the numbers behind the 2012 breakthrough, and follow the decade to AlphaGo’s famous move 37.

In the autumn of 2012, the results of an annual image-recognition contest were announced at a workshop in Florence. For two years the best systems had inched forward, cutting their error rate by a couple of points a year. This time an entry from the University of Toronto, called SuperVision, made about 15 mistakes in every hundred. The next-best team made about 26. The winning system was a deep neural network trained on two graphics cards bought for gaming. Within months, almost every computer-vision lab in the world had changed what it was working on. This page tells how that moment came about, and what followed in the four years after it.

Keeping the faith

In the mid-2000s neural networks were out of favour (see The Statistical Turn). Deep networks — many layers stacked — were hard to train because of the vanishing-gradient problem, and methods such as support vector machines did as well with less fuss. A small group kept going, notably Geoffrey Hinton in Toronto, Yann LeCun in New York and Yoshua Bengio in Montreal, helped by long-term funding from the Canadian Institute for Advanced Research (CIFAR).

In 2006 Hinton, Simon Osindero and Yee-Whye Teh published “A fast learning algorithm for deep belief nets”. Their trick was to train a deep network one layer at a time, each layer learning to model the patterns in the layer below without labels, before fine-tuning the whole stack. Bengio’s and LeCun’s groups published related methods. Ironically, this pretraining trick was soon found to be unnecessary — but it proved that deep networks could be trained, and the group rebranded their field as deep learning.

Three ingredients: algorithms, data, chips

Chips. Training a neural network is mostly multiplying large grids of numbers, and graphics processing units — built to calculate millions of pixels at once for video games — do exactly that in parallel. In 2007 NVIDIA released CUDA, which let programmers use its GPUs for general computing. In 2009 Rajat Raina, Anand Madhavan and Andrew Ng at Stanford reported training certain networks dozens of times faster on GPUs than on ordinary processors, and Dan Cireşan and colleagues in Jürgen Schmidhuber’s lab in Switzerland used GPU-trained networks to win several pattern-recognition contests in 2011–12.

Data. Fei-Fei Li, then at Princeton, believed that vision research was starved not of algorithms but of examples. From 2007 her team, including Jia Deng, built ImageNet, presented in 2009: images gathered from the web and labelled through Amazon Mechanical Turk, organised by the categories of the WordNet dictionary, eventually over 14 million images. From 2010 the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) asked teams to sort about 1.2 million training images into 1,000 categories and scored them on held-out test images. The headline measure was the top-5 error: how often the correct label was missing from a system’s five best guesses.

Algorithms. Small but important tweaks made deep networks train well: the simple ReLU activation function, which reduces vanishing gradients; dropout, which randomly switches off units during training to curb overfitting; and better ways to initialise weights.

2012: AlexNet and the ImageNet leap

Alex Krizhevsky, Ilya Sutskever and Hinton combined all three. Their convolutional network, later nicknamed AlexNet, had about 60 million parameters in eight learned layers, used ReLUs and dropout, and was trained for about a week on two NVIDIA GTX 580 GPUs. Its architecture was a scaled-up descendant of LeCun’s LeNet (see The Connectionist Revival). Watch the winning error rate fall, year by year:

Bar length is proportional to the winning top-5 error; shorter is better.

Consensus Beating a single annotator on ImageNet did not mean machines “see better than people”. The test covers a fixed set of 1,000 categories in fairly typical photos; the same networks could be fooled by tiny, carefully chosen changes to an image that a person would never notice.

Deep learning spreads

The 2012 result landed alongside others. In the same year, researchers from Toronto, Microsoft, Google and IBM jointly reported that deep networks cut errors substantially in speech recognition, and Google moved Android voice search onto them. A Google team led by Quoc Le and Andrew Ng, with Jeff Dean, trained a network on frames from YouTube videos using 16,000 processor cores; without being told what to look for, one of its units responded to cats. Then came a rapid run of ideas that still shape AI:

  1. 2013word2vecTomáš Mikolov and colleagues at Google trained fast networks to turn words into lists of numbers — embeddings — in which meaning became geometry: the vector for king minus man plus woman lands near queen.
  2. 2013–15DeepMind’s Atari player (DQN)A single network learned 49 different Atari games from screen pixels and the score alone, using reinforcement learning; in its 2015 Nature paper it reached a level comparable to a professional games tester across the set. Google bought DeepMind in 2014.
  3. 2014Generative adversarial networksIan Goodfellow and colleagues in Montreal pitted two networks against each other: a generator making fake images and a discriminator trying to spot them. GANs were the first networks to produce convincingly realistic faces — and, later, deepfakes.
  4. 2014Sequence to sequence and attentionIlya Sutskever, Oriol Vinyals and Quoc Le used LSTMs to translate whole sentences; Kyunghyun Cho and colleagues proposed a similar encoder–decoder. Dzmitry Bahdanau, Cho and Bengio then added attention, letting the decoder look back at the most relevant input words. Google Translate switched to neural translation in 2016.
  5. 2015ResNetKaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun at Microsoft Research Asia in Beijing added shortcut connections that let signals skip layers, making networks of over a hundred layers trainable.
  6. 2015–16TensorFlow and PyTorchGoogle open-sourced TensorFlow in November 2015; Facebook’s PyTorch followed in late 2016 (publicly launched in early 2017), building on earlier tools such as Theano, Torch and Caffe. Free frameworks put deep learning in every student’s hands.

AlphaGo and move 37

Go had long been the board game computers could not crack: there are vastly more possible positions than in chess, and good moves are hard to judge by rules. DeepMind’s AlphaGo combined deep networks — one to suggest promising moves, one to evaluate positions — with Monte Carlo tree search, a descendant of the search methods of the golden years. It learned first from human expert games, then by playing millions of games against itself. In October 2015 it beat the European champion Fan Hui 5–0.

In March 2016 in Seoul it played Lee Sedol, one of the strongest players of the era, and won 4–1. In game 2 AlphaGo played move 37, a “shoulder hit” so unconventional that commentators first took it for a mistake; it later looked brilliant. In game 4 Lee answered with his own celebrated move 78, which threw AlphaGo off and gave humanity its single win. In 2017 AlphaGo Zero learned Go entirely from self-play, with no human games, and beat the version that had defeated Lee by 100 games to 0.

Debated Was move 37 creative?

Other views
  • “Yes, in a meaningful sense”: the move was novel, valuable and surprising to experts — the usual tests of creativity — and professionals have since adopted ideas from AlphaGo’s play.
  • “No — search plus statistics”: sceptics note that AlphaGo had no intentions or understanding of Go as a game people play; it chose the move its networks and search scored highest. Calling it creative, they argue, reads human qualities into a machine — the ELIZA effect on a grand scale.

Why did it work — and who gets the credit?

In 2019 the Association for Computing Machinery gave the 2018 Turing Award, computing’s highest honour, to Bengio, Hinton and LeCun for making deep neural networks a critical component of computing. Debated What mattered most is still argued.

Other views
  • Compute and data: Richard Sutton’s essay “The Bitter Lesson” (2019) argues that, across AI’s history, general methods that exploit growing computation have beaten approaches built on human knowledge. On this view, GPUs and ImageNet were the revolution.
  • Ideas: others stress that without ReLUs, dropout, careful initialisation, residual connections and attention, more compute would not have helped.
  • Credit beyond the three: Schmidhuber and others point to earlier work — LSTM, GPU-trained networks that won contests before AlexNet, and Fukushima’s neocognitron — and to the many students and engineers behind each result.
Quick check

Why were GPUs so important to deep learning?

Chips built for graphics turned out to fit neural-network arithmetic; CUDA (2007) made them programmable for it.

In ILSVRC 2012, what was AlexNet’s top-5 error, against the runner-up’s?

A lead of nearly 11 points, when the previous year’s improvement had been about 2, convinced the vision community.

What made AlphaGo Zero different from the AlphaGo that beat Lee Sedol?

Starting from random play, it surpassed every earlier version — a striking example of learning without human examples.

// Pause and reflect

  1. Fei-Fei Li bet on data, Hinton on algorithms, NVIDIA on chips. Which do you think was the decisive ingredient, and why?
  2. Neural networks were dismissed twice before winning. What does that suggest about judging today’s unfashionable ideas?
  3. Would you call move 37 creative? What would a machine have to do for you to say yes without hesitation?

Your notes are saved privately in this browser.

What happened next

Attention, introduced in 2014 as a helper for LSTMs, was about to become the whole design. In 2017 a Google team published a paper whose title said it all, and the age of large language models began. Continue with Attention Is All You Need.

Key terms on this page

Sources & further reading