Attention Is All You Need

In 2017 eight researchers proposed a simpler way for neural networks to read language. Within three years it had produced BERT, GPT-3 and a protein-folding breakthrough. Here is what attention is and why it changed everything.

Read this sentence: “The trophy would not fit in the suitcase because it was too big.” What was too big? You knew instantly that “it” means the trophy, and if the last word were “small” you would know it meant the suitcase. To get that right you have to look back across the whole sentence and decide which earlier words matter for the word in front of you. Until 2017 the best language programs did this badly, because they read text the way you would read through a keyhole: one word at a time, carrying a fading memory of what came before. Then a paper with a cheeky title proposed letting every word look at every other word at once. That idea, attention, is the engine inside almost every AI system you use today.

The problem with reading one word at a time

By the mid-2010s, deep learning had conquered images (see The Deep Learning Revolution). Language was harder. The standard tool was the recurrent neural network, which reads a sentence word by word and updates a hidden “memory” vector at each step. In principle that memory can carry information from the start of a paragraph to its end. In practice it leaked: the signal that trains the network shrinks as it passes back through many steps, a problem Sepp Hochreiter analysed in his 1991 diploma thesis. His and Jürgen Schmidhuber’s answer, the long short-term memory network (1997), added gates that decide what to keep and what to forget, and LSTMs powered speech recognition and Google Translate for years.

Two weaknesses remained. First, distant words still blurred together. Second, and more important for what came next, recurrence is sequential: you cannot process word 50 until you have processed words 1 to 49. That made RNNs a poor fit for GPUs, which are fast precisely because they do thousands of sums in parallel. Training was slow, so models stayed small.

A partial fix arrived in 2014. Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio, working on translation, let the decoder “look back” at all the encoder’s states and choose which ones to focus on for each output word. They called it alignment; everyone else called it attention. It worked beautifully, but it was bolted on to an RNN. The radical question of 2017 was: what if attention was not an add-on, but the whole thing?

The 2017 paper

In June 2017 eight researchers posted “Attention Is All You Need” to arXiv: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin. Most were at Google Brain or Google Research; Gomez was a University of Toronto student working at Google Brain. A footnote states that the contributions were equal and the listing order random. The paper was presented at the NeurIPS conference in December 2017.

Its architecture, the Transformer, threw out recurrence entirely. Every word (strictly, every token) is processed at the same time, and layers of attention let each token gather information from all the others. Because word order is no longer implicit, a “positional encoding” is added to each token so the model knows where it sits. The authors tested it on machine translation and set new records on standard English–German and English–French benchmarks, while training far faster than the competition: their big model trained in 3.5 days on eight GPUs.

Consensus Nobody in 2017 claimed this would become the basis of general-purpose chat assistants. The paper is about translation. Its significance became clear only as others applied the same architecture to almost everything.

Attention, step by step

Here is the intuition without the maths. First, text is chopped into tokens. The little demo below shows one way a sentence might be split. Real tokenisers use learned vocabularies, so common words stay whole and rare words break into pieces.

Each token becomes a list of numbers, an embedding. Then attention lets the tokens “talk” to each other. Tap through the chain to follow one token, it, as it works out what it refers to.

Nobody hand-writes the queries, keys and values. They are produced by weights the model learns during training. The numbers in the chain are illustrative; real models spread attention in ways that are often harder to read. (The full mechanism, with multi-head attention and the residual stream, is planned for the Advanced track page Transformers in Depth.)

Pretrain, then adapt: 2018

The second ingredient was a training recipe. Instead of training a fresh model for each task, you first pretrain one big model on a mountain of unlabelled text, then fine-tune it briefly for the job at hand. 2018 was the year this recipe took over natural-language processing.

ELMoELMoAllen Institute for AI, February 2018

Matthew Peters and colleagues trained a two-way LSTM language model and showed that its internal representations, which change depending on the surrounding words, improved almost any language task they were plugged into. “Bank” by a river and “bank” with money finally got different vectors. ELMo was not a Transformer, but it proved the value of pretrained, context-sensitive representations.

GPTGPT-1OpenAI, June 2018

Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever pretrained a Transformer of about 117 million parameters to predict the next word in some 7,000 unpublished books, then fine-tuned it. The paper, “Improving Language Understanding by Generative Pre-Training”, gave the family its name: GPT.

BERTBERTGoogle, October 2018

Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova trained a Transformer to fill in deliberately hidden words using context on both sides. BERT (up to 340 million parameters) broke records across a wide range of language-understanding benchmarks, and in 2019 Google began using it in Search. For a couple of years BERT-style “encoders” and GPT-style “decoders” were rival camps.

T5T5Google, October 2019

Colin Raffel and colleagues recast every task (translation, summarising, answering questions, even classification) as “text in, text out”, and systematically compared training choices. Their largest model had 11 billion parameters, trained on a cleaned web corpus they released as C4.

GPT-2 and the staged-release debate

In February 2019 OpenAI announced GPT-2: the same basic recipe as GPT-1, but with 1.5 billion parameters and trained on about 40 GB of text from web pages linked on Reddit. It could continue a prompt with paragraphs of fluent, if unreliable, prose. Unusually, OpenAI said it would not release the full model at first, citing worries about automated disinformation and spam. It published progressively larger versions over the year and released the full model in November 2019, saying it had seen “no strong evidence of misuse so far”.

Debated The decision split the field, and the argument it started, about who decides when a model is too risky to share, is still running As of 2026.

Other views
  • Responsible caution: supporters said it was healthy to set a norm of thinking about misuse before release, even if this model turned out to be fairly harmless.
  • Hype by another name: critics argued that calling a model “too dangerous to release” was mainly publicity, and that withholding it made independent research harder.
  • Open is safer: others held that openly released models let many more people find and fix weaknesses, a position that resurfaced in the later open-weight debates covered in The Generative Era.

GPT-3, few-shot learning and scaling laws

In January 2020 Jared Kaplan and colleagues at OpenAI published “Scaling Laws for Neural Language Models”. They found that a language model’s test loss fell smoothly and predictably as a power law as you increased three things: parameters, data and compute. These scaling laws turned model-building from guesswork into something closer to engineering: spend more, and you can forecast roughly how much better the model will get.

OpenAI then spent. In May 2020 it described GPT-3, with 175 billion parameters, more than 100 times GPT-2. The paper’s title, “Language Models are Few-Shot Learners”, named its surprise. Without any fine-tuning, you could show GPT-3 a few examples inside the prompt (three English–French word pairs, say) and it would continue the pattern. This few-shot or “in-context” learning had not been designed in; it emerged with scale. GPT-3 was offered through a paid API from June 2020 rather than released, and the paper won a best-paper award at NeurIPS 2020.

Debated Scaling laws describe how loss improves, not which abilities appear or whether scaling alone leads to human-level intelligence. In 2022 DeepMind’s “Chinchilla” paper (Hoffmann and colleagues) also showed that Kaplan’s results had underweighted data: for a fixed budget, smaller models trained on more tokens did better.

Other views
  • Scale is the main road: many researchers at frontier labs read 2018–2020 as evidence that general methods plus more compute keep winning, echoing Richard Sutton’s “Bitter Lesson” (see Lessons from History).
  • Scale is not understanding: critics such as Gary Marcus, and the 2021 “Stochastic Parrots” paper by Emily Bender, Timnit Gebru and colleagues, argued that bigger text predictors do not acquire grounded meaning or reliable reasoning, and carry real costs in bias and energy.
  • Necessary but not sufficient: a middle view holds that scale unlocks a lot, but new ideas (better data, training methods, reasoning techniques) are needed too; many read the later turn to reasoning models as support for this view.

Beyond words: vision and proteins

Attention does not care what the tokens are. In October 2020 Alexey Dosovitskiy and colleagues at Google posted “An Image is Worth 16x16 Words”: chop a picture into 16-by-16-pixel patches, treat each patch as a token, and a plain Transformer, the Vision Transformer, matches the best convolutional networks once it is trained on enough data. One architecture was starting to serve text, images, audio and more.

The most dramatic result came from biology. Predicting the 3D shape a protein folds into from its sequence of amino acids had been an open problem for half a century, judged every two years by the CASP competition. At CASP14 in late 2020, DeepMind’s AlphaFold 2, led by John Jumper with Demis Hassabis, reached a median score of about 92 on the 0–100 GDT scale, accuracy comparable to experimental methods for many targets. Its design leaned heavily on attention to relate every part of a protein to every other. The method was published in Nature in 2021 together with open code, and a database of predicted structures followed. In 2024 Hassabis and Jumper shared half of the Nobel Prize in Chemistry; the other half went to David Baker for computational protein design.

Quick check

What was the main practical advantage of the Transformer over recurrent networks?

Recurrence forces word-by-word processing. Attention lets every token be handled at once, which suits parallel hardware and made far larger models affordable.

What did GPT-3 show that surprised people?

Few-shot, in-context learning emerged at scale. The Vision Transformer and AlphaFold 2 were separate 2020 results.

In attention, what does a token’s “query” represent?

Queries are matched against other tokens’ keys; the best matches decide whose values get mixed in.

// Pause and reflect

  1. The Transformer paper was about translation, yet its biggest impact was elsewhere. What does that suggest about predicting which research will matter?
  2. Was OpenAI right to stage GPT-2’s release? Would your answer change for a far more capable model?
  3. Scaling laws made progress more predictable. Does predictable progress make a technology easier or harder for society to govern?

Your notes are saved privately in this browser.

What happened next

By the end of 2020 the pieces were in place: an architecture that scaled, a recipe of pretraining then adapting, and evidence that more compute meant better models. What was missing was a way to make these models follow instructions, and a product that put them in front of everyone. That story, from DALL·E to ChatGPT and beyond, continues in The Generative Era.

Key terms on this page

Sources & further reading