The Statistical Turn

In the 1990s and 2000s AI quietly changed its method: instead of writing down rules, count patterns in data. You will see why “data beats rules” won, which landmark systems it produced, and how AI ended up hidden inside your inbox, search box and video queue.

Try writing rules that recognise spam. “If the message contains ‘free money’, block it.” Spammers write “fr3e m0ney”. You add a rule; they change again. Within a week you have hundreds of rules and the spammers are still winning. Now try something else: collect thousands of emails people have already marked as spam or not, count which words turn up in each pile, and let the counts decide. That second approach — learning from examples with probability — is the statistical turn. It did not make headlines like the expert-systems boom, but it is why the AI you used every day by 2010 actually worked.

From certainty to probability

Expert systems had struggled with uncertainty; MYCIN needed its own invented “certainty factors” (see Expert Systems). In 1988 Judea Pearl of UCLA offered a principled answer in Probabilistic Reasoning in Intelligent Systems. His Bayesian networks draw causes and effects as a graph — say, rain and sprinkler both pointing to wet grass — and attach probabilities to each link. Evidence entering anywhere updates beliefs everywhere else, following the laws of probability. Pearl received the Turing Award in 2011, and his later work on causality reshaped statistics, medicine and economics.

The bigger shift was in attitude. Researchers stopped asking “what rules does an expert use?” and started asking “what does the data say?” — and, crucially, began measuring every method on shared test sets so that progress could be counted rather than claimed.

Three domains where data beat rules

Rules approach: encode linguists’ knowledge of phonetics and grammar. Statistical approach: model speech as a hidden Markov model — a chain of hidden states (sounds, words) that each produce observable sound signals with certain probabilities — and estimate all those probabilities from hours of recorded, transcribed speech. Frederick Jelinek’s group at IBM, and James and Janet Baker (first at Carnegie Mellon, then at their company Dragon Systems), pursued this from the 1970s. It won: HMM-based recognisers dominated for three decades, and Dragon NaturallySpeaking (1997) brought continuous dictation to ordinary PCs.

Debated The era’s slogan is a remark usually attributed to Jelinek: “Every time I fire a linguist, the performance of the speech recognizer goes up.” The wording and date are uncertain — Jelinek himself recalled saying something milder about linguists leaving the group — and he worked closely with linguists throughout his career. It survives because it captures the mood.

Other views
  • “Data is enough”: engineers pointed to the results. In 2009 Google’s Alon Halevy, Peter Norvig and Fernando Pereira argued in “The Unreasonable Effectiveness of Data” that simple models trained on vast amounts of data tend to beat elaborate models built on less.
  • “Statistics without understanding”: Noam Chomsky and other linguists objected that predicting word sequences explains nothing about how language works, and that success on benchmarks is not science.
  • “Both”: many researchers built hybrid systems — statistical models whose structure was informed by linguistic knowledge — and argue the dichotomy was always overdrawn.

A toolbox of learners

The period produced supervised learning methods with solid mathematics and excellent results on the modest datasets of the day — often better than the neural networks of the time.

Landmarks on the road

  1. 1997Deep Blue beats KasparovOn 11 May 1997 IBM’s Deep Blue won a six-game rematch against world champion Garry Kasparov, 3½–2½, after losing their 1996 match. It was a triumph of specialised chips and brute-force search — around 200 million positions a second — with a hand-tuned evaluation, not of learning. Still, it fulfilled Simon’s 1958 prediction, nearly thirty years late.
  2. 1998PageRank and GoogleSergey Brin and Larry Page ranked web pages by the links pointing to them, treating the web’s own structure as data about importance. Search engines went on to use machine-learned ranking built from billions of clicks.
  3. 2002“A Plan for Spam”Paul Graham’s essay popularised statistical spam filtering; within a few years, learned filters were standard in email services.
  4. 2005Stanley wins the DARPA Grand ChallengeIn 2004 no vehicle finished DARPA’s desert race. In 2005 five did, and Stanford’s Stanley, led by Sebastian Thrun, won the $2 million prize over a course of about 212 km (132 miles). It used machine learning to tell drivable road from rough terrain.
  5. 2006–09The Netflix PrizeNetflix offered $1 million to whoever improved its film recommendations by 10%. Thousands of teams competed; “BellKor’s Pragmatic Chaos” won in 2009 with a huge blend of models. A planned sequel was cancelled after researchers showed that supposedly anonymous viewing data could be linked to real people — an early lesson in data privacy.
  6. 2011Watson wins Jeopardy!In February 2011 IBM’s Watson beat champions Ken Jennings and Brad Rutter on the quiz show, combining hundreds of statistical language-processing components to weigh candidate answers. It also famously answered “Toronto” to a question about US cities.

AI goes undercover

After the second winter, “artificial intelligence” was an embarrassing label. So researchers called their work machine learning, data mining, pattern recognition or statistics — and companies simply shipped it. By the late 2000s, statistical learning was filtering email, ranking search results, recommending products and videos, detecting card fraud, reading postcodes, recognising speech on phone lines and translating web pages. Consensus Much of the field’s success in this period was invisible precisely because it worked: once a technique became reliable, people stopped calling it AI. John McCarthy is often credited with the complaint that as soon as something works, no one calls it AI any more.

Three ingredients made this possible: the web produced oceans of data, cheap computers could process it, and shared benchmarks let researchers prove which methods really worked. Those same three ingredients — data, compute and benchmarks — would soon power a comeback for the neural networks that statistics had sidelined.

Quick check

How did a statistical spam filter decide what was spam?

Naive Bayes filters learn from labelled examples and keep adapting as spammers change wording.

What was the main source of Deep Blue’s strength in 1997?

Deep Blue was a search machine with a hand-tuned evaluation. Learning-based game playing came later, with AlphaGo.

Why did much 1990s–2000s AI stop being called “AI”?

Researchers avoided an over-promised label, and successful techniques became “just software”.

// Pause and reflect

  1. Is a system that predicts language well without “understanding” it doing science, engineering, or both? Does the distinction matter?
  2. The Netflix Prize showed that “anonymous” data can identify people. What does that suggest about the data used to train today’s AI?
  3. Why might a technique stop being called AI once it works? What does that tell us about how we define intelligence?

Your notes are saved privately in this browser.

What happened next

A few researchers — Hinton, LeCun, Bengio and their students — had never given up on deep neural networks. Around 2006 they found new ways to train them, and by 2012 graphics chips and a giant image dataset turned the field upside down. Continue with The Deep Learning Revolution.

Key terms on this page

Sources & further reading