Machines That Read

For most of its first two decades, a search engine ranked pages by matching the words in a query against the words on a page and then weighing links and other signals. From 2015 to 2021 that changed. Google and Microsoft put neural networks into ranking, researchers published language models that could write fluent paragraphs, and a few papers described systems that read documents and answered from them. This chapter follows that work, which built the technical base for the generated answers in the chapters that follow. It picks up where The Semantic Turn left off, with search engines trying to interpret meaning, and it overlaps with the voice assistants in The Voice Interlude.

October 2015: Google Confirms RankBrain

Google confirmed on October 26, 2015, that it was using a machine learning system called RankBrain to help process search results. RankBrain sorts queries into word vectors, which are lists of numbers that sit close together when the words they stand for are close in meaning. When it meets a word or phrase it does not recognize, it guesses which known words and phrases mean something similar and adjusts the results to match. That made it better at queries nobody had typed before. It also read small words such as "the," "and," and "without," which earlier systems had ignored and which can change what a person means.¹ Google said in 2019 that 15 percent of the billions of searches it sees each day are ones it has not seen before.²

In a 2015 interview, Google described RankBrain as the third most important factor in its ranking algorithm, after links and content. In a test, RankBrain guessed which page the rest of Google's ranking system would put first 80 percent of the time, and human search engineers guessed it 70 percent of the time.¹ Barry Schwartz of Search Engine Land wrote in 2019 that "Google has told us SEOs can't really optimize for RankBrain."³

June 2017: The Transformer

On June 12, 2017, eight researchers, Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin, posted a paper called "Attention Is All You Need" to arXiv. It proposed a network design called the Transformer, which relies only on attention mechanisms and drops the recurrent and convolutional layers that earlier language models used.⁴ Microsoft later gave a plain description of the difference: unlike earlier networks that processed words one at a time in order, a transformer weighs the context and relationship between each word and all the words around it in a sentence.⁵

The paper reported a score of 28.4 BLEU on the WMT 2014 English-to-German translation task, more than 2 points above the best earlier results, and 41.8 BLEU on English-to-French after 3.5 days of training on eight GPUs.⁴ Google later described the Transformer as something its research group invented and open-sourced in 2017.⁶ BERT, GPT-3, and LaMDA are all built on it.^7,8,6

The Transformer Architecture From "Attention Is All You Need"

October 2018: The BERT Paper

Four researchers at Google AI Language, Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, posted a paper on October 11, 2018, that introduced BERT, short for Bidirectional Encoder Representations from Transformers. BERT is trained on large amounts of unlabeled text by reading each word in light of the words on both sides of it. Once trained, it can be adapted to a specific task, such as question answering, by adding one output layer.⁷

The paper reported new state-of-the-art results on eleven language tasks, including a GLUE score of 80.5 percent, an absolute improvement of 7.7 points, and an F1 score of 93.2 on version 1.1 of the SQuAD question answering test. It described two sizes, BERT-base with 110 million parameters and BERT-large with 340 million.⁷ Google released the model as open source.³

The BERT Paper, First Page

February 2019: GPT-2

OpenAI announced GPT-2 on February 14, 2019. It followed a June 11, 2018, paper from OpenAI that introduced the first generative pre-trained transformer, or GPT. GPT-2 has 1.5 billion parameters and was pre-trained on a dataset of 8 million web pages. OpenAI first released only a smaller version, then a version with 774 million parameters on August 20, 2019, and the full model on November 5, 2019. The staged release followed concerns that the model could be widely misused, and by November 2019 OpenAI said it had "seen no strong evidence of misuse so far."⁹

October and November 2019: BERT Comes to Google and Bing

On October 25, 2019, Pandu Nayak, a Google Fellow and vice president of Search, wrote that Google was applying BERT to ranking and to featured snippets. Google said it would help with 1 in 10 searches in the United States in English, and that serving the model required Cloud TPUs, Google's own AI chips, for the first time in Search.² Nayak gave two examples. In the query "2019 brazil traveler to usa need a visa," the word "to" shows that a Brazilian is traveling to the United States, and earlier systems had missed it. In "do estheticians stand a lot at work," the older approach matched "stand" to "stand-alone." He added that Google hoped people could "let go of some of your keyword-ese" and search in a way that feels natural.²

Google Announces BERT in Search

Schwartz called it the largest change to Google's search system since RankBrain, almost five years earlier. He added that Google said BERT was already being used in featured snippets in all languages. To the question of whether sites could optimize for it, he answered: "It is unlikely." His advice was "Just write content for users, like you always do."³

Microsoft described similar work on November 18, 2019. Jeffrey Zhu of the Bing team wrote that Bing had used large transformer models since April of that year, that the models were now applied to every Bing search query worldwide, and that serving them took more than 2,000 Azure GPU virtual machines across four regions, handling over 1 million BERT inferences per second. His example was the query "what can aggravate a concussion," where the word "aggravate" shows that the person wants to know what to do after a concussion and not what causes one.⁵

May 2020: GPT-3 and Retrieval-Augmented Generation

On May 28, 2020, OpenAI researchers led by Tom Brown posted "Language Models are Few-Shot Learners," which described GPT-3, a model with 175 billion parameters, ten times more than any earlier non-sparse language model. The paper showed that a large enough model could perform new tasks from a few examples written into the prompt, without any further training.¹⁰ OpenAI opened an API beta on June 11, 2020, and on September 22, 2020, Microsoft announced that it had licensed GPT-3 exclusively. Others could still use the output of the public API, and only Microsoft could access the underlying model.⁸

Six days before the GPT-3 paper, on May 22, 2020, a team including Patrick Lewis from Facebook AI Research, University College London, and New York University posted a paper on retrieval-augmented generation, or RAG. The system paired a language model with a dense vector index of Wikipedia and a neural retriever that looked up passages for each question before the model wrote its answer. The authors noted that giving provenance for a model's decisions and updating its knowledge were still open problems, and RAG was a step toward both.¹¹ Later chapters describe products that work this way, retrieving documents first and writing from them.

October 2020: BERT in Almost Every Query, and Passage Ranking

On October 15, 2020, Prabhakar Raghavan of Google said BERT was now used in almost every English-language query. The same announcement described a new spelling system built on a deep neural network, introduced because 1 in 10 daily queries are misspelled, and a change to passage ranking. Google said that by judging the relevance of a specific passage in addition to the page as a whole, it could find an answer buried deep in a long page, and that the change would improve 7 percent of queries across all languages.¹² Passage ranking meant that a single paragraph of a page could be selected for a query on its own.

May 2021: MUM, LaMDA, and "Rethinking Search"

On May 18, 2021, Google introduced two models. Nayak described MUM, the Multitask Unified Model, as built on the T5 text-to-text framework and 1,000 times more powerful than BERT. It was trained across 75 languages, could take in images as well as text, and could generate language as well as understand it. His example was a hiker who has climbed Mt. Adams and wants to prepare for Mt. Fuji, which today means many separate searches on elevation, weather, trails, and gear. Google found that people issue eight queries on average for complex tasks like that, and MUM was meant to cut the number of searches needed.¹³ Eli Collins and Zoubin Ghahramani described LaMDA, a model built on the Transformer and trained for open-ended conversation.⁶

A month later, in the June 2021 issue of ACM SIGIR Forum, four Google Research scientists, Donald Metzler, Yi Tay, Dara Bahri, and Marc Najork, published "Rethinking Search: Making Domain Experts out of Dilettantes." The paper observed that classical search systems return references to answers instead of answers, and that language models can write answers directly but "are dilettantes" that are "prone to hallucinating" and "incapable of justifying their utterances by referring to supporting documents." It proposed combining the two. A footnote on the first page states: "This is a research proposal, not the roadmap for any Google product or service."¹⁴

"Rethinking Search," First Page

December 2021: WebGPT

On December 17, 2021, OpenAI researchers led by Reiichiro Nakano posted a paper on WebGPT. The team fine-tuned GPT-3 to answer long questions by using a text-based web browser that could search and follow links. To make accuracy easier for people to check, the model had to collect references while it browsed in support of each answer. On a set of questions taken from Reddit, human raters preferred its best answers to those written by human demonstrators 56 percent of the time, and to the highest-voted Reddit answer 69 percent of the time.¹⁵

By the end of 2021 the pieces of a search engine that answers in prose were all published. Retrieval, ranking by passage, a model that writes fluent text, and answers that attach references each existed in a research paper or a product. The release that brought them to the public came on November 30, 2022, and is the subject of The ChatGPT Shock.

Key Organizations