The Semantic Turn

For about fifteen years, search engines ranked pages by matching the words in a query to the words on a page and by counting links. Between 2011 and 2014, Google, Bing, and Yahoo! added systems that treated people, places, and things as objects with facts attached, and Google rewrote its core algorithm to weigh the meaning of a whole query. In 2012 Google summed up the change as "things, not strings."¹

The same years brought two large ranking updates aimed at low-quality pages and spam, and a shared markup vocabulary that site owners could use to describe their pages to search engines.

1994 to 2010: The Semantic Web Idea and Metaweb

Tim Berners-Lee, the inventor of the World Wide Web, coined the term "Semantic Web" for a web of data that machines can process, and he discussed the need for semantics in the web at the first International World Wide Web Conference in 1994. A 2001 Scientific American article by Berners-Lee, Hendler, and Lassila described an expected evolution of the web into a Semantic Web. In 2006, Berners-Lee and colleagues stated that "this simple idea" still "remains largely unrealized."²

Metaweb Technologies, a San Francisco company co-founded in 2005 by Danny Hillis, Veda Hlubinka-Cook, and John Giannandrea, developed Freebase, which it described as an "open, shared database of the world's knowledge."³ Freebase was a large collaborative knowledge base of structured data harvested from many sources, including user-submitted wiki contributions, and it ran publicly beginning in March 2007.⁴ Google acquired Metaweb in July 2010.³

February 2011: Google Changes Rankings to Reward High-Quality Sites

On February 24, 2011, Amit Singhal, a Google Fellow, and Matt Cutts, a Principal Engineer, announced "a pretty big algorithmic improvement" to Google's ranking that noticeably affected 11.8% of its queries.⁵ The update reduced rankings for low-quality sites, which Google described as sites that add little value for users, copy content from other websites, or are just not very useful. It raised rankings for high-quality sites with original content and information such as research, in-depth reports, and thoughtful analysis.⁵

The post said that some sites would go up and some would go down, and that the team had been tackling these issues for more than a year and working on this specific change for the past few months.⁵ Google also compared the change with data from the Personal Blocklist Chrome extension, which it had launched the week before. The update did not rely on that feedback, but of the top several dozen most-blocked domains in the extension, the algorithm change addressed 84%.⁵ The update launched in the United States first. On April 11, 2011, Google added a note that it had rolled the change out globally to all English-language users and incorporated new signals as it iterated.⁵ The change became known as Panda, after the Google engineer Navneet Panda, who developed the technology that made it possible.⁶

June 2011: Bing, Google, and Yahoo! Launch schema.org

Schema.org is an initiative launched on June 2, 2011, by Bing, Google, and Yahoo!, then the operators of the world's largest search engines. Its purpose is to create and support a common set of schemas for structured data markup on web pages. Yandex, whose search engine is the largest in Russia, joined in November 2011.⁷

Site owners use the schema.org vocabulary with the Microdata, RDFa, or JSON-LD formats to mark up their content with metadata about itself.⁷ The project's own getting-started guide explained the problem with an example. An HTML tag such as a heading reading "Avatar" tells a browser how to display the text, but it does not say whether "Avatar" means the 3D movie or a type of profile picture, which makes it harder for search engines to display relevant content to a user.⁸

Getting Started With schema.org, November 2011

April 2012: Google Announces a Webspam Update

On April 24, 2012, Matt Cutts announced a coming algorithm change aimed at webspam. His post separated what Google called ethical, or white hat, search engine optimization from "black hat webspam." Ethical optimization, he wrote, can make a site more crawlable and its pages easier to find, and it includes keyword research to make sure "the right words are on the page, not just industry jargon that normal people will never type."⁹

Webspam, in Cutts's description, covers techniques that look for shortcuts or loopholes to rank pages higher than they deserve, from keyword stuffing to link schemes. The post included a screenshot of keyword stuffing and a second example, a page with unusual linking patterns in which the outgoing links were unrelated to the text.⁹

Cutts also pointed to an earlier change that year, a page layout algorithm that reduces rankings for sites that do not make much content available above the fold. He wrote that the new algorithm would decrease rankings for sites that Google believed were violating its existing quality guidelines, and that Google would not divulge specific signals because it did not want to give people a way to game the results. His advice to webmasters was to focus on creating high-quality sites that create a good user experience and to use ethical search engine optimization.⁹

Cutts gave the scope. The initial Panda change had affected about 12% of queries to a significant degree, and the new algorithm affected about 3.1% of English queries to a degree a regular user might notice. The change affected roughly 3% of queries in languages such as German, Chinese, and Arabic, and 5% of Polish queries.⁹ The update is known as Penguin.¹⁰

May 2012: Google Launches the Knowledge Graph

Amit Singhal, then Google's senior vice president of engineering, announced the Knowledge Graph on May 16, 2012. He wrote that for more than four decades, search had essentially been about matching keywords to queries, so that to a search engine the words [taj mahal] were just two words. The Knowledge Graph was an "intelligent model" that understood real-world entities and their relationships to one another: "things, not strings."¹

At launch the Knowledge Graph held more than 500 million objects and more than 3.5 billion facts about and relationships between them. It was rooted in public sources such as Freebase, Wikipedia, and the CIA World Factbook and was augmented at much larger scale.¹ The post said that facts were chosen by studying in aggregate what people ask Google about each item. People want to know what books Charles Dickens wrote, for example, but ask more about the buildings Frank Lloyd Wright designed than the books he wrote.¹ Singhal listed three uses: finding the right thing when a word is ambiguous, getting the best summary of key facts, and going deeper and broader. In the first use, the ambiguous query was Taj Mahal the monument or Taj Mahal the musician, and Google could narrow results to the one a person meant. His example of a summary was Marie Curie, where the results show when she was born and died along with her education and scientific discoveries, and where the graph links her to her two children and her husband, Pierre Curie. Singhal wrote that the graph is not just a catalog of objects and that it also models the relationships between them. He added that the facts shown for Tom Cruise answer 37 percent of the next queries people ask about him.¹ Google began a gradual rollout to U.S. English users.¹

Introducing the Knowledge Graph, May 2012

Google first offered it in English. In December 2012 it expanded to Spanish, French, German, Portuguese, Japanese, Russian, and Italian.¹² The Knowledge Graph grew quickly. Within seven months it tripled in size to 570 million entities and 18 billion facts. By mid-2016, Google reported 70 billion facts, and the knowledge boxes appeared for roughly one-third of the 100 billion searches the company handled each month.¹²

October 2012: Wikidata Opens

Wikidata is a collaboratively edited multilingual knowledge graph hosted by the Wikimedia Foundation. It launched in October 2012, and it is developed mainly by Wikimedia Deutschland. Its data is released under the Creative Commons CC0 public domain dedication, and it supplies interlanguage links and some of the infobox data used by Wikipedia and other Wikimedia projects.¹³ The project's creation was funded by donations totaling 1.3 million euros from the Allen Institute for AI, the Gordon and Betty Moore Foundation, and Google. Wikidata was the first new project of the Wikimedia Foundation since 2006, and at its release only the centralization of language links was available.¹³

On December 16, 2014, Google announced that it would shut down Freebase over the following six months and help move the data from Freebase to Wikidata. Freebase was officially shut down on May 2, 2016.⁴

September 2013: Google Announces Hummingbird

Google announced Hummingbird on September 26, 2013, and it had already been in use for about a month.^14,15 Danny Sullivan of Search Engine Land, who reported the announcement, compared the change with a car whose old engine had been swapped for a new one. Panda, Penguin, and other updates had been changes to parts of the old algorithm, like a new oil filter or an improved pump, while Hummingbird was a new engine that continued to use some of the old parts.¹⁴

Google told Sullivan that the name came from being "precise and fast." Singhal, whom Sullivan described as Google's search chief, said that perhaps 2001, when he first joined the company, was the last time Google's algorithm had been so dramatically rewritten. PageRank remained part of the system, as one of over 200 major "ingredients" that go into Hummingbird.¹⁴

Hummingbird places greater emphasis on natural language queries, considering context and meaning over individual keywords.¹⁵ Sullivan illustrated the change with a spoken query such as "What's the closest place to buy the iPhone 5s to my home?" A traditional search engine might look for a page with the words "buy" and "iPhone 5s." Hummingbird, Sullivan wrote, should better focus on the meaning behind the words: that "place" suggests a brick-and-mortar store, and that an "iPhone 5s" is a particular type of device carried by certain stores.¹⁴ Before Hummingbird, Google had applied this kind of conversational search mainly within its Knowledge Graph answers. Hummingbird applied the meaning technology to billions of pages from across the web.¹⁴ Sullivan used another term for these connections, "entity search," and noted that Search Marketing Expo East held a session on it the following week.¹⁴

Google offered before-and-after examples of its own. A search for "pizza hut calories per slice" had listed an answer from another site, and now listed the answer directly from Pizza Hut. A search for "acid reflux prescription" had listed many drugs, and now included information about treatment in general.¹⁴ Sullivan wrote that there was no way for outsiders to run a before-and-after test, so Google's description was the main source. Google said there was nothing new for publishers to worry about: guidance remained to have original, high-quality content, and signals that had been important in the past remained important.¹⁴

Search engine optimization changed with Hummingbird, as web developers and writers were encouraged to use natural language on their sites instead of forced keywords.¹⁵

2014: Google Researchers Describe Knowledge Vault

In 2014, a team of nine Google researchers including Xin Luna Dong published a paper at the KDD conference titled "Knowledge Vault: A Web-Scale Approach to Probabilistic Knowledge Fusion." It opens by noting that large knowledge bases, including Wikipedia, Freebase, Microsoft's Satori, and Google's Knowledge Graph, had multiplied, and that even the largest open-source one, Freebase, was incomplete: 71% of people in it had no known place of birth, and 75% had no known nationality.¹⁶

The paper proposed building a knowledge base automatically from the whole web, combining extractions from text, tabular data, page structure, and human annotations with prior knowledge from existing repositories.¹⁶ The Knowledge Vault stored facts as triples of subject, predicate, and object, each with a confidence score. The team extracted about 1.6 billion candidate triples, and about 271 million of them had an estimated probability of being true above 90%, which the authors called confident facts. About a third of those 271 million were not previously in Freebase.¹⁶ In August 2014, New Scientist reported that Google had launched a Knowledge Vault project. Google then told Search Engine Land that Knowledge Vault was a research report and not an active Google service.¹²

Knowledge Vault: A Web-Scale Approach to Probabilistic Knowledge Fusion

Key Organizations