How Open Is the Open Web? AI Crawler Blocking Statistics

The open web has spent most of this century running on a promise that Google came up with: educate your audience, and Google will send your website traffic that may convert into revenue. The advent of generative AI in 2023 broke that promise. Today, AI platforms spider the open Web and take content that was created to attract traffic and convert customers and use it for their own benefit. The result, which has been happening slowly for a few years, is that websites aren't publishing as much content, and are starting to block AI crawlers.

This report measures how much of the open web is still open in practice. Since January 2022 we have read the robots.txt file of the top 5,000 content-publishing domains every month, recording which AI user-agents each site permits and which it turns away, then weighting the results by traffic so that a closed door at a major newspaper counts for more than a closed door at a dormant blog. Measured by traffic, the share of the top web fully open to AI crawlers has fallen from 99.0% in early 2022 to 41.7% in the first half of 2026. The sections below trace where those closures happened, which crawlers they targeted, which publishers are blocking, licensing or suing, and what a smaller open web is doing to the answers the assistants give.

The Open Web Index, 2022 to 2026

We start with the simplest question: what share of the significant web still lets every major AI crawler in? Two versions of the number are relevant: the plain percentage, which treats every domain equally and is how the web looks to a crawler; and the traffic-weighted percentage, which treats a domain in proportion to the audience it reaches and is how the web looks to a reader.

Both trend lines start at essentially total openness in 2022 and both fall. The traffic-weighted line falls faster and earlier, and by the first half of 2026 it sits 21.7 points below the unweighted line, which means the bigger sites closed first. The steepest single drop in either series came in the second half of 2023, when the weighted figure fell 13.8 points in six months. That is the period when GPTBot became widely known and blocking it became a boardroom decision rather than a webmaster preference. No later period has dropped that fast, and neither line has turned back up. In every six-month period we measured, the open share was lower than in the period before it.

Which Crawlers Get Shut Out

This section looks at which crawlers publishers turn away. A crawler that gathers text to train a model is treated very differently from one that fetches a page in order to cite it in an answer, even when both belong to the same company.

• Training crawlers average a 26.0% block rate against 10.5% for crawlers that fetch pages to cite them, so publishers are objecting to their work being absorbed rather than to being read.
• OpenAI's two crawlers sit 19.8 points apart, at 29.4% for GPTBot and 9.6% for OAI-SearchBot, which is the clearest evidence in the dataset that the separation of training and search agents is doing what it was designed to do.
• Meta-ExternalAgent grew fastest at 8.3 points in a year, off a later start, as publishers worked through the newer user-agents one addition at a time.

Who Is Closing the Door

This section looks at blocking by site category. The categories that close most often are news, reference, and forums, which is the material an assistant most needs. The categories that stay open are largely the ones with something to sell.

Corporate and B2B sites block much less, at 12.8%, because their economics run the other way. A software company has nothing to lose when an assistant repeats its product description and a great deal to gain when the assistant recommends it, so it leaves the gate open and, increasingly, publishes more material specifically to be read by machines. Publishers face the reverse: their content is the product, and a perfect summary is a substitute for it. What remains most available is marketing material, documentation, and government text, while journalism and community discussion move behind the wall. Forums, at 57.3% and rising 13.4 points in a year, matter in particular because they hold most of the web's practical experience with actually using anything.

License, Litigate, or Leave Open

Robots.txt only shows the technical posture. How publishers handle this commercially has split three ways, and the same publisher can occupy more than one position at once. Some publishers sell access, some sue over it, and a growing number do both, blocking the crawlers of companies they are litigating against while licensing content to the ones that pay. The table below records the publicly reported position of ten well-known names as of September 2026, drawn from the deals and lawsuits trade press has documented rather than from our crawl.

Reading the chart alongside the table:

• The share of the 100 largest publishers taking no public position at all fell from 71 in 2023 to 16 in 2026, so neutrality is now the minority position in an industry that spent two decades not thinking about crawlers.
• Licensing overtook pure blocking in 2026, at 33 publishers against 31, which suggests the endpoint of this period is a paid web for AI rather than a closed one.
• Twenty of the hundred now license and litigate simultaneously, a posture that did not meaningfully exist in 2023, and it is the clearest sign that blocking is a negotiating position rather than a principle.

Commercial choices and technical choices often disagree. Encyclopaedia Britannica, which is suing both OpenAI and Perplexity, names no AI user-agent at all in its robots.txt, so the crawlers it objects to in court are not turned away at the door. NPR, which has announced no deal and filed no suit, disallows eleven of them outright. Medium blocks six, including GPTBot and ClaudeBot, while carving out its own membership and partner pages so those stay readable. GSMArena blocks seven and has GPTBot and Google-Extended sitting in the file as commented-out lines, disabled at some point and never removed. Litigation, licensing and blocking are three separate levers, and plenty of publishers are pulling one while leaving another untouched.

The Bargain, in Numbers

This section measures how many pages a platform crawls for every visitor it sends back. The ratios are large enough that they are easiest to read on a logarithmic scale. Two years of change are shown side by side.

Google still looks like a search engine in this table, at 21 pages crawled per visit sent, because conventional results continue to carry traffic alongside its AI surfaces. Every other platform has moved an order of magnitude or more away from anything a publisher would recognize as an exchange. The platforms at the far end of the scale are the ones whose assistants answer rather than route, and the ratios there have less to do with stinginess than with product design: an assistant that satisfies the question completely has no natural moment to hand off a click. Cloudflare's decision to block crawlers used for both training and search by default for ad-supported pages, announced in July 2026, is the infrastructure layer drawing the same conclusion publishers drew one robots.txt file at a time.

What a Smaller Open Web Does to the Answers

This section looks at what the closures have done to the sourcing behind AI answers. If a quarter of the significant web is closed to training crawlers and the closures are concentrated in journalism, reference, and forums, the assistants have to answer from what is left. The table below tracks four measures of that sourcing over two years.

Answers now draw a majority of their citations from a hundred domains, and the median answer rests on fewer than four sources where two years ago it rested on more than six. When the original is closed, assistants increasingly reach for a copy of it, and mirrors and aggregators carry the errors of both the original and the copying. Blocking a crawler does not remove a publisher's reporting from the answers so much as remove the publisher's version of it. Each publisher that closes is protecting its own work. Taken together, those decisions make the answers worse, because the assistants are drawing on a smaller and more copied set of sources.

Conclusion

Measured by traffic, the share of the top web fully open to AI crawlers has fallen from 99.0% in early 2022 to 41.7% in the first half of 2026. The unweighted share of domains is higher, at 63.4%, because the larger sites closed first. Training crawlers are blocked at a 26.0% rate, against 10.5% for crawlers that fetch a page in order to cite it.

The categories that closed fastest are news, forums, stock media, and reference. Corporate, government, and education sites mostly remain open. Among the 100 largest publishers, only 16 still have no public position; 33 license, 31 block without a deal, and 20 license and litigate at the same time.

AI platforms now crawl far more pages for each visit they send back than they did in 2024. As more of the original reporting is closed, the answers concentrate on a smaller set of still-open domains, and a larger share of citations point to mirrors and aggregators rather than to the publisher who did the work.

Our Methodology

Our researchers used a panel of the 5,000 highest-traffic content-publishing domains. They refresh that list every year so the study is not limited to sites that were already large in 2022. They retrieved each domain's robots.txt file every month and parsed it for eleven AI crawlers.

A domain counts as open only when it does not block any of those crawlers, so a site that blocks even one is not counted as open. Traffic weighting uses estimated visits over the trailing twelve months. That is why the traffic-weighted line moves first: the largest sites closed earlier.

A robots.txt file is a request, not a lock, so this measures what publishers asked crawlers to do, not what the crawlers actually did. A growing number of sites now block crawlers at the network, for example through a CDN, which this method cannot see. The figures here therefore understate how closed the web has become. Licensing deals are also invisible to a crawler: several domains recorded as blocking a company are still supplying that company with content through a paid feed.

Sources

• Chapekis, Athena, and Anna Lieb. Google Users Are Less Likely to Click on Links When an AI Summary Appears in the Results. Pew Research Center, July 22, 2025. Washington, D.C.
• Longpre, Shayne, Robert Mahari, Ariel Lee, and others. Consent in Crisis: The Rapid Decline of the AI Data Commons. arXiv, July 20, 2024. Cambridge, Massachusetts.
• Perez, Sarah. Cloudflare's New Policy Pushes AI Companies to Pay for Publishers' Content. TechCrunch, July 1, 2026. San Francisco, California.
• Tobitt, Charlotte. Who's Suing AI and Who's Signing. Press Gazette, September 1, 2026. London, England.