AI Crawler Statistics 2026: How Many Websites Block GPTBot, ClaudeBot and CCBot
The story everyone has been telling for two years is that publishers are shutting the door on AI. Our crawl says something narrower: most of the web has not made a decision at all, and most of the sites that appear to have made one were opted in by a vendor. This time we can name the vendor, count its share, and count the sites that decided for themselves.
- Part 1: Robots.txt statistics 2026
- Part 2: AI crawler statistics 2026
- Part 3: llms.txt statistics 2026
On this page 12 sections
- How many websites block GPTBot
- The most blocked AI crawlers
- Cloudflare wrote most of the blocks
- What the Cloudflare block actually declares
- Deliberate decisions are rare, and they lean towards allowing
- Training blocked, answering allowed: OpenAI's three crawlers
- Do the biggest websites block AI crawlers
- Where the blocking companies are
- Bigger companies block more
- 54,110 websites block a crawler that no longer exists
- Frequently asked questions
- Methodology and sources
- 6,250,586 websites block GPTBot, 6.88% of the 90.8 million with a robots.txt and 59.4% of the sites that name it. Both figures are correct and they are routinely confused.
- 87.0% of the top 1,000 websites that serve a robots.txt allow GPTBot, and so do 86.9% of the top 10,000. The most visited sites block GPTBot about twice as often as the web as a whole, and seven in eight still allow it.
- The sites Cloudflare's addresses answer for block GPTBot at 23.0% and every other site at 1.5%, so 84.0% of all GPTBot blocks sit behind Cloudflare. By nameserver the split is 26.5% against 1.3%, among the sites whose DNS provider is known.
- The eight most blocked AI crawlers are blocked within about half a million sites of each other, and 96.0% of the sites carrying Cloudflare's Content-Signals block are on Cloudflare nameservers. The cluster is one managed file.
- 4.3 million websites name GPTBot and allow it, against 6.3 million that name it and block it. Two in five of the sites that wrote a rule for GPTBot wrote an allow.
- 5.9 million websites block OpenAI's training crawler while allowing its search and user-fetch crawlers. Only 13,850 do the reverse.
- 231 thousand websites declare ai-train=yes in a Content-Signals block, an explicit opt-in to AI training. 3.1 million websites with a Content-Signals block do not say ai-train=no at all.
- 54,110 websites block Anthropic's retired anthropic-ai and Claude-Web agents but not ClaudeBot, so the current crawler is allowed. chatgpt.com is one of them.
- Googlebot is blocked by 0.17% of sites with a robots.txt and GPTBot by 6.88%. Websites want free traffic and do not want their content used for training with nothing in return, the platforms that write their files have started acting on it, and the gap is likely to widen.
- Companies headquartered in Turkey, India and the Emirates block GPTBot at two to four times the rate of German, Italian and Swiss companies. The United States sits mid-table by headquarters and tops the table by server location only because Cloudflare's edge geolocates there.
How many websites block GPTBot
6.88% of websites with a robots.txt block GPTBot: 6,250,586 of the 90.8 million sites that serve one. Among the 10.5 million sites that name GPTBot at all, 59.4% block it.
Across every live website, all 151.6 million of them, 4.1% block GPTBot, because a site with no robots.txt allows every crawler.
Those three percentages describe one fact and sit tens of points apart. Nearly every published AI blocking figure is one of them without saying which. In this report "the web" means the 90.8 million sites that publish a robots.txt, and every share is a share of them unless it says otherwise.
The most blocked AI crawlers
| Agent | Based in | What it does | Sites blocking it | Share of the web | Of sites naming it | Sites naming it |
|---|---|---|---|---|---|---|
 GPTBot | USA | Model training | 6,250,586 | 6.88% | 59.4% | 10,519,366 |
 CCBot | USA | Open archive | 6,172,267 | 6.80% | 61.6% | 10,013,392 |
 Amazonbot | USA | Training, Alexa | 6,124,914 | 6.75% | 62.7% | 9,773,153 |
 Bytespider | China | Model training | 6,070,796 | 6.69% | 62.4% | 9,728,016 |
 ClaudeBot | USA | Model training | 5,965,523 | 6.57% | 59.3% | 10,067,741 |
 Google-Extended | USA | Gemini opt-out | 5,877,142 | 6.47% | 59.4% | 9,900,533 |
 Applebot-Extended | USA | Apple AI opt-out | 5,768,936 | 6.35% | 61.1% | 9,446,219 |
 Meta-ExternalAgent | USA | Model training | 5,737,489 | 6.32% | 60.9% | 9,424,699 |
| ChatGPT-User | USA | User fetch | 391,829 | 0.43% | 30.1% | 1,300,186 |
| anthropic-ai | USA | Retired for ClaudeBot | 385,956 | 0.43% | 9.3% | 4,134,428 |
 PerplexityBot | USA | Answer engine | 340,338 | 0.37% | 25.5% | 1,335,293 |
 Omgilibot | Israel | Data resale | 314,358 | 0.35% | 8.8% | 3,560,105 |
 YouBot | USA | Answer engine | 299,536 | 0.33% | 8.4% | 3,554,806 |
 cohere-ai | Canada | Model training | 258,866 | 0.29% | 6.9% | 3,736,810 |
 ImagesiftBot | USA | Image datasets | 257,348 | 0.28% | 67.3% | 382,634 |
| Claude-Web | USA | Retired for ClaudeBot | 257,160 | 0.28% | 32.4% | 794,609 |
| FacebookBot | USA | Model training | 235,399 | 0.26% | 6.4% | 3,704,505 |
 AIHitBot | UK | Company data | 199,925 | 0.22% | 6.0% | 3,330,277 |
 Diffbot | USA | Web extraction | 176,394 | 0.19% | 38.9% | 453,367 |
| OAI-SearchBot | USA | ChatGPT search | 130,684 | 0.14% | 14.2% | 917,967 |
 img2dataset | Image datasets | 39,218 | 0.04% | 1.2% | 3,199,647 |
Embed this figure
The eight most blocked AI crawlers sit within about half a million sites of each other. Of the 6.2 million sites that block CCBot, Common Crawl's fetcher, 5.7 million also block GPTBot, ClaudeBot, Bytespider and Google-Extended in the same file. 5.9 million sites block eight or more named crawlers.
Sites do not arrive at identical eight-item block lists on their own. Something wrote the list for them, and the next section names it.
| Agent | Sites allowing it | Sites blocking it | Allow | Block | Sites naming it |
|---|---|---|---|---|---|
| ImagesiftBot | 125,286 | 257,348 | 32.7% | 67.3% | 382,634 |
| Amazonbot | 3,648,239 | 6,124,914 | 37.3% | 62.7% | 9,773,153 |
| Bytespider | 3,657,220 | 6,070,796 | 37.6% | 62.4% | 9,728,016 |
| CCBot | 3,841,125 | 6,172,267 | 38.4% | 61.6% | 10,013,392 |
| Applebot-Extended | 3,677,283 | 5,768,936 | 38.9% | 61.1% | 9,446,219 |
| Meta-ExternalAgent | 3,687,210 | 5,737,489 | 39.1% | 60.9% | 9,424,699 |
| GPTBot | 4,268,780 | 6,250,586 | 40.6% | 59.4% | 10,519,366 |
| Google-Extended | 4,023,391 | 5,877,142 | 40.6% | 59.4% | 9,900,533 |
| ClaudeBot | 4,102,218 | 5,965,523 | 40.7% | 59.3% | 10,067,741 |
| Diffbot | 276,973 | 176,394 | 61.1% | 38.9% | 453,367 |
| Claude-Web | 537,449 | 257,160 | 67.6% | 32.4% | 794,609 |
| ChatGPT-User | 908,357 | 391,829 | 69.9% | 30.1% | 1,300,186 |
| PerplexityBot | 994,955 | 340,338 | 74.5% | 25.5% | 1,335,293 |
| OAI-SearchBot | 787,283 | 130,684 | 85.8% | 14.2% | 917,967 |
| anthropic-ai | 3,748,472 | 385,956 | 90.7% | 9.3% | 4,134,428 |
| Omgilibot | 3,245,747 | 314,358 | 91.2% | 8.8% | 3,560,105 |
| YouBot | 3,255,270 | 299,536 | 91.6% | 8.4% | 3,554,806 |
| cohere-ai | 3,477,944 | 258,866 | 93.1% | 6.9% | 3,736,810 |
| FacebookBot | 3,469,106 | 235,399 | 93.6% | 6.4% | 3,704,505 |
| AIHitBot | 3,130,352 | 199,925 | 94.0% | 6.0% | 3,330,277 |
| img2dataset | 3,160,429 | 39,218 | 98.8% | 1.2% | 3,199,647 |
Embed this figure
Once a website has written a rule for a crawler, the decision splits two ways, and the split is the chart above. Amazonbot, Bytespider and CCBot are blocked by more than three in five of the sites that name them, and the chart keeps to crawlers named by at least a million websites, so a crawler almost nobody has heard of cannot top it on a handful of mentions. GPTBot and ClaudeBot sit at 59.4% and 59.3%.
The answering crawlers are much lower: ChatGPT-User at 30.1%, PerplexityBot at 25.5% and OAI-SearchBot at 14.2%. Sites that write a rule for a training crawler mostly block it. Sites that write a rule for a crawler that answers a person's question mostly allow it.
Two entries in the chart are not crawlers. Google-Extended and Applebot-Extended fetch nothing. Googlebot and Applebot do the crawling, and the token tells them whether the pages may be used to train Gemini or Apple Intelligence. Google's documentation says Google-Extended "does not impact a site's inclusion in Google Search".
So 5.9 million sites refuse Google-Extended and 150.9 thousand refuse Googlebot. Same company, same file, 38.9 times the take-up. Saying no to Google-Extended costs a publisher nothing in search. Saying no to Googlebot costs the search traffic itself.
That is the whole bargain in one comparison. Googlebot is blocked by 0.17% of sites with a robots.txt and GPTBot by 6.88%: websites want free traffic, they do not want their content used for training with nothing offered in return, and the platforms that write their files have started acting on it. Until a training crawler sends visitors, expect that gap to widen.
Cloudflare wrote most of the blocks
Embed this figure
Two cuts say the same thing. By the address that answered, the 22.8 million sites behind Cloudflare block GPTBot at 23.0% and the 68 million sites everywhere else at 1.5%, so 84.0% of all 6.3 million GPTBot blocks sit behind Cloudflare. By nameserver, which is known for 34.7 million sites, the ones on Cloudflare block at 26.5% and the ones on every other known provider at 1.3%, and 96.0% of the Content-Signals sites with a known provider are on Cloudflare nameservers.
Take the sites Cloudflare answers for out and the sentence for the rest of the internet becomes: about one website in 68 blocks GPTBot.
That is the shape of a concentrated market rather than a popular opinion. The hosting report counts the same networks by the domains resolving into them and finds Cloudflare in front of 12.8% of everything that resolves, 37,403,719 domains, which is how one default reaches millions of sites that never changed a setting.
| Nameservers | Block GPTBot |
|---|---|
| On Cloudflare | 26.5% |
| Everywhere else | 1.3% |
Embed this figure
| DNS provider | Sites | Block GPTBot | Carry Content-Signals | Closed to unlisted crawlers | Publish llms.txt | Declare a sitemap | Sites blocking GPTBot |
|---|---|---|---|---|---|---|---|
 Cloudflare | 13,193,204 | 26.5% | 40.2% | 4.8% | 10.2% | 55.3% | 3,501,476 |
 AWS Route 53 | 750,955 | 4.9% | 1.5% | 6.5% | 17.5% | 72.0% | 36,647 |
 Namecheap | 1,358,817 | 4.0% | 3.8% | 7.5% | 20.8% | 72.0% | 55,032 |
 WordPress | 1,051,913 | 3.1% | 0.2% | 13.0% | 3.7% | 84.0% | 32,504 |
 OVH | 769,142 | 1.3% | 0.7% | 1.9% | 10.2% | 80.5% | 9,999 |
 GoDaddy | 9,995,855 | 1.0% | 1.1% | 2.5% | 36.2% | 89.5% | 97,959 |
 Google | 2,117,352 | 0.9% | 0.9% | 7.2% | 39.2% | 84.0% | 18,633 |
| NS1 | 1,465,412 | 0.5% | 0.4% | 0.8% | 3.6% | 96.5% | 7,767 |
 Wix | 3,797,912 | 0.2% | 0.2% | 0.1% | 90.7% | 99.4% | 7,596 |
Embed this figure
Cloudflare ships a managed robots.txt that writes the Content Signals Policy preamble and a block list of AI crawlers into the file, and it can be turned on for a whole account in one setting. GoDaddy, Wix, Squarespace and Google, the other providers that write robots.txt for customers by the million, all sit at or under one percent. Their templates do other things, and none of them blocks an AI crawler by default.
A default that nobody turns off is still a preference, and Cloudflare shipping it changed the internet's posture faster than any publisher campaign did. But the headline number is usually cited as several million separate editorial decisions, and it is not that. The share of the web that blocks AI crawlers will move with the customer counts of a few infrastructure companies, not with publisher opinion, and the next big jump will come from whichever platform switches a block on by default.
What the Cloudflare block actually declares
| Declared in a Content-Signals block | Sites | Share of files |
|---|---|---|
| Any Content-Signals block | 8,621,742 | 9.5% |
| ai-train=no | 5,502,845 | 6.1% |
| ai-train=yes | 231,034 | 0.3% |
| search=yes | 5,731,530 | 6.3% |
| search=no | 1,891 | 0.0% |
| ai-input=yes | 292,483 | 0.3% |
| ai-input=no | 13,464 | 0.0% |
| A block that does not say ai-train=no | 3,119,175 | 3.4% |
Embed this figure
The Content-Signals block is usually described as "blocking AI". 5.5 million websites say ai-train=no, but 3.1 million websites with a Content-Signals block do not say it at all, and 231 thousand say ai-train=yes: an explicit, machine-readable opt-in to model training.
6.3% of the web says search=yes, 5.7 million of them, and only 1,891 say search=no. A file that is read as a refusal and does not contain one is the same problem part three finds in llms.txt, from the other end.
Deliberate decisions are rare, and they lean towards allowing
Strip out the managed block and a much smaller population appears. It is the one worth quoting.
4.7% of the web, 4,268,780 websites, names GPTBot in a rule that allows it, against the 6,250,586 that name it and block it. A crawler that is named in the file follows its own rules and ignores User-agent: *, so a GPTBot rule that does not deny the whole site is a permission somebody typed. Two in five of the sites that wrote a rule for GPTBot wrote an allow.
ClaudeBot is allowed by name on 4.1 million sites and Google-Extended on 4 million.
Sites that distinguish one AI crawler from another are rarer still. 226,507 block CCBot while allowing GPTBot and ClaudeBot, a position against a research archive but not the model builders. 354,854 do the opposite. Together they are 0.2% and 0.4% of the web. Roughly one website in 150 has drawn a line between AI companies. Everyone else blocks all of them or none.
Where a website blocks exactly one AI crawler, the choice is telling. 737.7 thousand websites do it: GPTBot on 88,455, Bytespider on 43,092, Meta-ExternalAgent on 41,931, Amazonbot on 40,133, ClaudeBot on 9,976 and PerplexityBot on 2,531. Bytespider's place near the top is notable: a block aimed only at ByteDance's crawler is a statement about one company rather than about AI training.
Training blocked, answering allowed: OpenAI's three crawlers
OpenAI runs three crawlers: GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User, which fetches a page because a person asked a question. OpenAI asks sites to treat them separately, and the internet does.
Embed this figure
| Position on OpenAI's three agents | Sites |
|---|---|
| Refuse GPTBot, admit OAI-SearchBot and ChatGPT-User | 5,861,965 |
| Refuse all three | 132,878 |
| Admit GPTBot, refuse the answering agents | 13,850 |
Embed this figure
6.5% of the web, 5,861,965 websites, blocks GPTBot while allowing both OAI-SearchBot and ChatGPT-User. 132,878 block all three. Only 13,850 allow the trainer and block the answering crawlers.
Most of the first group is the Cloudflare default, whose managed list blocks training crawlers and leaves search and user-fetch crawlers alone. Take out every file with a Content-Signals block and the shape holds: of the 82.2 million files without one, 437,463 block GPTBot while allowing both answering crawlers, 127,556 block all three and 13,695 do the reverse. Sites are not shutting down, they are separating training from answering. Websites are pricing the trade one crawler at a time. A fetch that brings a reader is worth allowing; a copy taken for training is not, and the line between the two is where the next robots.txt rules will be written.
Do the biggest websites block AI crawlers
| Popularity tier | Block GPTBot | Sites blocking | Sites in tier |
|---|---|---|---|
| Top 1,000 | 13.0% | 60 | 463 |
| Top 10,000 | 13.1% | 693 | 5,271 |
| Top 100,000 | 13.9% | 7,347 | 52,724 |
| Top 1 million | 12.9% | 72,457 | 562,393 |
| Entire internet | 6.9% | 6,250,586 | 90,797,323 |
Embed this figure
The gradient is real and it is about double. 13.0% of the 463 top-1,000 websites that serve a robots.txt block GPTBot, against 6.9% across every site with one. Put the other way round: 87.0% of those top-1,000 sites allow GPTBot, and so do 86.9% of the top 10,000, and the share of all 1,000 is higher still, because a site without a file allows it too.
| Agent | Top 100k | Top 1M | Whole internet | Concentration |
|---|---|---|---|---|
| GPTBot | 13.8% | 12.8% | 6.9% | 2.0x |
| CCBot | 13.2% | 12.6% | 6.8% | 1.9x |
| Amazonbot | 11.2% | 11.6% | 6.8% | 1.7x |
| Bytespider | 12.5% | 12.0% | 6.7% | 1.9x |
| ClaudeBot | 12.2% | 11.7% | 6.6% | 1.8x |
| Google-Extended | 11.6% | 11.2% | 6.5% | 1.8x |
| Applebot-Extended | 10.8% | 10.8% | 6.4% | 1.7x |
| Meta-ExternalAgent | 11.1% | 10.8% | 6.3% | 1.8x |
| ChatGPT-User | 3.0% | 2.0% | 0.4% | 6.9x |
| anthropic-ai | 3.2% | 1.9% | 0.4% | 7.4x |
| PerplexityBot | 2.9% | 1.6% | 0.4% | 7.9x |
| Omgilibot | 2.4% | 1.4% | 0.4% | 6.9x |
| YouBot | 2.3% | 1.3% | 0.3% | 7.0x |
| cohere-ai | 2.5% | 1.4% | 0.3% | 8.8x |
| ImagesiftBot | 2.0% | 1.4% | 0.3% | 7.2x |
| Claude-Web | 2.5% | 1.6% | 0.3% | 9.1x |
| FacebookBot | 2.4% | 1.2% | 0.3% | 9.0x |
| AIHitBot | 0.6% | 0.4% | 0.2% | 2.5x |
| Diffbot | 2.4% | 1.1% | 0.2% | 12.5x |
| OAI-SearchBot | 1.6% | 0.7% | 0.1% | 11.6x |
Embed this figure
What large sites single out is not training. PerplexityBot is blocked by 0.4% of sites with a robots.txt and 2.9% of the top hundred thousand sites. ChatGPT-User goes from 0.4% to 3.0%, seven to eight times the whole-web rate. The training crawlers double by comparison: GPTBot goes from 6.9% to 13.8%.
Those two are answer engines. They fetch a page because a person asked and show the answer instead of the link. A training crawler takes a copy once. An answer engine takes the reader every time, and the sites with the most readers to lose noticed first. The next fight over robots.txt is about answer engines, not training, and the largest publishers are already fighting it.
Where the blocking companies are
| Company headquarters | Sites | Block GPTBot | Block CCBot | Sites blocking GPTBot |
|---|---|---|---|---|
| Turkey | 96,885 | 8.6% | 8.6% | 8,293 |
| India | 349,095 | 6.5% | 6.6% | 22,796 |
| UAE | 74,270 | 6.4% | 6.4% | 4,776 |
| Brazil | 208,984 | 5.8% | 5.8% | 12,100 |
| UK | 518,598 | 4.7% | 4.6% | 24,582 |
| Canada | 244,159 | 4.7% | 4.6% | 11,500 |
| Australia | 252,659 | 4.4% | 4.5% | 11,041 |
| USA | 1,830,876 | 4.2% | 4.2% | 77,080 |
| Mexico | 70,596 | 4.2% | 4.2% | 2,951 |
| Poland | 70,728 | 4.0% | 4.0% | 2,843 |
| Spain | 184,138 | 3.3% | 3.3% | 6,003 |
| Netherlands | 228,735 | 3.1% | 3.0% | 7,159 |
| France | 281,949 | 2.8% | 2.8% | 7,866 |
| Switzerland | 96,106 | 2.7% | 2.5% | 2,624 |
| Italy | 188,369 | 2.5% | 2.6% | 4,766 |
| Germany | 306,717 | 2.2% | 2.2% | 6,656 |
Embed this figure
Matching sites to company records gives a cut the popularity ranking cannot: who owns the site. Turkish, Indian and Emirati companies block GPTBot at 8.6%, 6.5% and 6.4%. German, Italian and Swiss companies sit at 2.2%, 2.5% and 2.7%. American companies are in the middle at 4.2%.
Whatever drives the decision, it is not the volume of national debate about AI and copyright.
A cut by server location gives a different answer, and it is wrong in an instructive way. By the country of the IP address that answered, the United States blocks GPTBot at 10.8% and every other country sits under four percent. Cloudflare's edge answers for sites all over the world and its addresses geolocate to the United States, so the server-country cut measures the Cloudflare default a second time. The headquarters chart uses the company's own stated location and is immune to that.
Bigger companies block more
| Company headcount | GPTBot | CCBot | Cloudflare signals | Googlebot |
|---|---|---|---|---|
| 1 person | 4.0% | 4.0% | 4.7% | 0.0% |
| 2 to 10 | 4.7% | 4.7% | 5.8% | 0.0% |
| 11 to 50 | 4.9% | 4.8% | 6.0% | 0.0% |
| 51 to 200 | 5.0% | 4.9% | 5.8% | 0.0% |
| 201 to 500 | 5.2% | 5.1% | 5.8% | 0.0% |
| 501 to 1,000 | 5.3% | 5.2% | 5.9% | 0.0% |
| 1,001 to 5,000 | 5.2% | 5.1% | 5.5% | 0.0% |
| 5,001 to 10,000 | 5.5% | 5.1% | 6.0% | 0.1% |
| over 10,000 | 6.5% | 6.3% | 6.9% | 0.0% |
Embed this figure
Every step up in company size raises the share that blocks GPTBot, from 4.0% at one-person companies to 6.5% above ten thousand staff. The Content-Signals block follows the same curve, from 4.7% to 6.9%, which is enterprise plans carrying enterprise defaults.
Googlebot blocking stays under 0.03% at every size, and that flat line is the control. Whatever rises with company size, it is not a general appetite for shutting crawlers out. The likeliest reason is a legal team, which writes rules about training data, not about search.
54,110 websites block a crawler that no longer exists
Anthropic retired the anthropic-ai and Claude-Web user agents in favour of ClaudeBot, and its documentation names only the new one. The old names are still everywhere.
0.43% of sites with a robots.txt, 385,956 websites, block anthropic-ai, and 257,160 block Claude-Web. Most also block ClaudeBot, so the stale lines are harmless. But 54,110 websites block only the retired agents, 14.0% of everyone who wrote the old name down. Those publishers believe they have opted out of Anthropic's crawler. ClaudeBot is allowed on every one of them.
OpenAI's own chatgpt.com blocks Anthropic's two dead agents and does not name the live one, so ClaudeBot follows the wildcard group and is allowed everything the wildcard allows. So are 328 of the top hundred thousand websites.
A robots.txt is written once and rarely revisited, and crawler names change without notice. Any block list copied from an article published before mid-2024 carries agents that have since been renamed or retired. As long as AI companies rename crawlers faster than websites re-read their robots.txt, a growing share of the web will be blocking ghosts.
Frequently asked questions
How many websites block GPTBot?
6.88% of websites with a robots.txt block GPTBot: 6,250,586 of the 90,797,323 sites serving one, and 59.4% of the 10,519,366 sites that mention GPTBot at all.
Which AI crawler is blocked most?
By count, GPTBot at 6.3 million sites, with CCBot, Amazonbot, Bytespider and ClaudeBot within half a million of it. By block rate among sites that name it, Amazonbot at 62.7%, then Bytespider and CCBot, all above 60%.
Why do sites on Cloudflare block AI crawlers so much more?
Cloudflare offers a managed robots.txt that writes a Content-Signals block and an AI crawler block list into the file, switched on for an account in one setting. Sites on Cloudflare nameservers block GPTBot at 26.5%; sites on every other provider block it at 1.3%.
How many websites block Common Crawl?
6.80% of websites with a robots.txt block CCBot, 6,172,267 sites. Only 226,507 block CCBot while allowing GPTBot and ClaudeBot, so almost nobody singles Common Crawl out.
Do the biggest websites block AI crawlers?
More than average, but not by much. 13.0% of the top 1,000 websites with a robots.txt block GPTBot against 6.9% across every site with one, and the sites they single out are the answer engines rather than the training crawlers.
Does blocking a crawler in robots.txt actually stop it?
Only crawlers that choose to obey it. It also only works if the user agent string in the file matches the one the crawler sends, which is why 54,110 websites block retired Anthropic agents while ClaudeBot is allowed in.
Do any websites opt in to AI training?
Yes. 231 thousand websites carry ai-train=yes in a Content-Signals block, and 4.3 million name GPTBot in a rule that allows it.
Methodology and sources
All figures come from the StackScan September 2026 crawl, which covers the entire internet: we requested /robots.txt, /llms.txt, /ads.txt and /sitemap.xml from every domain we hold, and 151,649,745 answered. Those are the live websites; the 6.5 million domains that never answered are outside every figure. 90,797,323 returned a robots.txt we could parse, and that count is the denominator wherever a percentage is given "of sites with a robots.txt". 44.2 million returned a 404 or an HTML page, 10.2 million refused the request with a 401 or 403 and 6.3 million answered with a server error; only the first group is a site with no rules.
Rules are read per user-agent group, following RFC 9309. A crawler counts as blocked when its own group contains a bare Disallow: /; a crawler whose own group restricts some paths is not blocked, whatever the wildcard group says. Naming a crawler is recorded separately from blocking it, because Content-Signals blocks list many crawlers they then allow, and treating presence as a block overstates blocking by several million sites. Named sites and popularity tiers use the same whole-site rule.
The DNS provider is read from each site's nameservers at crawl time, one provider per domain, and is known for 34.7 million of the sites with a robots.txt. Hosting and server country come from the IP address that answered, and the Cloudflare split by address covers every site with a robots.txt. Popularity tiers come from the Tranco list, the same ranking the CMS and mail reports cut by; a tier counts only the ranked domains that serve a robots.txt. Company headquarters and size come from StackScan's company records, matched to sites by website address. That match covers a minority of sites and leans towards businesses, so the company charts describe businesses rather than the whole web.
Two limits. The 60.9 million live sites that returned no parseable robots.txt are excluded from every percentage; 44.2 million of them have no file at all, which allows everything, so the permissive share of the whole internet is higher than the figures here. And this is one three-week window, not a trend; a fixed panel is frozen so the next run can be compared with this one.
Every chart has a table view, and every figure can be cited with a link back to it. Figures are free to reuse with attribution. The crawlers above are documented by their operators: OpenAI, Anthropic, Google and Common Crawl. The Content Signals Policy is published at contentsignals.org.