Robots.txt Statistics 2026: What 90.8 Million Websites Actually Configure

How many websites have a robots.txt

57.4% of the web has a robots.txt. That is 90,796,807 of the 158,187,361 live websites on the internet. The remaining 42.6% answered 404, returned an HTML page under the file's name, or could not be fetched.

Every share below is a share of those 90.8 million sites unless it says otherwise, and "the web" in this report means them. A site with no file allows every crawler, so a share of the whole internet, websites without one included, is lower by a little under half.

What websites actually do with robots.txt

24.4% of websites restrict nothing in their robots.txt. Some hold a bare User-agent: * with Allow: /, some hold only a Sitemap: line, and 2.2 million are zero bytes, published to stop a 404.

A further 17 million websites have a robots.txt of between one and fifty bytes. Together, a fifth of the web says nothing in it.

8.8% of the web tells every unlisted crawler to stay out, with User-agent: * followed by Disallow: /. Most of these are parked domains, staging hosts and closed shops. The web's default posture is open: a site has to be abandoned or private before it shuts everyone out. Many closed websites then allow specific crawlers by name, which is covered below. 96.6 thousand websites serve an HTML page with a 200 status where robots.txt should be. A crawler that fetches one gets a web page where the rules should be, and treats the site as having no rules.

About half of websites carry between one and five Disallow rules. A long, maintained robots.txt is rare: 1,204,683 sites, 1.3%, have more than fifty rules. 7.6 million websites have no User-agent: * group at all, so any crawler not named is unrestricted.

How many websites block Googlebot

150,934 websites block Googlebot from their entire site with a rule written for Googlebot by name. That is 0.17% of the web, and 2.8% of the sites that name Googlebot at all.

Published counts for this figure often run to a million or more. Those counts treat any Disallow under a Googlebot group as a block. Most Googlebot groups hold path rules, a /search/ here and a plugin folder there, which narrow what Googlebot may fetch without closing the site.

A restriction is any Disallow rule written for the crawler by name. A block is that rule denying the whole site.

Agent	Sites restricting it	Sites blocking it entirely	Share that are real blocks
Googlebot	1,024,512	150,934	14.7%
Bingbot	544,193	419,697	77.1%
GPTBot	6,279,457	6,250,526	99.5%
ClaudeBot	5,984,332	5,965,470	99.7%
CCBot	6,190,615	6,172,210	99.7%
PetalBot	5,462,131	5,457,338	99.9%
Baiduspider	1,273,482	1,215,035	95.4%
YandexBot	343,419	308,713	89.9%
DuckDuckBot	49,271	34,699	70.4%

Embed this figure

<a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-flags"><img src="https://content.stackscan.com/charts/1-flags.webp" alt="Sites that restrict a crawler against sites that block it entirely" width="880" style="max-width:100%"></a> <p>Source: <a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-flags">StackScan robots.txt analysis</a></p>

The gap is only large for Googlebot and Bingbot. Of the 1 million sites with any Googlebot restriction, 465 thousand are closed to unlisted crawlers and give Googlebot a group of path rules, which makes Googlebot the one crawler allowed in. Another 408.6 thousand are open sites narrowing a few paths.

The real blocks still contain mistakes. 125,994 of the 150.9 thousand sites block Googlebot while allowing GPTBot, which refuses the crawler that sends visitors and admits one that sends none. Only 43,536 sites block both Googlebot and Bingbot, roughly the number you would expect to be closed to search on purpose.

The usual causes are a staging Disallow: / that survived the move to production, a plugin rule for a bot name that also matches Googlebot as a substring, and a block list copied from an article that included a crawler the copier did not recognise.

Bingbot sits between the two readings: 544,193 sites restrict it and 419,697 block it outright. Bing's crawler is more often refused the whole site than Google's, and less often given a careful list of paths. The verdict is plain: a search engine is blocked in proportion to how little traffic it sends.

The most blocked search engine crawlers

Among the websites that write a rule for the crawler by name, the share that allow it and the share that block it

Chart Table view

Embed this figure

<a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-search"><img src="https://content.stackscan.com/charts/1-search.webp" alt="Allowed or blocked: what websites decide once they name a search crawler" width="880" style="max-width:100%"></a> <p>Source: <a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-search">StackScan robots.txt analysis</a></p>

The chart shows what websites decide once they have named a crawler: the share that allow it against the share that block it. A crawler nobody names cannot be blocked by anybody, so that split is the meaningful question.

Huawei's PetalBot is blocked by 93.8% of the sites that name it, the highest rate of any crawler on the internet, ahead of every AI crawler in part two. It is also, by a distance, the most common single target:

4.9 million websites block PetalBot and nothing else. The next most common single target, GPTBot, is blocked alone on 88,455 websites, fifty times fewer. PetalBot acquired a reputation for aggressive crawling in 2020 and 2021, and the two-line block became a template that hosting panels and site builders now write for customers who have never heard of the crawler.

Three more patterns sit in the search chart:

Regional engines are blocked outside their region. Baiduspider, SeznamBot, Exabot, Sogou and YandexBot are blocked by between 44.9% and 71.6% of the sites that name them. Inside their home markets these are the search engine. Everywhere else they are crawl load with no traffic in return.

Googlebot is blocked by 2.8% of the sites that name it. Bingbot by 12.0%. Where a crawler brings visitors, the block rate is low.

The Internet Archive is treated like a search engine. ia_archiver is blocked by 18.0% of the sites that name it, although it sends no traffic and takes none away.

Read together, the search chart says one thing: websites admit the crawlers that pay them in visitors and block the ones that only cost bandwidth, and they will keep doing so as the cost of being crawled rises.

The crawlers named most often in robots.txt

Sites carrying a rule for the crawler, whether that rule allows or blocks it

Chart Table view

AdsBot-Google and the Cloudflare managed list are matched in the file text; the rest come from parsed groups.

Embed this figure

<a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-named"><img src="https://content.stackscan.com/charts/1-named.webp" alt="The crawlers named most often in robots.txt" width="880" style="max-width:100%"></a> <p>Source: <a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-named">StackScan robots.txt analysis</a></p>

GPTBot is named by 10.5 million websites. Googlebot is named by 5.4 million. An AI training crawler that did not exist before August 2023 has a rule written for it on twice as many sites as the crawler that decides whether a site appears in Google.

Most of those GPTBot rules arrived in one Cloudflare template, which part two takes apart.

The most named crawler overall is AdsBot-Google, on 10.7 million websites, because several site builders write a rule for Google's ads crawler by name. The Cloudflare managed list explains the unfamiliar names further down: CloudflareBrowserRenderingCrawler, GoogleOther, DuckAssistBot, AI2Bot, TikTokSpider and Quora-Bot each appear on around three million websites because one vendor wrote them into a template.

SEO crawlers are named more than search engines

Sites carrying a rule for the crawler, whether that rule allows or blocks it

Chart Table view

These crawlers are matched in the file text, so the chart counts naming, not blocking.

Embed this figure

<a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-seo"><img src="https://content.stackscan.com/charts/1-seo.webp" alt="SEO and backlink crawlers, by how many sites name them" width="880" style="max-width:100%"></a> <p>Source: <a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-seo">StackScan robots.txt analysis</a></p>

AhrefsBot is named by 7 million sites and Moz's DotBot by 6.6 million. Both are ahead of Googlebot at 5.4 million.

Naming is a measure of attention, not hostility. A rule for AhrefsBot might allow it or block it, and the website carries a line either way. Twelve of the thirteen crawlers in the chart exist to sell competitive data about the site they crawl: they index who links to whom and rent the answer to the site's competitors. Several million site owners have written a line about that bargain. The web thinks harder about who is selling its backlink data than about which search engine indexes it.

Block rates for these crawlers are not published here. Our parser resolves robots.txt groups for a fixed list of agents, and these are not on it, so the only source is the raw text, which finds a block reliably in a two-line group and misses it in a structured file.

Crawlers allowed by name on sites that block everyone else

8.8% of the web, 7,976,520 sites, tells every unlisted crawler to stay out. Under the standard, a crawler that is named in the file follows its own rules and ignores the wildcard. A closed site that writes User-agent: Googlebot with a path rule, or with no rule at all, has allowed Googlebot in. No Allow: line is needed, and most of these websites do not have one.

Sites whose wildcard group is Disallow: / and whose group for this crawler is not

Chart Table view

Counted across the 8 million sites whose robots.txt blocks every unlisted crawler. Crawlers named on fewer than thirty thousand of them are left out.

Embed this figure

<a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-allow"><img src="https://content.stackscan.com/charts/1-allow.webp" alt="Crawlers allowed by name on sites that block everyone else" width="880" style="max-width:100%"></a> <p>Source: <a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-allow">StackScan robots.txt analysis</a></p>

Googlebot is allowed by name on 2.2 million closed sites, 27.8% of them. Yahoo's Slurp and the Internet Archive's ia_archiver follow at 1.8 million and 1.7 million.

Two of those three have not mattered for fifteen years. The pattern is a template from the mid 2000s, when the standard advice was to close the site and allow Google, Yahoo and the Archive, and the template outlived two of the three.

No AI crawler is allowed by name on even 1.2% of closed sites. ChatGPT-User leads at 99 thousand, GPTBot is at 0.9% and ClaudeBot at 0.9%, against 27.8% for Googlebot. Plenty of sites block AI crawlers. Almost none close the site and then make an exception for one. Search is the one crawler the web has always chosen to admit, and no AI crawler has earned that exception yet.

Crawl-delay: one value, one platform

12.0% of the web, 10.9 million sites, declares a Crawl-delay. 74.4% of them set it to 10. That is not a spread of tuning decisions. It is a template value that site builders and hosting panels write for their customers, most often as a limit on SEO crawlers rather than on search engines.

Google has never supported the directive. Bing, Yandex and most smaller crawlers honour it.

The rest of the table is a museum. Clean-param and Host are Yandex directives, and Yandex dropped Host years ago. Request-rate and Visit-time were never in any standard. Noindex: in robots.txt was dropped by Google in 2019 and survives on 34,214 sites.

Two lines are newer than the rest. 1 million websites mention llms.txt in their robots.txt, mostly a comment pointing a language model at the newer file. 213 mention TDM-reservation, the W3C protocol for reserving text and data mining rights that European publishers were expected to adopt. The internet took up the format an AI startup proposed over the one the standards community wrote, by a factor of several thousand.

Sitemap declarations

31.4% of the web, 28.5 million sites, declares no sitemap. The Sitemap: line is the cheapest line in the file and the one most often missing.

3.9% of the web, 3.5 million sites, points at a sitemap over plain http://, a line written before the site moved to HTTPS and never re-read. 4.4 million point at a different host, sometimes a CDN or a platform's sitemap service and sometimes a leftover from a migration. 151,591 point at example.com, the placeholder host in the documentation they were copied from. The cheapest line in the file is the one most often missing, which says how rarely anyone reads the file after it is written.

Which platforms wrote the rules

20.0% of the web, 18.1 million sites, carries the WordPress wp-admin rules. 16.1 million carry the exact two-line pair WordPress generates. That default is the most widely deployed crawl instruction in existence, and almost nobody who runs it chose it.

Cloudflare's managed robots.txt is on 9.5% of the web, all of it written since 2025. Yoast's signature is on 6 million, Wix's on 4.6 million, Squarespace's on 3.1 million and Shopify's on 2.9 million, including 522.4 thousand closed shops whose file says "we use Shopify as our ecommerce platform" above a Disallow: /.

Joomla, Blogger, the VestaCP and HestiaCP hosting panels, Drupal, PrestaShop and TYPO3 account for the rest of the chart. Add these together, allowing for websites that carry two signatures, and roughly half of the web's robots.txt rules were generated by a platform or a plugin. The robots.txt most people picture, written by a person who runs the site, is the minority. Most of what the web tells crawlers was never a decision, and any crawler policy that platforms adopt will reach the web faster than any campaign aimed at site owners.

What robots.txt exposes

A Disallow line is public. It has to be, because the crawler reads it. That makes robots.txt the one place on a website where the owner lists, in plain text, the paths they would rather nobody visited.

1.1% of the web mentions password in its robots.txt, 986.2 thousand sites. 708.8 thousand mention backup, 545.8 thousand mention .env, 186.7 thousand mention .sql and 78.9 thousand mention .git. 11,715 name a phpMyAdmin path and 31,817 name wp-config.

Some are plugin defaults refusing a folder that does not exist. Many are exactly what they look like. A Disallow does not hide a path, it advertises one, and every scanner on the internet reads robots.txt first. Nearly a million sites have handed attackers a map in the name of keeping search engines polite.

robots.txt quality by company size

Sitemap declaration falls as companies grow, from 88.2% at one-person companies to 79.3% above ten thousand staff. The mean number of Disallow rules moves the other way, from 8.7 to 11.7. Large companies write more rules and miss the one line that helps a crawler find the content.

The reason is the same one behind most of this report. A one-person site runs on WordPress, Wix or Shopify, and the platform writes a correct file with the sitemap line in it. A ten-thousand-person company has a hand-maintained file, edited by several teams over several years, that nobody owns. The larger the organisation, the more likely its robots.txt is an accident.

Frequently asked questions

How many websites have a robots.txt?

57.4% of the web has one: 90,796,807 of the internet's 158,187,361 live websites. The other 42.6% have none, which allows all crawling.

How many websites block Googlebot?

150,934 websites block Googlebot from their entire site with a rule written for Googlebot by name, 0.17% of the web. Higher figures in circulation count any Googlebot group with a Disallow in it, including sites that only keep Googlebot out of a search page.

What is the most blocked crawler?

Huawei's PetalBot, blocked by 93.8% of the websites that name it and alone on 4.9 million single-entry block lists. Among AI crawlers, GPTBot leads by count at 6.3 million sites.

Which crawler is named most often in robots.txt?

AdsBot-Google, on 10.7 million websites. GPTBot follows on 10.5 million, then ClaudeBot and CCBot. Googlebot is named on 5.4 million.

Does Crawl-delay work?

Bing, Yandex and most smaller crawlers honour it. Google has never supported it. 12.0% of the web declares one and 74.4% of those use the value 10, a site builder template rather than a decision.

Is robots.txt a security measure?

The opposite. It lists paths in public. 545.8 thousand websites mention .env and 708.8 thousand mention backup. Anything that must stay private needs authentication.

Methodology and sources

All figures come from the StackScan September 2026 crawl, which covers the entire internet: we requested /robots.txt, /llms.txt, /ads.txt and /sitemap.xml from 158,187,361 live websites in the September 2026 crawl. 90,796,807 returned a robots.txt we could parse, and that count is the denominator wherever a percentage is given "of sites with a robots.txt".

Rules are read per user-agent group, following RFC 9309: a crawler obeys the group that names it, otherwise the * group. A crawler counts as blocked when its own group contains a bare Disallow: /. A crawler whose own group holds only path rules is restricted, not blocked, whatever the wildcard group says. Both readings are shown in the Googlebot section.

Stored file bodies have their line breaks collapsed to spaces, so text matches work on whitespace-separated tokens. Where a chart is built from text rather than parsed groups (the SEO crawlers, AdsBot-Google, the Cloudflare list, platform signatures, Crawl-delay values) it says so, and those charts count naming rather than blocking.

Popularity tiers come from the StackScan domain ranking. The DNS provider is read from each site's nameservers at crawl time. Company size comes from StackScan's company records, matched to sites by website address. That match covers a minority of sites and leans towards businesses, so the company charts describe businesses rather than the whole web.

Two limits. The 67.4 million live sites with no robots.txt are excluded from every percentage. And this is one three-week window, not a trend; a fixed panel of the ranked million plus a random slice is frozen so the next run can be compared with this one.

Every chart has a table view, and every figure can be cited with a link back to it. Figures are free to reuse with attribution. For the directives referenced above see Google's robots.txt documentation, the original robotstxt.org pages and the Yandex documentation for Clean-param.