Robots.txt Statistics 2026: What 90.8 Million Websites Actually Configure

How many websites have a robots.txt

59.9% of the web has a robots.txt. That is 90,797,323 of the 151,649,745 live websites on the internet. 29.1% have no file: they answered 404 or served an HTML page under the file's name. 6.7% refused the request with a 401 or 403, which a crawler treats as no file although the site may well have one it did not show us, and 4.2% answered with a server error, which RFC 9309 tells a crawler to treat as a complete disallow. A site that never answered at all is not counted as live.

Every share below is a share of those 90.8 million sites unless it says otherwise, and "the web" in this report means them. A site with no file allows every crawler, so a share of the whole internet, websites without one included, is about three fifths as large.

What websites actually do with robots.txt

24.4% of the files restrict nothing. Some hold a bare User-agent: * with Allow: /, some hold only a Sitemap: line, and 2.2 million are zero bytes, published to stop a 404.

Counting the files under fifty bytes as well, 35.8% of all robots.txt say nothing in them: 32.5 million files.

8.8% of the files tell every unlisted crawler to stay out, with User-agent: * followed by Disallow: /. 522.4 thousand of them are closed Shopify shops, and the rest are not classified. The web's default posture is open: a site has to be abandoned or private before it shuts everyone out. Many closed websites then allow specific crawlers by name, which is covered below. 96.6 thousand websites serve an HTML page with a 200 status where robots.txt should be. A crawler that fetches one gets a web page where the rules should be, and treats the site as having no rules.

About half of websites carry between one and five Disallow rules. A long, maintained robots.txt is rare: 1,204,692 sites, 1.3%, have more than fifty rules. 7.6 million websites have no User-agent: * group at all, so any crawler not named is unrestricted.

How many websites block Googlebot

150,935 websites block Googlebot from their entire site with a rule written for Googlebot by name. That is 0.17% of sites with a robots.txt, and 2.8% of the sites that name Googlebot at all.

Published counts for this figure often run to a million or more. Those counts treat any Disallow under a Googlebot group as a block. Most Googlebot groups hold path rules, a /search/ here and a plugin folder there, which narrow what Googlebot may fetch without closing the site.

A restriction is any Disallow rule written for the crawler by name. A block is that rule denying the whole site.

Agent	Sites restricting it	Sites blocking it entirely	Share that are real blocks
Googlebot	1,024,517	150,935	14.7%
Bingbot	544,197	419,699	77.1%
GPTBot	6,279,517	6,250,586	99.5%
ClaudeBot	5,984,385	5,965,523	99.7%
CCBot	6,190,672	6,172,267	99.7%
PetalBot	5,462,135	5,457,342	99.9%
Baiduspider	1,273,490	1,215,040	95.4%
YandexBot	343,430	308,722	89.9%
DuckDuckBot	49,271	34,699	70.4%

Embed this figure

<a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-flags"><img src="https://content.stackscan.com/charts/1-flags.webp" alt="Sites that restrict a crawler against sites that block it entirely" width="880" style="max-width:100%"></a> <p>Source: <a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-flags">StackScan robots.txt analysis</a></p>

The gap is only large for Googlebot and Bingbot. Of the 1 million sites with any Googlebot restriction, 465 thousand are closed to unlisted crawlers and give Googlebot a group of path rules, which makes Googlebot the one crawler allowed in. Another 408.6 thousand are open sites narrowing a few paths.

The real blocks still contain mistakes. 125,995 of the 150.9 thousand sites block Googlebot while allowing GPTBot, which refuses the crawler that sends visitors and admits one that sends none. Only 43,536 sites block both Googlebot and Bingbot.

48,774 of the 150.9 thousand sit on sites closed to every crawler, the shape of a staging Disallow: / that survived the move to production. The rest name Googlebot on an otherwise open site, which is either a decision or a block list copied with a crawler the copier did not recognise, and the file cannot say which.

Bingbot sits between the two readings: 544,197 sites restrict it and 419,699 block it outright. Bing's crawler is more often refused the whole site than Google's, and less often given a careful list of paths. The verdict is plain: a search engine is blocked in proportion to how little traffic it sends.

The most blocked search engine crawlers

Among the websites that write a rule for the crawler by name, the share that allow it and the share that block it

Chart Table

Embed this figure

<a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-search"><img src="https://content.stackscan.com/charts/1-search.webp" alt="Allowed or blocked: what websites decide once they name a search crawler" width="880" style="max-width:100%"></a> <p>Source: <a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-search">StackScan robots.txt analysis</a></p>

The chart shows what websites decide once they have named a crawler: the share that allow it against the share that block it. A crawler nobody names cannot be blocked by anybody, so that split is the meaningful question.

Huawei's PetalBot is blocked by 93.8% of the sites that name it, the highest rate of any crawler we parse, ahead of every AI crawler in part two. It is also, by a distance, the most common single target:

4.9 million websites block PetalBot and nothing else. The next most common single target, GPTBot, is blocked alone on 88,455 websites, 54.9 times fewer. PetalBot acquired a reputation for aggressive crawling in 2020 and 2021, and the two-line block became a template that hosting panels and site builders now write for customers who have never heard of the crawler.

Three more patterns sit in the search chart:

Regional engines are blocked by most of the sites that name them. Baiduspider, SeznamBot, Exabot, Sogou and YandexBot are blocked by between 44.9% and 71.6% of the sites that name them. Each is the search engine in one market and crawl load with no traffic in return everywhere else.

Googlebot is blocked by 2.8% of the sites that name it. Bingbot by 12.0%. Where a crawler brings visitors, the block rate is low.

The Internet Archive is treated like a search engine. ia_archiver is blocked by 18.0% of the sites that name it, although it sends no traffic and takes none away.

Read together, the search chart says one thing: websites admit the crawlers that pay them in visitors and block the ones that only cost bandwidth, and they will keep doing so as the cost of being crawled rises.

The crawlers named most often in robots.txt

Sites carrying a rule for the crawler, whether that rule allows or blocks it

Chart Table

AdsBot-Google and the Cloudflare managed list are matched in the file text; the rest come from parsed groups.

Embed this figure

<a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-named"><img src="https://content.stackscan.com/charts/1-named.webp" alt="The crawlers named most often in robots.txt" width="880" style="max-width:100%"></a> <p>Source: <a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-named">StackScan robots.txt analysis</a></p>

GPTBot is named by 10.5 million websites. Googlebot is named by 5.4 million. An AI training crawler that did not exist before August 2023 has a rule written for it on twice as many sites as the crawler that decides whether a site appears in Google.

Most of those GPTBot rules arrived in one Cloudflare template, which part two takes apart.

The most named crawler overall is AdsBot-Google, on 10.7 million websites, because several site builders write a rule for Google's ads crawler by name. The Cloudflare managed list explains the unfamiliar names further down: CloudflareBrowserRenderingCrawler, GoogleOther, DuckAssistBot, AI2Bot, TikTokSpider and Quora-Bot appear on between 3.1 million and 5.4 million websites each because one vendor wrote them into a template.

SEO crawlers are named more than search engines

Sites carrying a rule for the crawler, whether that rule allows or blocks it

Chart Table

These crawlers are matched in the file text, so the chart counts naming, not blocking.

Embed this figure

<a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-seo"><img src="https://content.stackscan.com/charts/1-seo.webp" alt="SEO and backlink crawlers, by how many sites name them" width="880" style="max-width:100%"></a> <p>Source: <a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-seo">StackScan robots.txt analysis</a></p>

AhrefsBot is named by 7 million sites and Moz's DotBot by 6.6 million. Both are ahead of Googlebot at 5.4 million.

Naming is a measure of attention, not hostility. A rule for AhrefsBot might allow it or block it, and the website carries a line either way. Ten of the thirteen crawlers in the chart build backlink indexes: they record who links to whom and rent the answer to the site's competitors. The other three audit sites, resell SEO data or rank visibility. Several million robots.txt carry a line about that bargain. The web thinks harder about who is selling its backlink data than about which search engine indexes it.

Block rates for these crawlers are not published here. Our parser resolves robots.txt groups for a fixed list of agents, and these are not on it, so the only source is the raw text, which finds a block reliably in a two-line group and misses it in a structured file.

Crawlers allowed by name on sites that block everyone else

8.8% of the files, 7,976,545 sites, tell every unlisted crawler to stay out. Under the standard, a crawler that is named in the file follows its own rules and ignores the wildcard. A closed site that writes User-agent: Googlebot with a path rule, or with no rule at all, has allowed Googlebot in. No Allow: line is needed, and most of these websites do not have one.

Sites whose wildcard group is Disallow: / and whose group for this crawler is not

Chart Table

Counted across the 8 million sites whose robots.txt blocks every unlisted crawler. Crawlers named on fewer than thirty thousand of them are left out.

Embed this figure

<a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-allow"><img src="https://content.stackscan.com/charts/1-allow.webp" alt="Crawlers allowed by name on sites that block everyone else" width="880" style="max-width:100%"></a> <p>Source: <a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-allow">StackScan robots.txt analysis</a></p>

Googlebot is allowed by name on 2.2 million closed sites, 27.8% of them. Yahoo's Slurp and the Internet Archive's ia_archiver follow at 1.8 million and 1.7 million.

Yahoo's Slurp has not mattered since Bing took over Yahoo search in 2010. The pattern is a template from the mid 2000s, when the standard advice was to close the site and allow Google, Yahoo and the Archive, and the template outlived two of the three.

No AI crawler is allowed by name on even 1.2% of closed sites. ChatGPT-User leads at 99 thousand, GPTBot is at 0.9% and ClaudeBot at 0.9%, against 27.8% for Googlebot. Plenty of sites block AI crawlers. Almost none close the site and then make an exception for one. Search is the one crawler the web has always chosen to admit, and no AI crawler has earned that exception yet.

Crawl-delay: one value, one platform

12.0% of the files, 10.9 million sites, declare a Crawl-delay. 74.4% of them set it to 10. That is not a spread of tuning decisions. It is a template value that site builders and hosting panels write for their customers, most often as a limit on SEO crawlers rather than on search engines.

Google has never supported the directive. Bing, Yandex and most smaller crawlers honour it.

The rest of the table is a museum. Clean-param and Host are Yandex directives, and Yandex dropped Host years ago. Request-rate and Visit-time were never in any standard. Noindex: in robots.txt was dropped by Google in 2019 and survives on 34,214 sites.

Two lines are newer than the rest. 1 million websites mention llms.txt in their robots.txt, mostly a comment pointing a language model at the newer file. 213 mention TDM-reservation, the W3C protocol for reserving text and data mining rights that European publishers were expected to adopt. TDMRep lives in its own file and in HTTP headers, so a mention in robots.txt is a side channel, and the two counts are not a comparison of adoption.

Sitemap declarations

31.4% of the files, 28.5 million sites, declare no sitemap. The Sitemap: line is the cheapest line in the file and the one most often missing.

3.9% of the files, 3.5 million sites, point at a sitemap over plain http://, a line written before the site moved to HTTPS and never re-read. 4.4 million point at a different host, sometimes a CDN or a platform's sitemap service and sometimes a leftover from a migration. 151,592 point at example.com, the placeholder host in the documentation they were copied from. Every one of those is a line written once and never re-read.

Which platforms wrote the rules

Websites whose robots.txt carries the platform's default rules, and share of all robots.txt. A website can carry more than one.

Chart Table

Embed this figure

<a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-signatures"><img src="https://content.stackscan.com/charts/1-signatures.webp" alt="Platform signatures in the web's robots.txt" width="880" style="max-width:100%"></a> <p>Source: <a href="https://www.stackscan.com/blog/robots-txt-statistics#fig-signatures">StackScan robots.txt analysis</a></p>

20.0% of all robots.txt, 18.1 million sites, carry the WordPress wp-admin rules. 16.1 million carry the exact two-line pair WordPress generates. That default is the most widely deployed crawl instruction in existence, and almost nobody who runs it chose it.

Cloudflare's managed robots.txt is on 9.5% of all robots.txt, all of it written since 2025. Yoast's signature is on 6 million, Wix's on 4.6 million, Squarespace's on 3.1 million and Shopify's on 2.9 million, including 522.4 thousand closed shops whose file says "we use Shopify as our ecommerce platform" above a Disallow: /.

Joomla, Blogger, the VestaCP and HestiaCP hosting panels, Drupal, PrestaShop and TYPO3 account for the rest of the chart. Counted as one set, allowing for websites that carry two signatures, 44.1 million robots.txt carry the default rules of a platform or plugin we can fingerprint, 48.6% of all of them, and templates we do not fingerprint would only raise that. Nearly half of what the web tells crawlers was never a decision, and any crawler policy that platforms adopt will reach the web faster than any campaign aimed at site owners.

What robots.txt exposes

A Disallow line is public. It has to be, because the crawler reads it. That makes robots.txt the one place on a website where the owner lists, in plain text, the paths they would rather nobody visited.

1.1% of the files mention password, 986.2 thousand sites. 708.8 thousand mention backup, 545.8 thousand mention .env, 186.7 thousand mention .sql and 78.9 thousand mention .git. 11,716 name a phpMyAdmin path and 31,817 name wp-config.

Some are plugin defaults refusing a folder that does not exist. Others are exactly what they look like. A Disallow does not hide a path, it advertises one, and every scanner on the internet reads robots.txt first. Nearly a million sites have handed attackers a map in the name of keeping search engines polite.

robots.txt quality by company size

Sitemap declaration falls as companies grow, from 88.2% at one-person companies to 79.3% above ten thousand staff. The mean number of Disallow rules moves the other way, from 8.7 to 12.0. Large companies write more rules and miss the one line that helps a crawler find the content.

The reason is the same one behind most of this report. A one-person site runs on WordPress, Wix or Shopify, and the platform writes a correct file with the sitemap line in it. A ten-thousand-person company has a hand-maintained file, edited by several teams over several years, that nobody owns. The larger the organisation, the more likely its robots.txt is an accident.

Frequently asked questions

How many websites have a robots.txt?

59.9% of the web has one: 90,797,323 of the internet's 151,649,745 live websites. 29.1% have no file, which allows all crawling, and the rest refused the request or answered with an error.

How many websites block Googlebot?

150,935 websites block Googlebot from their entire site with a rule written for Googlebot by name, 0.17% of sites with a robots.txt. Higher figures in circulation count any Googlebot group with a Disallow in it, including sites that only keep Googlebot out of a search page.

What is the most blocked crawler?

Huawei's PetalBot, blocked by 93.8% of the websites that name it and alone on 4.9 million single-entry block lists. Among AI crawlers, GPTBot leads by count at 6.3 million sites.

Which crawler is named most often in robots.txt?

AdsBot-Google, on 10.7 million websites. GPTBot follows on 10.5 million, then ClaudeBot and CCBot. Googlebot is named on 5.4 million.

Does Crawl-delay work?

Bing, Yandex and most smaller crawlers honour it. Google has never supported it. 12.0% of the files declare one and 74.4% of those use the value 10, a site builder template rather than a decision.

Is robots.txt a security measure?

The opposite. It lists paths in public. 545.8 thousand websites mention .env and 708.8 thousand mention backup. Anything that must stay private needs authentication.

Methodology and sources

All figures come from the StackScan September 2026 crawl, which covers the entire internet: we requested /robots.txt, /llms.txt, /ads.txt and /sitemap.xml from every domain we hold, and 151,649,745 answered. Those are the live websites; the 6.5 million domains that never answered are outside every figure. 90,797,323 returned a robots.txt we could parse, and that count is the denominator wherever a percentage is given "of sites with a robots.txt" or "of the files". 44.2 million returned a 404 or an HTML page, 10.2 million refused the request with a 401 or 403 and 6.3 million answered with a server error. Only the first group is a site with no rules; a crawler that follows the standard treats a refusal as no file and a server error as a complete disallow.

Rules are read per user-agent group, following RFC 9309: a crawler obeys the group that names it, otherwise the * group. A crawler counts as blocked when its own group contains a bare Disallow: /. A crawler whose own group holds only path rules is restricted, not blocked, whatever the wildcard group says. Both readings are shown in the Googlebot section.

Stored file bodies have their line breaks collapsed to spaces, so text matches work on whitespace-separated tokens. Where a chart is built from text rather than parsed groups (the SEO crawlers, AdsBot-Google, the Cloudflare list, platform signatures, Crawl-delay values) it says so, and those charts count naming rather than blocking.

Popularity tiers come from the Tranco list, the same ranking the CMS and mail reports cut by; a tier counts only the ranked domains that serve a robots.txt. The DNS provider is read from each site's nameservers at crawl time. Company size comes from StackScan's company records, matched to sites by website address. That match covers a minority of sites and leans towards businesses, so the company charts describe businesses rather than the whole web.

Two limits. The 60.9 million live sites that returned no parseable robots.txt are excluded from every percentage. And this is one three-week window, not a trend; a fixed panel of the ranked million plus a random slice is frozen so the next run can be compared with this one.

Every chart has a table view, and every figure can be cited with a link back to it. Figures are free to reuse with attribution. For the directives referenced above see Google's robots.txt documentation, the original robotstxt.org pages and the Yandex documentation for Clean-param.