Research from a crawl of the entire internet, 399M+ websites. Each report counts what websites run, publish and configure, says plainly what its figures are a share of, and can be cited with a link.
How many websites run a CMS, which platforms build them, what the rest of the web is built with, and why the platform that builds the most websites is not the platform that builds the most important ones.
Read the reportHow many domains publish SPF and DMARC, what those records actually say, where the reports go, and which mail providers authenticate their customers. Counted across every domain on the internet with an MX record.
Read the reportHow many websites have a robots.txt, what they allow and block, which crawlers are blocked and named most often, and which platforms wrote the rules. Counted across the entire internet, not sampled.
Read the reportHow many websites block GPTBot, ClaudeBot, CCBot and the other AI crawlers, which of the biggest sites do it, and why most of the blocks trace back to one vendor's default. Counted across the entire internet, with named sites.
Read the reportHow many websites have an llms.txt, which platforms generated most of them, what they contain, where their links point, and why the most quoted adoption figure counts placeholders and templates rather than decisions.
Read the reportFigures may be reproduced with attribution to StackScan. Atom feed
