Your firewall doesn't read your robots.txt

A site's robots.txt says what crawlers may do. The security layer at the edge decides what a crawler actually gets. Nothing checks that the two agree — so we checked, on 2,684 of the web's top sites, every one with a robots.txt that permits the crawler we sent.

The crawler is the kind robots.txt was invented to govern. It gives its name on every request, carries a signature that proves the name, and follows every rule in the file it's handed. In Part 1 of this study we read what the top 10,000 files declare. A file is only a declaration. This is what happens when a crawler takes the declaration at face value.

The baseline: 79% serve a readable page

79% of these declared-open sites did what their file says. The crawler asked; the origin served a page it could read. The declared-open web mostly works as declared, and any telling that opens with "the web is closing" has skipped its own baseline.

One site in six gives the crawler nothing

One declared-open site in six gave the identified crawler nothing it could read. Not one site in six on the web — one in six of the sites whose own published robots.txt permits exactly this visit.

Some of the 468 blocked or challenged the crawler outright: a refusal, or a page asking it to prove it's human. The rest let it in, returned a success code, and served a page with no readable text on it — in most cases nobody's decision at all.

Where the refusals come from

One declared-open site in eleven refused the crawler. Just over half of those refusals came as a challenge — the "verify you are human" page — shown to a visitor that had said, truthfully, that it isn't one. The rest were flat refusals, mostly 403s from security stacks that don't identify themselves, so we won't guess whose they are.

The challenges do identify themselves. All 147 challenged responses carry the same response header: cf-mitigated: challenge. No response in any other bucket carries it. That header is set by Cloudflare's bot management layer. There is no fingerprinting or inference in that attribution.

How specific are the refusals to the crawler? A block only counts as one in this study when a browser-shaped request to the same site was not refused, so the blocked bucket is crawler-specific by definition. The challenges we went back and tested: two days after the study we re-probed a sample of the challenged sites, sending each the identified crawler and a browser-shaped request at the same moment, from the same network. On more than half, the browser-shaped request sailed through while the identified crawler was challenged. The rest challenged everything we sent.

That re-probe covered a sample of every bucket, not just the challenges, and it doubles as a check on the snapshot itself: the verdicts reproduced almost exactly — every served site served again, every challenged site challenged again, 38 of 40 blocked sites still blocked.

Which kinds of sites turn the crawler away

You might expect the most defended sites — banks, government, anything with a legal department — to refuse most often. The data runs the other way.

One store in five turns the crawler away: the highest rate of any category, and fourteen times the rate of developer-tool sites, the most open on the list. Government and education sites are the second most open. The sites whose business depends on being found are the likeliest to contradict their own robots.txt.

Why? We measured responses, not reasons. A response header carries no motive, and we won't invent one per company. The mundane explanation — the file and the firewall are maintained by different people, against different problems, with nothing checking that the two agree — fits what we see, but it's a story. The measurement is only the contradiction, and where it concentrates.

Reachable, with nothing to read

One declared-open site in twelve let the crawler straight in — no challenge, no block, a real 200 — and served a page with no readable text on it. The HTML is real; every visible word arrives only after JavaScript runs. We verified each of these pages is a genuine shell rather than an error. In a browser they look completely normal. To any reader that doesn't execute scripts, the site says nothing.

Size decides which failure a site gets. In the top 500, the empty shell is the more common way to give a crawler nothing: 11.6% of sites, versus 9.1% turned away. In the bottom quarter of our sample the ratio flips: 4.8% shells, 12.3% turned away. Big sites fail by architecture — a modern frontend built for browsers, where no one ever decided anything about crawlers. Smaller sites fail by enforcement — a security product someone switched on. The crawler reads nothing either way.

No firewall setting fixes the shell. And there's a harder question this study can't see: where text exists, does it answer what buyers ask? That's the half Part 1 ended on, and we'll come back to it.

Why you'd never notice

The gap stays invisible because nothing on your side records it. robots.txt doesn't log anything — it can't tell you whom the edge turned away. Your analytics run as JavaScript, which a crawler never executes, so the refused visit leaves no trace there either. From the origin's side, a challenged request can even look served, because the edge answered before your server heard about it. If you're a Cloudflare customer the challenge events are in your dashboard — if you think to look for a disagreement you have no reason to suspect. By default, the contradiction is visible from exactly one place: the crawler's side of the connection.

fetchcheck is this study's verdict engine pointed at whatever site you give it — the same signed crawler, the same verdicts you've just read, run live on your site. "Challenged by bot protection" or "blocked" means your edge is overruling your robots.txt, and you now know which layer to go argue with. "Reachable, but almost no content" means your pages only render in a browser. A readable page puts you in the 79%, which leaves the harder question: do those pages answer what buyers ask? Finding the questions your pages can't answer is the next step.

Part 1 argued that whatever posture you choose toward crawlers — seal, sell, or open — you should choose it on purpose, in the file. This study adds the corollary: then check that the layer with the actual power agrees with you. On one declared-open site in six, today, it doesn't.

Run fetchcheck on your site — see what an identified crawler actually gets →

Method

What this is and isn't. On 2026-07-21 we sent a signed, honestly identified crawler to the sites of the Tranco top 4,012 domains and recorded what each one served. This is a study of observed HTTP responses to one identified crawler. We did not measure any AI model's output, any site's traffic or revenue, or anyone's intent, and no figure here says a site "blocks AI" — the figures say how each site's stack treated one polite, verifiable crawler that its robots.txt permits. Every figure names its denominator.

• Sample. Of the 4,012 raw domains, 3,107 resolved to real, distinct websites — roughly one raw entry in five didn't turn out to be a website at all. We probed 3,103, then excluded 317 that redirected to another property of the same entity and 102 where the failure was on our side, leaving 2,684 cleanly measured sites: the denominator for every rate above.
• Declared-open by construction. Every measured site has a robots.txt that permits our crawler. That is the study's design, not a discovery — so every rate reads as a conditional: among sites that declare this crawler welcome. The fact that 100% of the refused sites have welcoming files is a consequence of that design, and we make no gotcha of it.
• The crawler. Identifies itself as SitedexBot on every request, signs its requests under Web Bot Auth so the identity is cryptographically verifiable, and obeys robots.txt. The verdict engine is the same one behind the public fetchcheck tool, so any result in this study can be reproduced on demand.
• Verdicts. Readable content: 2,132 (79.4%). Block or challenge: 253 (9.4%) — 147 challenged, 106 blocked. Empty shell: 215 (8.0%) — a success response, text/html, and no readable text; all 215 were verified as genuine JS-rendered shells, not errors. Blocked for everyone: 58 (2.2%) — these sites refuse browser-shaped requests too, so they are excluded from every crawler-specific claim. The remaining 26 (1.0%): 19 ambiguous edge failures we can't confidently call blocks, 5 success responses that weren't HTML pages, and 2 sites whose robots.txt had changed to disallow by probe time — none appears in any figure above. The headline "1 in 6" is the sum of the refusals and the shells: 253 + 215 = 468 of 2,684 (17.4%).
• Pairing. In the main run, the paired browser-shaped fetch fired only when the crawler-identified fetch failed outright — a block or an error. That pairing is how blocks are separated from sites that refuse everyone; challenged fetches did not trigger it. The challenge pairing was added in a re-probe two days later: 60 of the 147 challenged sites, each sent the identified crawler and a browser-shaped request at the same moment, from the same network. 34 of the 60 served the browser-shaped request clean while challenging the crawler; 25 challenged both; one browser-shaped request errored.
• The Cloudflare attribution. Our probe recorded one vendor signal — the cf-mitigated: challenge response header. All 147 challenged responses carry it; zero responses in any other bucket do. So the attribution to Cloudflare's bot management is solid for the challenged bucket, but it is not a vendor comparison: we did not check for other vendors' signatures. A challenge from Akamai, Fastly, or another stack — which identify themselves with different headers we didn't read — would land in the block bucket below, not here. We can say every challenge we detected was Cloudflare's; we cannot say Cloudflare challenges more than anyone else.
• The hard-block hedge. The 106 outright blocks are heterogeneous — mostly 403s, with scattered other status codes. We did not record their response headers beyond the one Cloudflare signal, so we attribute them to no vendor — not because they carry none, but because we didn't look. A block counts as crawler-specific because the paired browser-shaped request was not refused — but that request mostly returned redirects rather than confirmed content, so what this bucket demonstrates is "not refused," not "served a readable page." The direct browser-gets-through evidence comes from the challenged-bucket re-probe described under Pairing.
• Categories. Each measured site was classified into one of eight named categories or "other" (sites that fit none of them). The table's rates count blocks and challenges only; empty shells are excluded from it.