fetchcheck

See how any site handles a signed crawler.

SitedexBot is our crawler. It signs every request so a site can verify it's really us. Paste a URL and we send one signed request to the page, then show you exactly what came back: allowed, blocked, challenged, or a page with nothing in it.

Enter any domain. We check robots.txt, then fetch the page as SitedexBot. We don't keep your page. Don't paste URLs with secrets in them — the address is what we check.

Why check whether a signed crawler gets in

Bot protection is on by default now. WAFs and bot-management rules block or challenge automated traffic before it reaches your server. That is good for stopping scrapers and abuse. It also catches the crawlers you may actually want to let in.

When someone asks an AI assistant about your company, product, or docs, it often fetches your pages to answer. If your site turns that request away, or hands it a blank page with no content, the assistant answers from whatever else it can find instead of what you published.

A signed crawler proves who it is on every request, using an open standard that large networks already recognize. That lets you allow the crawlers you trust by name instead of blocking everything. fetchcheck sends one signed request from SitedexBot and shows you the result, so you can see whether a trusted crawler gets in, and whether your content is actually there when it does.

What fetchcheck does

  • Sends one signed GET from SitedexBot, our crawler. Every request is signed so a site can verify it is really us.
  • Respects robots.txt. If your rules tell SitedexBot not to fetch a page, we don't. We report the rule instead.
  • Never stores or republishes your page. We show the response status, the headers, and whether the HTML had readable content. We don't keep the page.
  • Checks one URL per run, using up to four requests: your robots.txt, the page (signed, as SitedexBot), and — only if that signed request is turned away — one unsigned retry for comparison. That retry comes from our own infrastructure, not a real browser or a home connection; it simply drops the signature, so we can see whether the refusal is aimed at signed or identified crawlers specifically. And if https doesn't answer at all, one plain http check, to tell a site that is down from one that only answers over http.

SitedexBot signs its requests using Web Bot Auth, an open standard for identifying trusted crawlers. Our signing keys are public. More: sitedex.dev/bot.

Checking your own signing-key setup instead? Try keycheck, the Web Bot Auth key check.