Crawler identity

SitedexBot

SitedexBot is the crawler Sitedex uses to fetch the pages of a domain that has been submitted for analysis. If you found this page from your server logs, this explains who we are, why we fetched your site, and how to control or contact us.

User-Agent

Every request SitedexBot makes to your origin carries this exact User-Agent string:

SitedexBot/1.0 (+https://sitedex.dev/bot)

Match on the SitedexBot token in your robots.txt — that token is stable and will not change.

Why SitedexBot fetched your site

Sitedex analyzes how readily AI agents can understand a website. When a domain is submitted, SitedexBot fetches a small number of publicly accessible pages — the homepage, robots.txt, and a few linked pages — to produce that analysis. We do not attempt to access private, authenticated, or disallowed areas of your site.

Robots compliance

SitedexBot reads and honors robots.txt before crawling. It matches the SitedexBot user-agent group first, then falls back to the * wildcard group if you have not named us specifically, following the RFC 9309 semantics (most-specific group wins; longest matching rule wins; Allow beats an equal-length Disallow).

To disallow SitedexBot from all or part of your site, add a group to your robots.txt:

User-agent: SitedexBot
Disallow: /

Verifiable identity

SitedexBot publishes a Web Bot Auth key directory at /.well-known/http-message-signatures-directory, a signed list of the Ed25519 public keys we use to prove requests are genuinely ours (per the IETF HTTP Message Signatures directory draft). The directory response is itself signed, so a verifier can confirm the keys belong to sitedex.dev. Signed request verification is rolling out progressively.

Contact

Questions, or want us to stop crawling a domain you control? Email hello@sitedex.dev.