Insights
What AI gets right — and wrong — about real sites.
Field reports from the index: what buyers ask AI, where sites dead-end, and what the model says when yours stays silent.
Your firewall doesn't read your robots.txt
Part 2 of our robots.txt study. Last time we read what the top sites' files declare. This time we tested the declaration: a polite, signed crawler visited 2,684 top sites whose own robots.txt welcomes it. Four in five served a readable page. One in six gave it nothing — and for every site that challenged the crawler, a single response header proves which layer overruled the file.
Read the reportYour robots.txt is a 2023 war memorial
We read the robots.txt of the top 10,000 sites. 38% of the GPTBot rules we could date were written in a single quarter of 2023 — right after GPTBot launched and the Times sued. Then every AI company quietly added a second bot, the one that answers questions in real time, and almost nobody wrote a rule for it.
Read the reportIs llms.txt worth it? Mostly no — and that's not the interesting part.
Almost nothing reads it yet, so it won't move your AI visibility. Add it anyway — it's a five-minute hedge — then go do the part that actually decides whether an AI can answer for you.
Read the reportMost AI-readiness tools pad the list. Ours only shows what's really broken.
Every problem we put in front of you is one we're sure is real — because we keep our own false alarms off your list before you ever see them.
Read the reportYour buyers are asking AI about you. The site can't answer 83% of the time.
We ran 7,160 of the questions buyers actually ask against 271 live websites. Here's how often the site had the answer — and what happens when it doesn't.
Read the report