← Insights

Data Study · Part 1 of 2

An AI agent has to get through three gates to use your site

A site can lose the agent at any of them, and nobody on your side gets told. Two of the gates you can check free in a minute. The third is what Sitedex measures.

When a buyer asks an AI assistant about you, an agent goes to your site to look. It has to get past your bot protection, land on a page with words on it, and find the answer to what the buyer asked. Fail any one of the three and the assistant goes back to the buyer with nothing from your site. Gate 1 shows up as a challenged request in a firewall log. Gates 2 and 3 leave no trace.

An AI agent's path through three gates A buyer asks an AI about you. An agent visits your site. The answer comes from your site. The answer comes from somewhere else. 1 · THE DOOR Bot protection lets it in Refused, or must prove it's human 2 · THE PAGE Words it can read Words only after JavaScript 3 · THE ANSWER Answers the question Silent, vague, or contradictory An AI agent's path through three gates A buyer asks an AI about you. An agent visits your site. The answer comes from your site. The answer comes from somewhere else. 1 · THE DOOR Bot protection lets it in Refused, or must prove it's human 2 · THE PAGE Words it can read Words only after JavaScript 3 · THE ANSWER Answers the question Silent, vague, or contradictory

We measured all three this summer. Gates 1 and 2 come from one visit to 2,684 top sites in July. Gate 3 comes from a separate study of 271 sites in June.

Where AI agents get stuckthree measurements
Gate 1 · The door Does your bot protection let it in? 1 in 11 top sites refused or challenged a crawler that identified itself, after welcoming it in robots.txt Stores: 1 in 5. Free check Run fetchcheck
Gate 2 · The page Does it see a real page? 1 in 12 let the crawler in, then served a page with no readable text on it AI crawlers don't run JavaScript, so text that only appears after it runs doesn't reach them. Free check Shown in the same fetchcheck result
Gate 3 · The answer Does the page answer the question? 83% of the questions buyers ask had no clear answer on the site Readable pages with no answer on pricing, security, limits, or integrations. Sitedex audit Find the questions your site can't answer
Gates 1 and 2 are from a signed crawler's visit to 2,684 top sites whose robots.txt welcomes it, July 2026. Gate 3 is from 7,160 buyer questions across 271 live sites, June 2026.

Gate 1: the door

Walmart is an announced partner of Meta's agent, Muse, and told TechCrunch it wants to be where its customers are. Users still reported Muse failing to check out there. The cause was Walmart's own prove-you're-human button: when the agent's flow interrupted it, the check failed and the agent was turned away. A Walmart spokesperson told TechCrunch the blocks weren't intentional. Amazon did the opposite, on purpose. In September it barred Muse from browsing and buying, with a notice that "continued access by an unauthorized AI agent violates Amazon's Conditions of Use." Its reasons included that Muse didn't identify itself as an AI agent while browsing (Engadget, Tech Times). TechCrunch's list of complaints runs on through Yelp, eBay, Zillow, Pizza Hut, Adidas and several airlines.

By accident walmart.com Turned away by its own human check An announced Muse partner. The agent's flow interrupted the prove-you're-human button, the check failed, and the agent was turned away. Walmart says the blocks weren't intentional.
On purpose amazon.com Barred by policy "Continued access by an unauthorized AI agent violates Amazon's Conditions of Use." Amazon's reasons included no notice, and that Muse didn't identify itself as an AI agent while browsing.
Reported by TechCrunch, October 6, 2026, and Engadget, September 21, 2026.

In July we sent a crawler that gives its name and signs every request to 2,684 top sites whose robots.txt permits that exact visit. It is not on Cloudflare's verified-bot list, which is how a new agent arrives at your site. 1 in 11 refused it or sent it a prove-you're-human page instead. Among stores, 1 in 5. Walmart's accident is the common case. The full study is Your firewall doesn't read your robots.txt.

This is what the result looks like when you run it yourself.

Web Bot Auth www.lego.com Challenged by bot protection Your site returned a challenge instead of the page (cf-mitigated: challenge, a 403 response). A challenge is meant to stop automated traffic by asking it to prove it is a real browser.
fetchcheck on www.lego.com, checked October 4, 2026. The same site was challenged in our July study.

A prove-you're-human page instead of the real page. Challenges fire on reputation: a crawler your protection hasn't verified gets that page whether it signs or not. Since September 15, a new Cloudflare domain blocks bots classified as Agent, meaning automated activity acting for a user in real time, on pages that display ads, by default (Cloudflare's changelog). TechCrunch reports that existing sites which had chosen to block AI bots were shifted to block AI agents on any page with ads. The exact settings, including that one, are in part 2.

Gate 2: the page

This is the gate your own team cannot see. The crawler gets in, your server records a normal success, and the page it was handed has no words on it. 1 in 12 of the 2,684 sites did this. The words on those pages arrive only after JavaScript runs, and AI crawlers don't run JavaScript. Open the same address in a browser and it looks normal.

The bigger the site, the more likely this is how it fails. Among the top 500, 11.6% served an empty page and 9.1% turned the crawler away. In the smallest quarter of the sample, 4.8% served an empty page and 12.3% turned it away. Big sites fail by how they're built, a frontend made for browsers. Smaller sites fail by a security product someone switched on. The agent reads nothing either way.

Nothing on your side records it. Your analytics run as JavaScript, which the crawler never executed, so the visit left no trace there. The server logged a success. The only place the empty page is visible is the crawler's side of the connection. fetchcheck shows it as "Reachable, but almost no content" for any page you paste, and the Sitedex report checks your homepage for it as part of the score. No firewall setting fixes this one. The fix is rendering the page on the server, so the words are in the HTML before any script runs.

Gate 3: the answer

Getting in is the start. The buyer asked something specific, what it costs at their size, or what happens when something breaks, and the agent is on your page looking for that line. In June we checked 7,160 of these questions against 271 live sites. The site gave a clear on-page answer 17% of the time. The rest, 83%, were dead ends. The site goes silent on the question, or it hedges and the words on the page don't add up to an answer, or two pages contradict each other and whichever one the agent reads first wins. The assistant then answers the buyer from somewhere other than your site, and nobody tells you the question was asked.

Sitedex audit sitedex.dev On the page, no answer "For white-labeling purposes, can I host the MCP endpoint on my own subdomain instead of api.sitedex.dev, or is the hosted endpoint strictly tied to your infrastructure?" Our pricing FAQ said white-label was on the table and to email sales. It didn't say yes or no to a custom subdomain. The fix is to answer it in the hosting docs.
Sitedex audit of sitedex.dev, June 2026.

Knowing which questions are missing is the hard part, and it is what a Sitedex audit does: the questions buyers ask about you, the page that should have answered each one, and the fix. The June numbers are in The site can't answer 83% of the time.

Gates 1 and 2 are free to check, and fetchcheck reports both in one result. Gate 1 is usually a setting. Gate 2 means rendering your pages on the server. Gate 3 is the work, question by question, and it is the one that decides whether the assistant answers with your site or without it.

Run the full report for your site →

Or start with the free part: check gates 1 and 2 with fetchcheck.