← Insights

Data Study

Most AI crawlers don't run JavaScript. Does your site depend on it?

Most AI crawlers read your HTML without running JavaScript, and on about 1 in 7 sites we checked, at least one page arrives nearly empty.

Open your pricing page in a browser and every plan is there. Now open View Source and look at what your server actually sent. If the prices are in that file, the crawlers that feed ChatGPT, Claude and Perplexity can read them. If the file is mostly script tags, they can't, because they don't run your JavaScript.

That much has been known since December 2024, when Vercel and MERJ studied more than half a billion GPTBot fetches. Of the major AI crawlers, only Google's and Apple's render the page; the rest read the HTML as served. Netlify confirmed the pattern again in July 2026, and pointed out that AI app builders such as Lovable and Bolt turn out a lot of sites that only come together in the browser.

If you're picking a framework, or wondering whether the one you picked costs you AI readers, the useful question is how often real sites end up in the second case. We couldn't find anyone who had measured it. On November 1, 2026 we did, on the sites people have audited with Sitedex.

Most pages arrive with their text. Some don't.

For each site we requested the homepage and a few other pages the way a non-rendering crawler does: one plain request, no scripts run. Then we counted the readable text in what came back. A page with under 500 characters, roughly 80 words, counts here as arriving with almost nothing to read.

Pages that arrived with almost no readable text before JavaScript ran, measured 2026-11-01
What we checkedArrived nearly empty
Homepagesabout 1 in 40
Other pages on the same sitesabout 1 in 12
Sites with at least one such pageabout 1 in 7
Sites with nothing readable on any page we checkedabout 1 in 70

Mostly, this is good news. These sites send their words in the HTML, and the warning that the web has gone client-side doesn't show up here. Where we could compare a page as sent against the same page drawn in a browser, the raw HTML carried nearly all of the text about 9 times in 10.

The catch is where the misses sit. The homepage, the page everyone tests, was the least likely to come back empty. Pages further in were about three and a half times as likely to.

Two limits. These are established sites, each with at least 30 pages, that someone chose to audit for AI readiness, and AI and developer-tool companies are over-represented, so read the rates as a picture of reasonably built sites rather than a census of the web. And 500 characters is a floor. A page can clear it on menu and footer text alone and still load its price table from a script.

What it means for your site

The framework's name matters less than one decision: does the server send the text, or does the browser build it? Next.js, Nuxt, Astro, SvelteKit and Remix can all render pages on the server or generate them at build time. A plain single-page React or Vue app renders in the browser by default, and so does much of what AI app builders produce.

You don't have to server-render everything. A logged-in dashboard can stay as it is; no crawler is reading it. The pages that need to arrive as text are the ones that answer what a buyer asks before they ever talk to you:

  • Pricing. What it costs, what each plan includes, where the limits are.
  • Docs. How it works and what it connects to.
  • Comparison pages. How you differ from whatever else they're looking at.
  • Contact and how to buy. Who to talk to, and the next step.

If a rewrite isn't on the table this quarter, a prerendering layer can hand crawlers a rendered copy; Netlify offers a free one. It works, but it's one more thing to keep in step with the live site. Sending the text from the server leaves one copy to keep right.

Google is the exception, since its crawler renders JavaScript. Its guidance for AI features, updated in July, asks for nothing new: a page has to be indexed and eligible to show with a snippet. Most other AI crawlers read the page as your server sent it.

How to check your own site

The quick test takes a minute. Pick a plain sentence from your pricing page, one with no links or bold in it, and search for it in the page source, or from a terminal:

curl -s https://yoursite.com/pricing | grep "a sentence from the page"

If it comes back, a crawler that doesn't run JavaScript can read it. If it doesn't, that sentence only exists in the browser. Do the same for your docs, comparison and contact pages. Don't stop at the homepage, since it's the page most likely to pass.

A Sitedex audit runs this count on your homepage as its Server-rendered content check, and shows how many readable characters your server sends before any script runs. The rest of the report takes on the question this study can't: whether the text that does arrive answers what buyers ask. Your first audit is free.

Run your first audit free →


Method

On 2026-11-01 we took every active site in the Sitedex index with at least 30 pages read in earlier audits, and requested each site's homepage plus up to four of those pages. Each page got one plain request, with a standard desktop Chrome user agent, redirects followed, a 20-second timeout and no JavaScript. A site that serves a prerendered copy only to named crawlers would look emptier to us than to those crawlers, so if anything these rates run high. We counted visible characters after removing scripts, styles, noscript blocks and tags, using the same counting rule as the Server-rendered content check, and drew the line at 500. In all, 1,110 pages across 222 sites; the rates use the 213 homepages, 691 other pages and 215 sites that returned a usable page. Requests refused with a 403, and links that now return a 404, were left out rather than counted as empty, and so were pages that were only a redirect to another page, since a crawler follows the redirect. The comparison with rendered pages covers the sites where we still held a browser-rendered copy of the same page. We measured what servers sent, not what any AI model does with it.