Skip to content
English
Contact us
  • There are no suggestions because the search field is empty.

Verify AI tools can read your pages (JavaScript)

Most AI crawlers don't run JavaScript, so any content that only appears after JavaScript runs is invisible to them — even when the page looks perfect in your browser. This how-to gives you a 60-second test to see exactly what an AI crawler sees, and what to do if your content isn't there.

Why this happens

When you open a web page, your browser does two things. First, it downloads the raw HTML the server sends. Then, for many modern sites, it runs JavaScript that fills in the actual content (text, product details, prices, FAQs) a moment later. You don't notice the gap because it happens in a fraction of a second.

There are broadly two ways a site can deliver its content:

  • Server-side rendering (or static HTML): the server sends a complete page with the content already in the raw HTML. Older sites, most WordPress sites, and frameworks configured for server rendering work this way.
  • Client-side rendering: the server sends a near-empty shell (often just
     
    plus some scripts), and JavaScript builds the visible content afterward in the browser. Common with single-page apps built in React, Vue, or Angular.

A human browser handles both perfectly. Most AI crawlers only do the first part. They request the page, read the raw HTML that comes back, extract whatever text is there, and move on. They generally don't run the JavaScript, don't wait for content to load, and don't retry. If your content lives only in the client-side-rendered version, the crawler sees the empty shell and leaves with nothing.

A widely-cited analysis by Vercel and MERJ looked at more than 500 million fetches by OpenAI's GPTBot and found no evidence of JavaScript execution. The same pattern held for Anthropic's ClaudeBot, PerplexityBot, and others.

The trap that hides the problem

Google renders JavaScript; the AI crawlers mostly don't. Googlebot runs your scripts, builds the full page, and indexes the finished result. So a client-side-rendered page can rank perfectly well in Google Search while being a blank shell to ChatGPT, Claude, and Perplexity.

Every signal you normally watch says things are fine. The page ranks, it gets traffic, it looks great. One URL, two very different readers, two completely different outcomes.

One commonly-noted exception: Google's own AI, Gemini, uses Google's existing rendering infrastructure, so it can process JavaScript where the others can't. But the platforms that account for most AI chatbot usage today read raw HTML only.

How to check what AI tools actually see

Step 1 — The 30-second View Source test

Open any important page on your site in your browser. Right-click anywhere and choose View Page Source (not "Inspect", which shows the rendered version and will mislead you). A new tab opens showing the raw HTML the server sent — essentially what an AI crawler receives.

Use your browser's find function (Ctrl+F or Cmd+F) and search for a sentence you can see on the page: a product description, a headline, an FAQ answer. If you find that text in the source, the crawler can see it too. If the source is mostly script tags and an empty , and your visible text isn't there, your content is being added by JavaScript and is likely invisible to AI crawlers.

Do this for a few of your most valuable page types — different pages can behave differently.

Step 2 — The "JavaScript off" test

For a clearer picture, view the page the way a crawler would: with JavaScript disabled. Do this through your browser settings, or with a browser extension that toggles JavaScript on and off. Reload the page with JavaScript off. Whatever remains visible is roughly what AI crawlers get. If the page goes blank or loses its main content, that's your answer.

Step 3 — Pay closest attention to your highest-value pages

The pages most likely to be cited by AI (product pages, comparison pages, pricing, FAQ and documentation pages) are unfortunately also among the most likely to load content dynamically. Prioritize testing those. A marketing homepage that's mostly static might be fine while your dynamically-loaded product catalog is invisible.

Step 4 — Confirm in your server logs

The View Source test tells you what crawlers can see. Your server logs tell you what they're actually doing. After testing, check your access logs (or your CDN's bot analytics) for AI crawler names like GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, and PerplexityBot. If you see these bots repeatedly hitting pages that you've confirmed have thin raw HTML, that's a direct signal you're losing visibility to the rendering gap. There's no "Search Console for AI bots" — logs are your main window.

How to fix it

If your testing shows content missing from the raw HTML, the goal is simple: get your important content into the HTML the server sends, before any JavaScript runs. Three common approaches, roughly from most to least robust:

  • Server-side rendering (SSR): configure your framework to assemble pages on the server. Next.js (for React), Nuxt (for Vue), and Angular's server-rendering setup all support this natively. Usually the cleanest long-term fix.
  • Static generation: if your content doesn't change constantly, pre-build pages into static HTML at deploy time. Same frameworks support this; simplest and fastest where it fits.
  • Prerendering: a service or middleware detects crawlers and serves them a fully-rendered HTML snapshot while humans still get the JavaScript app. Works without rewriting your app, though it adds a moving part to maintain.

Which one fits depends on your stack and how often your content changes. The point of this article is the diagnosis; the fix is an engineering decision.

One caution: if you serve crawlers meaningfully different content than you serve humans, that can stray into "cloaking", which search engines discourage. The accepted versions of these techniques serve the same content, just in a form crawlers can read.

Key takeaways

  • Most AI crawlers (GPTBot, ClaudeBot, PerplexityBot) don't execute JavaScript — they read raw HTML only.
  • Google indexes the rendered page; AI doesn't. A page that ranks fine on Google can be invisible to AI chatbots.
  • Run the View Source test on your highest-value pages and search for visible text in the raw HTML.
  • If your important content isn't in the raw HTML, switch to server-side rendering, static generation, or prerendering — and serve crawlers the same content you serve humans.