By Mehdi HassanWhy might a JavaScript-rendered website be invisible to AI crawlers?
A JavaScript-rendered website can be invisible to AI crawlers because those crawlers read the HTML the server sends and stop there. In a client-rendered app, that HTML is an empty container and a script tag; the browser downloads the script, runs it and only then builds the page. A crawler that doesn't run the script sees the container and nothing else.
Google is the exception that hides the problem. Googlebot renders JavaScript in a second pass, so a single-page React marketing site can rank in Google and still be blank to OAI-SearchBot, ClaudeBot and PerplexityBot. Rendering is the first layer of an AI-search-ready website, because no amount of good writing helps if the words aren't in the HTML.
Crawler | Runs JavaScript? | What it sees on a client-rendered page |
|---|---|---|
Googlebot | Yes, in a later rendering pass | The full page, eventually, if scripts and APIs load |
Bingbot | Partly, with limits | Often the page, but less reliably than Google |
OAI-SearchBot and ChatGPT-User | Generally no | The empty shell and any text in the initial HTML |
ClaudeBot and Claude-User | Generally no | The empty shell |
PerplexityBot | Generally no | The empty shell |
How do you test whether AI crawlers can see your content?
Test by fetching the page the way a non-rendering crawler does and checking whether your content is there. It takes a minute per page and needs no special tools. Test your homepage, one product page, the pricing page and one blog post, since each often uses a different template.
Fetch the raw HTML with curl
Run curl with the -s flag on the page URL and pipe it to grep with a phrase from the page, such as your hero headline or a price. No match means the text isn't in the HTML.
Try a crawler user agent
Repeat with the -A flag set to a crawler user agent string, such as one containing GPTBot. Some sites serve bots differently, and some CDNs block them with a 403.
Disable JavaScript in the browser
In Chrome DevTools, open the command menu, choose Disable JavaScript and reload. A blank or skeleton page is what most AI crawlers get.
Compare view-source with the inspector
View-source shows the server HTML; the Elements panel shows the page after scripts ran. Text that's only in the second one is client-rendered.
Check the details
Look at the title tag, meta description, canonical, JSON-LD and FAQ answers in view-source. These are often injected by scripts and missed.
What a failing result usually looks like
In a typical client-rendered app built with Create React App or a default Vite setup, view-source shows a head with a generic title, a body with a single empty div with an id like root, and one or two script tags. Everything a buyer reads, from the headline to the pricing table, is created after the script runs.
A passing result shows the same headings, paragraphs and table rows in view-source that you see on screen. Interactive parts like a pricing toggle can still be JavaScript, as long as the default state is in the HTML.
What's the difference between CSR, SSR and SSG?
They differ in where and when the HTML is built. Client-side rendering builds it in the visitor's browser, server-side rendering builds it on the server for each request, and static site generation builds it once at deploy time and serves the same file to everyone. Only CSR leaves crawlers with an empty page.
Approach | Where HTML is built | AI crawlers see content? | Best for |
|---|---|---|---|
Client-side rendering (CSR) | In the browser, after JavaScript runs | No | Logged-in app screens |
Server-side rendering (SSR) | On the server, per request | Yes | Personalized or fast-changing pages |
Static site generation (SSG) | At build or deploy time | Yes | Marketing pages, blog, docs |
Incremental static regeneration (ISR) | Statically, refreshed on a schedule or on demand | Yes | Large CMS-driven sites |
Prerendering service | A headless browser snapshots pages for bots | Yes, if configured well | A stopgap for an existing SPA |
For a marketing site, static generation is usually the right default: pages are fast, cache well and are complete for every crawler. Our comparison of static and server rendering in Next.js goes deeper on when each makes sense.
How do you set up SSR for AI crawlers in Next.js?
In Next.js, the fix is mostly about not fighting the framework. In the App Router, components are server components by default, so a page that fetches its content on the server and renders it ships complete HTML. The problem appears when a page is marked as a client component and loads its content in the browser.
- Fetch content in server components. Load CMS data in the page component on the server, not in a browser effect after mount.
- Keep client components small. Mark only interactive pieces, like a menu or a pricing toggle, as client components, and pass them content as props.
- Pre-build dynamic routes. Use generateStaticParams for blog posts and case studies so each one is built as a static page.
- Revalidate rather than re-render. Use time-based or on-demand revalidation so CMS edits go live without making every page dynamic.
- Render metadata and JSON-LD on the server through the metadata API and the page component, so titles and schema are in the HTML.
- Avoid content behind client-only tabs. If tab content is fetched on click, crawlers never see it. Render all tabs in the HTML and toggle visibility.
The Next.js documentation on rendering covers server and client components in detail. For the wider search picture, our JavaScript SEO guide explains how Google handles rendering.
What if your marketing site is a React SPA?
If your marketing site is a single-page React app, you have three options: move it to a framework that renders on the server, add a prerendering layer, or move the marketing pages to a different platform and keep the app where it is. For most startups, splitting the marketing site from the product app is the cleanest fix.
Pros
- Rebuilding in Next.js keeps your React components and fixes rendering at the root.
- Prerendering services can fix crawler visibility quickly without a rebuild.
- Moving marketing pages to Framer or Webflow gives static HTML and easier editing for marketers.
Cons
- A rebuild needs redirects and SEO checks so you don't lose existing rankings.
- Prerendering adds a moving part that can serve stale or broken snapshots to bots.
- Splitting platforms means two codebases and a shared design system to keep in sync.
Our guide to turning a React SPA into an SEO-friendly website covers the migration path, and if a rebuild is on the table, read about keeping AI Overview and ChatGPT citations through a migration.
Do Framer and Webflow sites have this problem?
Generally no. Framer and Webflow both publish pages as pre-built HTML, so the text, headings and CMS content are in what the server sends. The risks on those platforms are smaller and specific: content loaded by embedded third-party widgets, text inside images, and custom code that fetches data in the browser.
If you use an embed for reviews, pricing or FAQs, check it with the same curl test. Embedded widgets often load in an iframe or by script, which means the content belongs to another domain or doesn't exist in your HTML at all. When the content matters for buyers, put it in the page itself. Then run a full AI visibility audit to check crawler access and structure.
