For fifteen years, the job was clear. Rank on Google, earn the click, land them somewhere fast enough they don’t bounce. Every technical decision downstream of that was broadly correct.

Then the front door moved, and a lot of people didn’t notice. More journeys now start with a question typed into an AI assistant than a query typed into a search box. Sometimes that ends with a citation and a click, other times with the answer and no click at all.

Either way, something has to read your website first. And that something does not behave like Googlebot. We’ve noticed this disconnect first-hand, particularly as we expanded our search offering to include Hybrid Engine Optimisation (HEO).

Natasha wrote an excellent piece (link here). However, I’ll be focusing on the technical side, and what you could be losing if you don’t pay attention, particularly for developers out there.

The bit everyone assumes is handled

Googlebot renders JavaScript. It has for years. It loads your page, runs your framework, waits for the content to appear, and indexes what a human would see. That capability quietly underwrote an entire era of front-end architecture. We moved to client-side rendering en masse and it was fine, because the crawler that mattered kept up.

Most AI crawlers never made that investment. When Vercel analysed real traffic across their network in December 2024, none of the major ones rendered JavaScript at all. Across 500 million GPTBot fetches, zero evidence of execution.

The funny detail is that they download JavaScript. ChatGPT’s crawler fetched JS files in 11.5% of requests, Claude’s in 23.84%. They just never run it.

So the question that matters for your site isn’t “is it fast” or “does it rank”. It’s blunter than that:

What’s in the HTML before a single line of JavaScript runs?

<head>
<title>Acme | Digital Transformation</title>
<meta name="description" content="...">
<script src="/_nuxt/entry.a91f2c.js"></script>
</head>

That’s the page. That’s what gets read. Your copy, your service pages, the case study you spent weeks on. All of it arrives a moment later, in a rendering pass that never happens.

Two caveats, because the version of this doing the rounds on LinkedIn is sloppier than it should be. Google’s Gemini is the exception: it runs on Googlebot’s infrastructure, so it renders JavaScript fine. And “AI crawler” is really two jobs.

GPTBot and ClaudeBot build training datasets. OAI-SearchBot and ChatGPT-User fetch pages to answer a question someone is asking right now. They’re controlled separately in robots.txt. Blocking the training crawler is a defensible business decision. Blocking the one that would have cited you this afternoon is usually just an accident.

To be fair to the frameworks

Client-side rendering won for real reasons. Better developer experience, easier interactions, component models that made large sites maintainable in a way the old templating spaghetti never did. I’d make most of those calls again tomorrow.

The mistake, if there even is one, is subtler. We adopted an architecture on the assumption that the crawler will run our JavaScript. That held completely, right up until a new category of crawler arrived with no intention of doing so. Nobody made a bad decision. The ground moved underneath a good one. Which is exactly why it gets missed: there’s no error, nothing is broken, your monitoring is green and your rankings haven’t moved.

What it costs

The numbers cut both ways, so here’s the honest version.

AI referral traffic is still small. Conductor’s 2026 benchmark, across 13,770 domains and 3.3 billion sessions, puts it at 1.08% of all website traffic. Anyone telling you your traffic is about to collapse is selling something.

But look at what that 1% does. When Ahrefs published their own analytics in mid-2025, AI search was 0.5% of their visits and 12.1% of their signups. Twenty-three times the conversion rate of regular organic. Conductor found those visitors stay 2.3x longer, and that 1.08% grew roughly 340% year on year.

Small, growing fast, and converting like nothing else you’ve got. Which makes sense when you think about the mechanism. Someone arriving from “which agency should I talk to about X” has been pre-qualified and effectively recommended before they ever reach you. That isn’t an impression. It’s a warm introduction.

And losing it is worse than a ranking drop, because a ranking drop is visible. There’s no position 11 to watch here. If an assistant can’t read your page it doesn’t flag an error, it just reaches for whoever it could read, quite possibly a competitor with a server-rendered site, and cites them instead. Do that enough times and you haven’t lost a visit, you’ve lost a default.

Ten seconds to find out where you stand

Don’t take any of that on trust. The Vercel research is the best data we have and it’s also from December 2024, and most of what’s been written since quotes it rather than retesting. So test your own site.

Not in DevTools, which shows you the page after JavaScript has run. Use curl:

curl -s https://yoursite.com | grep -i "a sentence from your homepage"

Empty means you’ve just seen your site the way most AI crawlers see it. If your content’s there, you’re fine. Go and worry about something else.

If it came back empty, in rough order of payoff:

Server-render the pages that argue your case. Not the whole app. Services, case studies, pricing, the blog. Next, Nuxt, SvelteKit and Astro all do SSR or static generation per route. This is a config decision far more often than a rewrite.
Put the answer near the top. These systems extract, they don’t hunt. State your point plainly in the first paragraph, under a heading that matches the question someone would actually ask.
Check what you’re blocking. Plenty of sites refuse AI crawlers in robots.txt because it arrived with a plugin nobody revisited. Worth knowing which crawlers, and whether you meant it.
Measure the channel. Segment AI assistant referrals in your analytics. You can’t argue for the work, or prove it landed, on a number nobody’s watching.

The honest version

Nobody’s site is about to fall off a cliff over this. Google isn’t gone, the traffic hasn’t evaporated, and the panicked version of this argument is partially click-bait.

But a gap is opening between sites that are legible to machines and sites that only look legible because one very sophisticated crawler has been carrying them for a decade. It’s cheap to close now and gets more expensive the longer it stays open, because the citations forming this year are sticky.

We spent years making sites fast for people. The overlooked bit is that being read at all comes first. A fast site nobody can read is just a very well-optimised empty div.

Worth ten seconds with curl to find out which one you’ve got.

Need a hand?

References:

The rise of the AI crawler, Vercel, 17 December 2024
Does AI Search Traffic Convert Better Than Traditional Search?, Ahrefs, 16 June2025
The 2026 AEO / GEO Benchmarks Report, Conductor, January 2026

Michael Watts

Technical Lead

Follow Effect Digital

About Me

Michael is a Technical Lead at Effect, working mainly in Laravel and Nuxt on large multi-tenant platforms. He spends as much time reviewing other developers' work and shaping scope with clients as he does writing code. Outside work he follows baseball, which takes some commitment from the UK.

Next Insight

Adding AI to Your Customer Journey Isn’t Always the Answer

6 min read

You Optimised Your Site for Google. The Thing Replacing Google Can't Read It.