Next.js SEO · Part 4 · AI crawlers

What are AI crawlers

How AI crawlers differ from Googlebot, what they can read in a page's HTML, and why content in the initial HTML matters.

What is a crawler

A crawler is an automated program that visits web pages and reads information from them. Googlebot, from Part 1, is one example.

But Google isn't the only company that uses crawlers. AI companies also operate crawlers for different purposes, such as discovering information for AI-powered search or collecting publicly available content for model development.

Googlebot and AI crawlers

Imagine a website that teaches JavaScript.

learnjavascript.com

Googlebot
Discovers pages for Google's search indexing systems
AI crawler
May collect or retrieve content for an AI-related service

Both may request the website's HTML, but what they do with it can be different.

For example, OpenAI documents different web-related user agents:

Crawler	Main purpose
OAI-SearchBot	Discovering websites for ChatGPT search
GPTBot	Crawling content that may be used to improve AI models
ChatGPT-User	Fetching pages for certain user-initiated ChatGPT requests

These have different purposes and controls. See OpenAI's crawler documentation for details.

What AI crawlers can read

Suppose a page contains this HTML:

index.html

<!DOCTYPE html>
<html lang="en">
  <head>
    <title>JavaScript for Beginners</title>
    <meta
      name="description"
      content="Learn JavaScript with simple examples."
    />
  </head>

  <body>
    <h1>Learn JavaScript</h1>
    <p>JavaScript makes websites interactive.</p>
    <a href="/lessons">View all lessons</a>
  </body>
</html>

A crawler that successfully retrieves this HTML can potentially read:

Page title: JavaScript for Beginners
Description: Learn JavaScript with simple examples.
Heading: Learn JavaScript
Paragraph: JavaScript makes websites interactive.
Link: /lessons

But there is an important detail: different crawlers have different capabilities. Some mainly read the HTML returned by the server. Others may process additional resources or render JavaScript.

Why HTML matters for AI crawlers

Imagine two websites.

1.Content in the HTML

Website A

<h1>JavaScript Basics</h1>
<p>Learn variables and functions.</p>

A crawler can find this content directly in the HTML response.

2.Content added later

Website B

<div id="app"></div>
<script src="/app.js"></script>

If the content is generated only after JavaScript runs, a crawler that doesn't execute JavaScript may miss it.

AI visibility has no guarantees

Does adding AI metadata guarantee ChatGPT will recommend a website? No.

Just as traditional SEO doesn't guarantee first place on Google, making a website accessible to AI crawlers doesn't guarantee it will be cited, recommended or included in an AI-generated answer.