Next.js SEO · Part 3 · Crawlers and link previews

How Google and social media crawlers read HTML

How apps like WhatsApp, LinkedIn and Slack build link previews from a page's HTML, what Open Graph tags are, and how that differs from Google Search.

Why link previews matter

Imagine sharing a link to a website in a WhatsApp group.

Sometimes the link shows a rich preview with an image, a title and a description. Other times, it shows only the URL. Why?

Apps like WhatsApp, LinkedIn, Slack and X can request information from a website to create a link preview.

Search engines also read information from a website, but they use it for different purposes.

What is inside the head

Here is an example of a page's head:

HTML
<head>
  <title>Learn Python - Beginner Tutorial</title>

  <meta
    name="description"
    content="Learn Python programming with simple examples."
  />

  <meta
    property="og:title"
    content="Learn Python from Scratch"
  />

  <meta
    property="og:description"
    content="A beginner-friendly Python tutorial."
  />

  <meta
    property="og:image"
    content="https://example.com/python-cover.png"
  />
</head>

The tags fall into two groups.

1.General page metadata

<title> and <meta name="description"> describe the page. Google can use them when generating search result titles and snippets.

2.Open Graph metadata

Tags beginning with og: help apps such as LinkedIn, Slack and other social platforms create link previews.

TagWhat it tells an app
og:titleThe preview title
og:descriptionThe preview summary
og:imageThe preview image
og:urlThe page's preferred URL for the shared object

X also supports Twitter Card metadata, such as twitter:card and twitter:image.

Google and social media crawlers

Both can read HTML, but they don't necessarily use the same information.

ServiceMain purposeImportant information
Google SearchDiscover and evaluate pages for search resultsPage content, title, description, links and more
LinkedInCreate a shared-link previewOpen Graph metadata
SlackShow a link preview in messagesPage metadata, including Open Graph
XCreate a post's link cardTwitter Card and supported Open Graph metadata
AI crawlersRetrieve or discover web content for different AI-related purposesPage content, links and sometimes metadata

Inspecting what a crawler receives

Developers use a terminal command called curl. It can send an HTTP request to a website and display the response. For example:

Terminal
curl -A "Twitterbot" https://example.com

Here is what each part means:

  • curl: makes a request to a URL.
  • -A: sets the User-Agent, a text label identifying the requesting client.
  • "Twitterbot": the User-Agent label being sent.
  • https://example.com: the URL being requested.

This shows the HTML returned when a request identifies itself as Twitterbot.