Googlebot vs Google-Extended: Search and Gemini Controls

The essential distinction: Googlebot is the Search crawler; Google-Extended is a control token

The easiest way to get Googlebot vs Google-Extended wrong is to assume the names identify two crawlers doing parallel jobs. Google’s documentation describes something subtler.

Googlebot is a crawler. More precisely, it is the program Google Search uses to fetch pages from the web. Google documents two variants, Googlebot Smartphone and Googlebot Desktop, which simulate visits from mobile and desktop devices. Both use Googlebot as their robots.txt product token. The variants affect how Google fetches and renders a page; they do not turn into separate policy controls in robots.txt. Google’s Googlebot documentation defines them as two forms of the web crawler used by Google Search.

Google-Extended is different. Google describes it as a robots.txt product token or control. It governs whether content that Google has crawled may be used for specified downstream purposes, including helping train future versions of Gemini models. Google also describes Google-Extended as a way to limit AI training and grounding in some of its other systems. It is not documented as the crawler Google Search uses to fetch your pages.

So the revealing part of the comparison is that it is not really crawler versus crawler. It is crawler versus use control.

This distinction matters because crawling and later use are separate events. Google says it may reuse crawled data rather than make wasteful repeat requests, while continuing to respect robots.txt choices such as Google-Extended. The same Google crawling documentation says using Google-Extended does not affect inclusion in Search and is not a Search ranking signal.

Google’s update history also associates Google-Extended with Gemini Apps, while explicitly saying it does not affect Google Search. The official documentation does not give a complete, current list of every Google product or workflow covered by the token, so it would be too broad to say Google-Extended controls all Gemini access or all Google generative AI use.

Some third-party pages call Google-Extended a dedicated crawler or user agent. That description conflicts with the narrower first-party classification. Where the labels disagree, Google’s own classification is the useful one: Googlebot is the established Search crawler; Google-Extended is a robots.txt control token for specified uses of crawled content.

What Googlebot access means inside Google Search

Allowing Googlebot answers one question: may Googlebot fetch the covered URL? It does not answer all the questions that come after that.

Google describes Search as a three-stage process: crawling, indexing, and serving. During crawling, Googlebot downloads discovered pages and their resources. Google may render a page with a recent version of Chrome and run its JavaScript, because some content does not appear until scripts run. During indexing, Google analyzes text, metadata, images, videos, and other page information; it may select a canonical page and store information in the Google index. During serving, Google looks in that index for information relevant to a user’s query. These stages are set out in Google’s guide to how Google Search works.

Each step is conditional. Permission to crawl does not guarantee that Google will crawl. A crawl does not guarantee indexing. Indexing does not guarantee that a page will be served, much less served for every query. Google explicitly says it does not guarantee crawling, indexing, or serving even when a page follows its Search requirements.

This is why a Googlebot rule is best understood as a Search-pipeline decision, not a blanket promise about visibility. If Googlebot cannot crawl a page, Google cannot fetch and render its contents for the normal Search process. If Googlebot can crawl it, the page can move to later decisions—but those decisions still belong to indexing and serving systems.

The same distinction explains an often confusing robots.txt result. A robots.txt Disallow controls crawling; it is not a reliable way to keep a URL out of Search. Google may discover a blocked URL through links elsewhere and show the URL without a description, even though it cannot crawl or index the blocked page’s contents. Google’s robots.txt introduction therefore points to noindex or password protection when the goal is not merely to stop crawling.

A noindex directive can appear in a robots meta tag or an X-Robots-Tag HTTP header. Other directives can limit snippets or previews. But there is a catch (and it is an important one): Google must be able to crawl the URL to discover those directives. If robots.txt blocks the page, Google cannot read the page-level or header-level indexing instruction, so that instruction will not be found and followed. Google explains this dependency in its robots meta tag documentation.

Googlebot access also underlies eligibility for Search’s AI features. Google says a page must be indexed and eligible to appear in Search with a snippet before it can be eligible as a supporting link in AI Overviews or AI Mode. This is eligibility, not guaranteed selection.

Thus “allow Googlebot” means something narrower than “put this page in Google,” but more consequential than “permit one HTTP request.” It leaves the door open to crawling, rendering, indexing, ordinary Search results, and Search AI-feature eligibility; every later stage still has its own test.

What Google-Extended changes for Gemini-related use

Google-Extended acts farther downstream. Google says publishers can use it to control, among other things, whether crawled content helps train future versions of Gemini models. Its Search AI-feature documentation also points to Google-Extended for limiting AI training and grounding in some other Google systems.

Training and grounding are not the same thing. Training is the process of building or improving a model from data. Grounding is the use of outside information to help a system produce a response tied to supporting material. Google groups both under the documented purpose of Google-Extended in some systems, but the supplied documentation does not expose the full implementation behind either one.

The relationship to Gemini is direct but bounded. Google’s documentation update history says the change from Bard to Gemini Apps clarified that Gemini Apps is affected by Google-Extended. Earlier wording described the token as a control over whether sites help improve Bard and Vertex AI generative APIs, including future model generations powering those products.

What does allowing Google-Extended establish, then? It establishes permission for the category of use Google documents. It does not establish that the content will be used to train a model, selected for grounding, retrieved for a particular prompt, quoted in Gemini, or reflected in any given answer.

Permission is not selection.

The documentation is especially thin on request paths. It does not fully say whether every Gemini product fetches web content through a separate network crawler, through standard Google requests, through stored crawled data, or through some mix of paths. Nor does it provide a complete product-by-product account of Gemini retrieval and response generation. Describing Google-Extended as a dedicated Gemini crawler would fill those gaps with an assumption.

There is also a clean Search boundary. Google says Google-Extended does not affect Search inclusion and is not a Search ranking signal. That means blocking the token is not documented as an instruction to remove pages from the Google Search index. Conversely, allowing it is not a way to improve Search rankings.

The practical question has changed again. It is no longer merely, “Can Google fetch this page?” It is, “For which documented purpose may Google use content after it has been crawled?” Google-Extended addresses part of that second question, not every possible use of the content and not every way Gemini might obtain information.

Google Search and Gemini are related, but their documented access controls differ

Google Search now contains generative AI features, which makes the product names easy to blur. But AI Overviews and AI Mode operate inside Google Search; they are not interchangeable with Gemini Apps.

Google says AI Overviews and AI Mode can issue multiple related searches—a process it calls query fan-out—and surface supporting web links. To be eligible as one of those links, a page must be indexed and eligible to appear in Google Search with a snippet. Google therefore identifies Googlebot directives as the robots.txt control for how sites are crawled for Search. Its Search AI-feature documentation separately points to Google-Extended for AI training and grounding in some other systems.

The comparison looks like this:

Service or purpose	Identifier or control	Content stage	Documented outcome
Google Search	Googlebot	Discovery, crawling, and rendering before possible indexing	Access can support indexing and Search eligibility; none is guaranteed
AI Overviews and AI Mode	Googlebot plus Search indexing and presentation controls	Search index and Search eligibility	An indexed, snippet-eligible page may qualify as a supporting link
Specified Gemini and other generative AI uses	Google-Extended	Use of content after crawling	Controls specified training and grounding uses without affecting Search inclusion or ranking

This table maps documented controls, not Google’s internal model architecture. It does not prove that Search and Gemini maintain wholly separate stores of content, nor that every Gemini workflow uses one fetcher. The public documentation simply does not settle those implementation questions.

Claims about AI Overviews need particular care. Some third-party sources say blocking Google-Extended removes content from AI Overviews; others say it does not. Google’s current official material places AI Overviews and AI Mode inside Search, makes their link eligibility depend on Search indexing, and names Googlebot as the Search crawling control. It describes Google-Extended separately as a control for training and grounding in other systems.

The official baseline therefore does not support treating Google-Extended as the control for AI Overviews. But it also does not disclose every data pathway behind every generated Search response, so a broader claim about all possible uses would go beyond the evidence.

The central comparison is functional. Googlebot controls an upstream Search access step. Google-Extended controls specified downstream uses. They can both concern the same website content without doing the same job.

How robots.txt rules apply—and what allowing or blocking one does not imply

A robots.txt file groups rules beneath a User-agent field. Allow identifies a path that may be accessed; Disallow identifies a path that may not be accessed by the crawler or product token named in that group. Google’s robots.txt specification says its automated crawlers download and parse this file before crawling.

Google chooses the most specific matching user-agent group. A specific group is not combined with the global * group, though rules from multiple groups for the same specific user agent are combined. Group order does not matter. For paths, * matches zero or more characters and $ marks the end of a URL. If more than one path rule matches, Google uses the longest, most specific match; where conflicting rules are equally specific, it uses the least restrictive one.

These generic mechanics matter, but they do not establish a hidden precedence rule between Googlebot and Google-Extended. The official material supports treating them as separate policy controls for separate documented purposes. It does not describe one as an automatic fallback for the other, and the supplied passages do not contain a complete Google-Extended-specific configuration example covering every combined allow/block case.

The supported practical outcomes are narrower:

• Restricting Googlebot prevents Googlebot from crawling the affected paths. That can impair content indexing and Search eligibility, including eligibility to appear as a supporting link in AI Overviews or AI Mode. A blocked URL may still be discovered and shown without a snippet.
• Restricting Google-Extended limits the specified training and grounding uses associated with that token. Google documents this choice as not affecting Search inclusion or serving as a Search ranking signal.
• Allowing Googlebot does not by itself grant or prove every downstream Gemini-related use.
• Allowing Google-Extended does not guarantee use in training, grounding, Gemini retrieval, or any generated response.

There are a few mundane limits worth keeping in mind. A robots.txt file applies only to the same host, protocol, and port on which it is served. Google generally caches the file for up to 24 hours and may keep it longer when refreshing it is not possible, so a changed rule need not take effect at once.

More fundamentally, robots.txt is a voluntary crawler protocol, not an access-control system. Google says its standard crawlers follow it, but other crawlers may not. It cannot make public content secret. A URL blocked from crawling may still be discovered through links or other public information, and a rule for one crawler identity does not automatically govern every other crawler.

Nor should robots.txt be confused with an indexing instruction. If you block Googlebot and place noindex on the same page, the crawl block can prevent Google from seeing the noindex. The controls do different jobs; stacking them without regard to sequence can defeat the very instruction you expected Google to read.

Request identification: do not confuse a robots.txt token with a crawler signature

A robots.txt token tells a compliant system which policy applies. It does not authenticate an incoming HTTP request.

Google says its crawlers identify themselves through three signals: the HTTP User-Agent header, the source IP address, and the reverse-DNS hostname of that IP. Its crawler overview and verification documentation presents these together because a User-Agent string alone is easy to spoof.

This matters if you are reading server logs or building an allowlist. A request claiming to be Googlebot is not necessarily from Google. Google’s own Googlebot guidance recommends checking reverse DNS or comparing the source address with Google’s published IP ranges before deciding that troublesome traffic is legitimate Google traffic.

Google-Extended adds another reason not to equate tokens with signatures. Google officially establishes it as a robots.txt control, but the supplied first-party evidence does not establish a distinct Google-Extended HTTP User-Agent or universal log signature. Cloudflare’s crawler analysis likewise notes that robots.txt tokens do not always appear as substrings in HTTP User-Agent values and says Google-Extended-related traffic may use standard Google user agents. That observation helps explain the log problem, but it does not disclose a universal Google implementation.

So a User-agent: Google-Extended group in robots.txt does not prove that you should expect a request carrying that exact text. And seeing Googlebot in a request does not prove either its source or every downstream purpose for which fetched data may be considered.

The safe conceptual order is policy first, authentication second. Use the token to state the policy; use the request’s User-Agent, source IP, and DNS information together to assess whether the underlying request is genuinely from Google.

Google has also documented an experimental mechanism for cryptographically signed bot requests. But Google describes it as early-stage; User-Agent and IP-based verification remain the present practical standard. It is not yet a general replacement for the ordinary checks, and it does not turn Google-Extended into a distinct request identity.

The boundary of what the documentation establishes

The firm part is fairly simple. Googlebot is the documented crawler for Google Search. Its access supports the crawl and render stage that can lead to indexing, serving, and eligibility for Search features, including AI Overviews and AI Mode. Google-Extended is a separate robots.txt product token controlling specified AI training and grounding uses of crawled content, with a documented connection to Gemini Apps.

Google says blocking Google-Extended does not affect Search inclusion and is not a Search ranking signal. Current official documentation does not establish Google-Extended as the control for AI Overviews; conflicting third-party claims should not be promoted to fact.

Beyond that, the map gets incomplete. Google does not publicly establish in the supplied material the full list of products covered by Google-Extended, every Gemini grounding or retrieval path, a distinct Google-Extended request signature, or every outcome created by combined Googlebot and Google-Extended rules. The controls are separate by documented purpose, but that does not reveal all the machinery beneath them.

This leaves four questions where people often collapse everything into one. Can Google crawl the page? Can the page be indexed and served in Search? May its crawled content be used for specified AI training or grounding? Will it appear in or influence a particular generated response?

Those are four different questions. Googlebot helps answer the first and bears on the second. Google-Extended addresses part of the third. Neither permission, by itself, answers the fourth.