AI Crawlers Explained: Roles, Behaviour, and Access Controls

The phrase “AI crawler” sounds more precise than it is.

It suggests there is one new kind of bot, with a standard way of identifying itself, fetching pages, obeying rules, and using what it finds. There isn’t. GPTBot, ClaudeBot, OAI-SearchBot, ChatGPT-User, and Google-Extended may all appear in a discussion of AI crawlers, but they do not all have the same role—or even necessarily the same technical function.

The useful question is not merely whether a request came from an AI company. It is what identity the request claims, what purpose the operator documents, how access is controlled, what behavior you can observe, and what downstream use remains beyond your view.

Those are separate facts.

AI crawlers are defined by AI-related use, not by one universal bot type

A workable definition is simple: an AI crawler is automated software that accesses, reads, extracts, or analyzes web content for some AI-related downstream use. That use might be model training, AI search, content indexing, live retrieval, summarization, or another AI-powered service. This is an industry working definition, not a universal technical standard; the label describes why the material is being gathered more than it describes one kind of software architecture.

That matters because several quite different things are often bundled together as “AI bots.”

A training crawler gathers web material that may be used to build or improve models. A search or indexing crawler gathers material to support searchable results in an AI product. A user-triggered retrieval system fetches a page because someone has asked an AI service for specific or current information. Operators including OpenAI, Anthropic, and Mistral publish separate identities for some of these roles.

These roles can overlap at the system level. OpenAI says, for example, that when a site permits both GPTBot and OAI-SearchBot, it may use the result of one crawl for both use cases to avoid duplicate crawling. But that does not erase the separate controls or make training and search the same purpose.

A conventional search crawler is usually described as discovering and cataloging pages so they can appear in ordinary search results. An autonomous agent has a different job: it performs actions on a user’s behalf and may continue after the initial instruction. A scraper usually means software aimed at extracting selected pieces of data, though in practice the boundary between scraper and crawler is loose. Ordinary browser traffic generally represents a human visit; a person clicking a link supplied by an AI service is referral traffic, not the crawler request that may have preceded it.

So there are at least four signals you can use to classify traffic:

• Identity: the user-agent or product token it claims.
• Stated purpose: training, search, user action, or something else documented by the operator.
• Access method: automatic crawling, a user-triggered fetch, or another request path, along with mechanisms such as robots.txt.
• Observed behavior: the URLs requested, timing, volume, response outcomes, rendering, and apparent compliance with instructions.

None proves all the others. A user-agent is a claim about identity, not authentication. An operator’s documented purpose describes intent, not every use of every fetched response. A burst of requests is behavior, not proof of training. And allowing one token in robots.txt says nothing by itself about what another token may do.

Even familiar behavioral contrasts have to be handled carefully. Some measurements describe AI crawler traffic as intensive or unpredictable; other measurements find that named crawlers respect robots.txt. Rendering also varies. These are tendencies seen in particular crawlers and datasets, not properties that define an AI crawler.

The category is therefore purpose-based and fuzzy at the edges. That is less tidy, but more useful.

How an AI crawler request moves from a task to downstream use

There is a general path an AI crawler request can take, though not a universal lifecycle.

Some task starts the process. It might be a URL selected for broad crawling, a page due for recrawling, an indexing job, or a request triggered by a user. The client connects to the host and sends an HTTP request. The server, CDN, or edge service returns a response—or denies, redirects, throttles, or challenges it. The crawler may then read the returned text, images, structured data, and other files. Some systems render the page; others work mainly from the response they received. Extracted material then passes to an indexing, retrieval, training, or other product system.

Almost every step can vary.

URL selection may follow links, a stored queue, a sitemap, prior crawl history, or a user-supplied target. Crawl rates and revisit schedules differ. A crawler may use caching or conditional requests, fetch supporting files, stop at a file-size limit, or fail to see material created only after JavaScript runs. Restricted content may encounter robots.txt rules, authentication, a firewall, a CAPTCHA, or a rate limit.

Google’s documentation gives a useful example of how detailed one operator’s implementation can be. Google says its crawlers and fetchers run across many machines and data centers, can use HTTP/1.1 or HTTP/2, and may or may not use caching depending on the product. It also documents file-size limits and the use of cache-related headers in recrawl decisions. These are Google details, not a specification for AI crawlers generally.

Rendering is similarly uneven. A Vercel study of selected sites and network traffic reported that the major AI crawlers it measured did not execute JavaScript, even when some fetched JavaScript files. The same study said Google’s Gemini-associated infrastructure and AppleBot could render JavaScript. The right conclusion is not that AI crawlers cannot render JavaScript. It is that you have to check the crawler and the measurement date rather than assume.

Robots.txt handling varies too. OpenAI documents separate controls for its crawlers, and a Cloudflare test reported that ChatGPT-User fetched robots.txt and stopped when disallowed. The same Cloudflare report attributed undeclared, evasive crawling to Perplexity after its declared crawler was blocked. These observations concern specific tests and operators; they do not justify saying either that all AI crawlers obey robots.txt or that none do.

Then there is the user-agent string. It is useful because it tells your server what the requester claims to be, and it lets robots.txt or infrastructure rules target that claim. But the value can be changed. When authenticity matters, it needs corroboration.

Google’s verification procedure, for example, combines the request’s user-agent with its source IP and DNS information. You perform a reverse DNS lookup on the IP, confirm that the resulting hostname belongs to an expected Google domain, and then perform a forward lookup to see whether it resolves back to the original IP. Google also publishes crawler IP information. Other operators may provide different or less complete ways to verify requests.

This leads to a slightly surprising answer to “how do AI crawlers work?” At the network level, they often work like other automated web clients. What makes them AI crawlers is mainly the downstream job; what makes any particular request trustworthy is a separate verification problem.

OpenAI’s crawler roles: GPTBot, OAI-SearchBot, and ChatGPT-User

Calling every OpenAI-related request “GPTBot traffic” throws away the most useful distinction.

GPTBot is the OpenAI crawler associated with content that may contribute to training OpenAI’s generative AI foundation models. This is the training-related identity. OpenAI’s documentation lets a site owner use the GPTBot robots.txt token to indicate that crawled content should not be used for that purpose.

OAI-SearchBot has a search role. OpenAI describes it as the independently controlled identity relevant to appearing in search results; secondary documentation describes it as supporting links, surfaced websites, and citations in ChatGPT search. It should not be treated as another name for GPTBot.

ChatGPT-User is attributed a user-triggered retrieval role: fetching a page when someone asks ChatGPT or a Custom GPT to visit or read a specific URL. OpenAI’s supplied first-party passage says its product actions may be automatic or user-triggered but does not give the same level of detail about ChatGPT-User as it does for the other two tokens. The more specific role description comes from secondary documentation, and should be read as such.

The practical distinction is that search visibility and training-related access need not be one decision. OpenAI explicitly says the GPTBot and OAI-SearchBot settings are independent. You may permit OAI-SearchBot while disallowing GPTBot. OpenAI also notes that changes affecting search can take about 24 hours to work through its systems.

That independence does not mean the underlying fetching is always duplicated. As noted earlier, OpenAI says that if both bots are allowed, it may use one crawl’s result for both purposes. The control identities describe the choices exposed to site owners; they do not necessarily map one-to-one onto separate network fetches in every case.

Third-party traffic reports add another layer, but not proof of intent. Measurements may show high request volume, unusual URL selection, or a lack of JavaScript execution. A community report may describe excessive GPTBot requests to a particular endpoint. These observations can reveal an operational problem on a site. They cannot, by themselves, establish that OpenAI documented or intended that behavior, nor can they prove how the returned content was used.

The question is therefore not “What does the OpenAI crawler do?” There are several identities, and the answer depends on which one you mean.

ClaudeBot and Google-Extended show why names and control tokens are not interchangeable

ClaudeBot looks like the ordinary case: a named crawler with a documented purpose and robots.txt token.

Anthropic says ClaudeBot collects public-web content that could contribute to training its generative AI models. Its robots.txt token is ClaudeBot. Anthropic says restricting it signals that future site material should be excluded from its model-training datasets, and documents this full-site rule:

User-agent: ClaudeBot
Disallow: /

Anthropic also publishes separate identities. Claude-SearchBot analyzes online material to improve the relevance and accuracy of search responses. Claude-User may access pages when a person asks Claude a question. Once again, one brand has at least three roles: training, search, and user-directed retrieval.

Anthropic states that its bots honor standard robots.txt directives, try to minimize disruption, consider crawl speed, and do not attempt to bypass CAPTCHAs. It also supports the non-standard Crawl-delay extension for ClaudeBot, giving this example:

User-agent: ClaudeBot
Crawl-delay: 1

The word “non-standard” matters. Anthropic supports this field; Google says it does not. A directive understood by one operator cannot safely be assumed to work for another.

Anthropic also warns that blocking its source IPs is not a reliable or persistent way to express an opt-out, because doing so can stop its bots from reading robots.txt. At the same time, it says an address on its published list indicates Anthropic origin. Those two facts serve different ends: robots.txt communicates a crawler preference, while IP information helps with attribution.

Observed behavior may add detail without replacing the documentation. Vercel’s study reported that Claude-related crawling in its sample fetched JavaScript files but did not execute them. That is a dated measurement across selected sites, not a permanent ClaudeBot specification.

Google-Extended is the revealing counterexample.

It is commonly called a crawler, but the supplied evidence supports a narrower description: Google-Extended is a robots.txt product-control token. Reporting on Google’s documentation says it governs whether site content contributes to generative-AI products including Gemini Apps and Vertex AI generative APIs, while not affecting inclusion or ranking in Google Search. Google’s own crawler documentation lists Google-Extended among tokens that can be targeted in a User-agent line.

What the evidence does not establish is just as important. It does not show that Google-Extended independently sends HTTP requests as an ordinary fetching crawler, and it does not identify which concrete Google client performs every associated fetch. A log entry, product preference, and network requester need not share a one-to-one identity.

This reframes the naming problem. ClaudeBot is documented as a crawler identity that gathers material for a stated role. Google-Extended is documented chiefly as a control identity that governs product use. Both can appear after User-agent: in robots.txt; that does not make them technically interchangeable.

A comparison should prioritize documented purpose over brand recognition

The cleanest additional comparison comes from Mistral because its operator documentation separates the roles explicitly.

That table is more useful than a long directory of names because each row asks the same questions: who operates it, what exact token is established, what purpose is documented, whether the action is automatic or user-triggered, and what control is published.

Other identities can be placed more cautiously. Third-party sources attribute an indexing role to PerplexityBot and a user-action role to Perplexity-User. They identify Bytespider with ByteDance, CCBot with Common Crawl’s open web dataset, Amazonbot with Amazon services, and FacebookBot with Meta-related model or speech-recognition work. But the supplied evidence does not provide equally strong first-party documentation and control details for all of them. Claims such as Bytespider often ignoring robots.txt, or Amazonbot respecting it, are third-party attributions here—not operator-established facts.

The same caution applies to old or poorly attributable names. Claude-Web and Anthropic-ai are described in the supplied secondary evidence as retired legacy training crawlers, so they should not be listed as current Anthropic identities. A Vercel measurement excluded Microsoft Copilot because it lacked a unique user-agent suitable for separate tracking; that is not evidence for inventing a “Copilot crawler” row.

Sometimes the honest purpose category is simply undeclared. Cloudflare uses that label when neither the operator nor other industry sources provide reliable purpose information. It is better than guessing from a company name.

Robots.txt expresses a crawler-specific preference, not a universal technical block

Robots.txt is visible, simple, and easy to overrate.

At its core, the file tells identified crawlers which paths they should or should not fetch. A User-agent field selects the crawler token; Allow and Disallow apply path rules. A Sitemap field points to a fully qualified sitemap URL. But parser behavior is not necessarily identical across operators.

Google documents its own matching rules in detail. Its crawlers choose the most specific matching user-agent group. Crawler-specific groups are not combined with the global * group. For URL paths, Google supports * as a wildcard and $ as an end marker; the most specific path rule wins, and Google uses the least restrictive rule to resolve a conflict. Google supports User-agent, Allow, Disallow, and Sitemap, but not Crawl-delay.

Those are Google’s documented semantics. Anthropic, by contrast, explicitly supports Crawl-delay for ClaudeBot. This is why copying one operator’s example and assuming universal behavior can fail in quiet ways.

Purpose-specific groups let you make narrower choices. OpenAI documents separate GPTBot and OAI-SearchBot controls, so a site can allow search participation while declining the training-related use associated with GPTBot. Anthropic documents ClaudeBot separately from its search and user identities. Google-Extended lets a publisher express a product preference without changing Google Search inclusion or ranking, according to the supplied documentation and reporting.

Changes may not be immediate. OpenAI says search adjustments may take about 24 hours. Google generally caches robots.txt for up to 24 hours and may retain it longer if it cannot refresh the file. An apparent failure shortly after an edit may therefore be stale policy rather than deliberate noncompliance.

But even a perfectly written file is not a lock.

Google says plainly that robots.txt cannot enforce crawler behavior and is not a security mechanism. Different crawlers may interpret syntax differently, and some may ignore it. A disallowed URL may also remain discoverable through links and, in Google Search, may appear without a snippet. Preventing a compliant crawler from fetching a page is not the same as making the URL secret or removing all prior knowledge of it.

Nor does robots.txt necessarily govern every request path. Google excludes some user-controlled and safety-related crawlers from its stated Robots Exclusion Protocol behavior. Spoofed or unidentified traffic may not match your intended group at all. And a user-mediated request may be handled differently from automatic crawling, depending on the operator.

Measurements conflict because they observe different things. Vercel reported robots.txt effectiveness for all crawlers in its measured sample. Cloudflare has reported AI crawling that ignored robots.txt, and Botify attributes frequent noncompliance to Bytespider. The sensible resolution is narrow: robots.txt can work for an identified crawler that chooses to honor it. It is not a universal technical block.

There have been proposals to express distinctions such as storage, indexing, training, and inference more directly, but the supplied IETF material is an expired Internet-Draft—a work in progress, not an adopted standard. For now, much of the web still tries to encode purpose through names.

Layered access control and evidence-based crawler verification

You can allow a crawler, limit it, or block it. These are different operations.

Robots.txt declares a preference to crawlers that identify themselves and honor the file. It is useful for purpose-specific choices and can be maintained directly or through a managed service. It does not enforce the preference at the network layer.

Server, CDN, edge, or WAF rules enforce a response. They can deny, redirect, challenge, or otherwise handle matching requests. This is stronger than robots.txt because the server refuses the request rather than asking the client not to make it. But a rule based only on the user-agent still depends on honest identification, and overly broad rules can catch legitimate users or useful product functions.

Rate limiting reduces frequency without necessarily removing access. An implementation may return HTTP 429 Too Many Requests and a Retry-After header after a threshold is crossed. That can protect resources while preserving some visibility, though distributed clients may spread requests across IPs or identities, and repeated denied requests still cost something.

Authentication, API keys, paywalls, or scoped access policies create a stronger boundary. An unauthenticated crawler ordinarily cannot fetch protected material. The trade-off is obvious: legitimate people, developers, search functions, and open access may be restricted too.

Policies can also be scoped by site area. You might permit an identified AI crawler on public documentation while denying it elsewhere, or apply tighter limits to expensive endpoints. There is no need to turn one concern into a site-wide rule if the actual problem is local.

Each choice changes more than server load. Full blocking may reduce participation in training, indexing, live retrieval, citations, or AI-driven discovery, depending on the crawler’s role. Allowing access may increase bandwidth use and complicate analytics. The evidence does not support a universal estimate of either effect; the trade-off depends on the crawler, product, and site.

Enforcement also raises the identity problem again. User-agent-only rules catch requests that honestly use the expected name. They do not catch a covert scraper pretending to be a browser. You can supplement them with source-IP checks, published network ranges, reverse-and-forward DNS validation, header analysis, and behavioral or network signals. IP ranges can change or be shared, so they require maintenance and still do not form a universal authentication system.

Google’s reverse-then-forward DNS procedure is a documented example, not a protocol for every operator. Some operators publish ranges or other infrastructure information; the supplied evidence does not establish complete coverage or update schedules for each one.

Cryptographic request signatures can provide a stronger identity signal where they are available. The supplied evidence describes ChatGPT Agent—not GPTBot generally—as signing outbound requests with HTTP Message Signatures and a Signature-Agent value, which can be checked against a published key. Broader web-bot authentication work aims to reduce reliance on spoofable headers and changing IP ranges. But it would be a mistake to infer that GPTBot, ClaudeBot, or every major crawler currently signs its requests.

Finally, logs tell you less than they seem to.

Server and CDN logs may record the request time, URL, source IP, user-agent, response status, frequency, redirects, and errors. This can show that a request reached a page, which sections were targeted, and whether your infrastructure returned content. Patterns may help corroborate a classification: in one 2025 Cloudflare cohort, user-action traffic showed daily cycles while training traffic was more erratic. Vercel observed differences in URL choice, JavaScript handling, and error rates in its selected sample.

But behavior is not authentication, and authentication is not purpose.

Even a verified operator IP proves only who operated the request. It does not prove that the response went into training rather than search, retrieval, caching, or another system. Request volume does not prove downstream use. Nor does a crawler request equal a human visit: AI-generated referrals may appear as ordinary pageviews, direct traffic, or unattributed traffic, while server-side fetching and caching can separate the crawler request from any later user interaction.

Traffic comparisons therefore need matched time periods, site populations, purpose categories, and denominators. They also need to account for robots.txt policy, WAF rules, rate limits, errors, rendering, caching, regional infrastructure, and crawlers that lack unique identifiers. Otherwise two neat-looking numbers may be measuring different systems.

The operational boundary is precise. Logs can show requests and responses. Verification can strengthen the identity claim. Operator documentation can state a purpose. None of these, alone or together, gives you a complete view of what happened downstream.

So the last question is not merely whether an AI crawler accessed the page. It is which part of the chain you have actually observed—and which part you are still inferring.