ClaudeBot vs Claude-SearchBot: How to Control Anthropic Crawlers

The names are similar enough to invite a bad shortcut. You see an Anthropic request in a log, think “Claude crawler,” and write one rule for it.

But there is no single control hiding behind these three names. Anthropic describes ClaudeBot, Claude-SearchBot, and Claude-User as separate identities for separate jobs. One concerns content that could contribute to model training, one concerns search indexing and answer quality, and one concerns retrieval prompted by an individual user. The right question is not simply whether to allow Anthropic. It is what kind of access you want to allow.

The three identities: one Anthropic taxonomy, three purposes

Anthropic’s first-party crawler documentation says it uses different robots to gather public-web data for model development, search the web, and retrieve content at users’ direction. The three names are request or robot identities within that taxonomy, not three versions of Claude and not three separate products.

ClaudeBot is the public-web collection robot. Anthropic says it collects web content that could potentially contribute to the training of its generative AI models. “Could potentially” matters: a fetch by ClaudeBot does not prove that the page entered a training dataset, much less that it affected a model. It identifies the documented collection purpose, not every downstream use of the fetched page.

Claude-SearchBot navigates and analyzes web content to improve the relevance and accuracy of search responses. Anthropic also says that disabling it prevents the company from indexing the site for search optimization. That establishes a search and indexing role, but not the machinery behind it. The documentation does not disclose the index architecture, ranking system, refresh schedule, or whether a given search causes a fresh crawl.

Claude-User supports retrieval initiated by a Claude user. Anthropic says that when an individual asks Claude a question, the system may access websites using the Claude-User agent. This is narrower than saying every question triggers a request. It is also broader than saying every request fetches exactly one page once and then disappears; Anthropic’s documentation does not establish those operational details.

So the useful comparison is:

There has apparently been a change in how this taxonomy is presented. Secondary reporting from Search Engine Journal says an older crawler page referred more broadly to ClaudeBot and that the older Claude-Web and Anthropic-AI identifiers are deprecated. But the supplied evidence contains no first-party version history or precise change date, so this historical account should not be treated as an authoritative timeline.

The current first-party definitions are the firmer ground. They make the policy split plain: training collection, search indexing, and user-directed retrieval are different workflows.

How the request contexts differ in practice

ClaudeBot has the most familiar crawler-shaped purpose. It gathers material from the public web that may be useful in model development. Yet Anthropic does not document its crawl frequency, scheduling rules, or exact route from collection to a training dataset. In a log, the ClaudeBot identity tells you the stated reason for the request; it does not let you reconstruct what happens to the content later.

Claude-SearchBot belongs to a different stage of the system. Anthropic says it navigates and analyzes web content to improve search responses, and that blocking it prevents indexing for search optimization. This suggests a body of search-oriented work rather than model-training collection—but “search-oriented” is about purpose, not a complete technical specification.

There is an important trap here. Anthropic’s separate web-search API documentation describes a workflow in which Claude decides whether a prompt calls for web search, the API runs one or more searches, and Claude produces an answer with citations. Anthropic’s announcement of the web-search API adds that Claude can generate targeted queries, analyze results, perform progressive searches, and refine later searches based on earlier ones. Developers can cap searches with a max_uses parameter.

None of this proves that every HTTP request behind that workflow uses Claude-SearchBot.

The web-search documentation explains answer generation. The crawler documentation explains Claude-SearchBot’s purpose. They plainly concern related search work, but the supplied evidence does not map every search, index lookup, result fetch, or page retrieval to a particular agent. Treating the two documents as a complete network diagram would go beyond what they say.

Claude-User has another context again. Its documented trigger is an individual asking Claude a question, after which Claude may access a site. This makes it a user-initiated retrieval mechanism. It does not make it ordinary human traffic—the requester is still an automated system—and it does not establish that each visit is transient, limited to one URL, or tied neatly to one visible question.

What, then, can you infer from a request? Mainly its claimed workflow:

• ClaudeBot claims the training-collection context.
• Claude-SearchBot claims the search-quality and indexing context.
• Claude-User claims the user-directed retrieval context.

Those are first-party purpose definitions. Any stronger description of request shape, retention, crawl depth, or scheduling is an operational inference unless Anthropic documents it elsewhere.

Anthropic also states several operating principles. Its bots honor standard robots.txt directives, support the non-standard Crawl-delay directive, do not try to bypass CAPTCHAs, and aim to minimize disruption by considering how quickly they crawl the same domains. These are documented policies, not independent measurements. They do not establish that every site will see low traffic, nor do they show which of the three identities sends the most requests.

What identifies an Anthropic request—and what does not

The three names—ClaudeBot, Claude-SearchBot, and Claude-User—are the available classification tokens for log review and crawler policy. Anthropic’s official example explicitly uses ClaudeBot in a User-agent field.

But the supplied first-party passages do not provide complete literal HTTP User-Agent header strings for all three agents. They do not establish version syntax, capitalization variants, or a full agent-specific log signature. You should not build a brittle parser around details the documentation here does not supply.

More fundamentally, a User-Agent value is a claim, not authentication. As general bot-identification guidance from Arcjet notes, the field is an arbitrary string and is easy to spoof. A request calling itself ClaudeBot may be useful for initial classification, but that name alone does not prove Anthropic sent it.

A cautious log review should therefore inspect the request in context:

• the claimed user-agent;
• the source IP address;
• DNS identity checks where they are applicable;
• the requested URL;
• the response status;
• request timing and frequency.

The last three fields do not authenticate the sender. They help you understand what the traffic is doing: which resources it asks for, whether your controls return the expected response, and whether the pattern is causing an operational problem.

Source verification is less tidy. Anthropic says that a source IP appearing on its crawler list indicates that the crawler comes from Anthropic. But the supplied evidence does not include a complete current list or a definitive, agent-specific verification protocol. Secondary sources also disagree about whether a usable published range is currently available.

General bot-verification practice may use reverse DNS followed by a forward lookup to confirm that the resulting hostname maps back to the original IP, or compare an address with an operator’s published list. That is useful technical guidance, but it is not an Anthropic-specific authentication procedure in the evidence here. There are no documented cryptographic signatures or agent-specific authentication headers in the supplied material.

So you can classify, gather supporting signals, and spot suspicious behavior; you cannot turn the three names into proof of origin by string matching alone.

Nor can logs from one site settle which agent is generally “heavier.” Anthropic says it aims to minimize disruption and respects Crawl-delay where appropriate, but no authoritative comparative data here shows that ClaudeBot, Claude-SearchBot, or Claude-User has a higher request volume, crawl frequency, or server-load profile. Your logs describe your site, not the whole system.

Separate controls in robots.txt

The taxonomy becomes useful when you write policy. If the three agents do different things, the controls need to name the thing being controlled.

Anthropic says its bots honor standard robots.txt directives and support Crawl-delay, though Crawl-delay is not part of the standard Robots Exclusion Protocol. Its official example shows the following pattern for ClaudeBot:

User-agent: ClaudeBot
Crawl-delay: 1

This limits activity; it does not block ClaudeBot. Anthropic says it respects the directive “where appropriate,” so the example should be read as a documented request-rate control, not a hard real-time rate-limit guarantee.

To block a named bot across a site, Anthropic documents the familiar pattern:

User-agent: ClaudeBot
Disallow: /

Disallow: / is the opt-out signal. Crawl-delay is the non-blocking throttle. They answer different questions: “May this agent crawl?” and “How quickly should it crawl?”

The reveal is that there is no universal “Claude” decision here. If you want a different policy for each purpose, each identity needs its own group. For example, the control logic for blocking all three is:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

Anthropic’s supplied first-party example explicitly prints the ClaudeBot group, while the other two names come from its documented three-agent taxonomy. Secondary operational guidance states directly that each token needs its own directive and that blocking ClaudeBot does not block Claude-SearchBot or Claude-User.

The same separation lets you choose rather than merely block everything. You might block ClaudeBot while allowing the two retrieval and search identities, or allow ClaudeBot while blocking user-directed retrieval. You could also assign different Crawl-delay values to named groups. The evidence establishes the ability to manage the identities separately; it does not prescribe which mix is right for a given site.

Subdomains are another easy place to make a false assumption. Anthropic says to add the blocking rule to the robots.txt file for every subdomain from which you want to opt out. A rule on the main host should not be treated as a documented opt-out for all its subdomains. If both www.example.com and docs.example.com are in scope, the relevant robots.txt file on each must carry the intended agent-specific policy.

For crawler opt-out, Anthropic recommends robots.txt rather than IP-address blocking. Its explanation is practical: blocking the crawler’s IP may stop it from reading the policy file, and IP blocking may not correctly or persistently guarantee an opt-out. This does not mean network controls have no security use. It means the supplied evidence does not establish them as a reliable replacement for Anthropic’s documented crawler opt-out route.

If traffic appears to ignore policy or malfunction, Anthropic directs site operators to email [email protected] from an address that includes the affected domain. That is a support route, not a complete incident-response runbook; you would still need your own logs and operational process.

What allowing or blocking each agent changes—and what remains unknown

Blocking ClaudeBot changes the training-collection policy. Anthropic says the restriction signals that the site’s future materials should be excluded from its model-training datasets. The wording is careful. It does not say that previously collected content is deleted, and the supplied evidence does not establish what happens to such material.

Blocking Claude-SearchBot changes the search-indexing policy. Anthropic says it prevents the system from indexing the site for search optimization and may reduce the site’s visibility and accuracy in user search results. There is no documented number attached to that effect. It does not prove a ranking loss, traffic loss, or any broader business outcome.

Blocking Claude-User changes the user-retrieval policy. Anthropic says it prevents the system from retrieving the site’s content in response to a user query and may reduce visibility for user-directed web search. Again, “may” is doing real work. The evidence does not quantify the effect or connect it to referrals, revenue, or ranking.

Put differently, the controls divide like this:

These are first-party policy effects, not guarantees about every downstream system or an immediate global data state. Blocking one identity does not establish that the other two are blocked. Nor does a rule for these three identities govern ordinary human access or prove that every other Anthropic mechanism is covered.

Several boundaries remain unknown in the supplied evidence. It does not say how the rules affect cached copies, third-party indexes, previously collected content, or mechanisms outside these three documented identities. It does not establish what happens when unrelated or spoofed traffic claims one of the names, and of course a request that does not honor robots.txt cannot be controlled merely by trusting the label it sends.

This is why “allow Anthropic or block Anthropic?” is the wrong level of abstraction. The real choice has three parts: may future material be collected through ClaudeBot for potential training use; may Claude-SearchBot index the site for search work; and may Claude-User retrieve pages when a person asks Claude for them? Once you ask those separately, the robots.txt file can finally express the policy you actually mean.