AI crawler controls
You can let AI crawlers in, charge them, or block them. Cloudflare just made charging real, which changes the negotiation between publishers and AI labs in a fundamental way.
Strong signal and real results. Worth committing a pilot to.
llms.txt·Schema
What It Is
AI crawler controls are the technical mechanisms publishers use to govern how AI bots access their content. Robots.txt rules now routinely include directives for OpenAI's GPTBot, Anthropic's ClaudeBot, Google-Extended (the AI-specific Google crawler that's separate from Googlebot), PerplexityBot, and others. Per Cloudflare, AI crawlers now generate up to 80% of all bot traffic, which makes crawler control a meaningful publisher operations decision rather than an obscure SEO setting.
Why It Matters
Cloudflare's Pay-Per-Crawl, in private beta in late 2025 with Q1 2026 public launch expected, introduces a paid-access model. AI crawlers either present payment intent in request headers (HTTP 200 with payment) or receive a 402 Payment Required response with pricing. Publishers get three settings per crawler: Allow (free access), Charge (require payment at the publisher's price, around $0.01-0.05 per request in early estimates), or Block (no access, no payment option).
For AEO buyers, this matters in two directions. As a publisher, you decide whether to be in or out of the answer-engine surface, and at what price. As a brand whose presence depends on being cited, you suddenly care about which sources are in the indexes. If major news publishers go paid-access and AI engines decline to pay, those sources stop feeding answer engines and the citation pool tilts toward whoever stayed open.
Key Developments
- Q1 2026 (expected): Cloudflare Pay-Per-Crawl public launch.
- 2025: Pay-Per-Crawl entered private beta. AI crawler share of bot traffic confirmed at around 80% by Cloudflare.
- 2024: robots.txt rules for AI bots became standard. Google-Extended introduced as separate from Googlebot for AI training opt-out.
What to Watch
Watch which major publishers turn on Pay-Per-Crawl and how AI labs respond. The first round of negotiations will set precedents. Track AI bot traffic share and the response from publishers. As publisher traffic continues to drop (25-38% YoY for news sites is the recent benchmark), the pressure to monetise AI access grows. Watch the IETF and W3C work on standardising AI access controls. Cloudflare's approach is one model. ai.txt and other emerging standards are alternatives.
Strengths
- Publisher-side lever: First credible mechanism for publishers to monetise AI crawler access at scale.
- Standardised on existing infrastructure: Builds on robots.txt and HTTP 402, so adoption doesn't require new client behaviour.
- Cloudflare scale: Cloudflare fronts a large share of the web, which gives Pay-Per-Crawl immediate distribution.
- Three-way choice: Allow, Charge, Block lets publishers match policy to content type and AI lab.
Considerations
- Adoption uncertain: AI labs may decline to pay, depending on the price and the source. Negotiations are unsettled.
- AEO double-edge: Charging AI crawlers may protect revenue but also reduce citation visibility, which is a real AEO cost.
- Implementation complexity: Per-crawler pricing and policy gets complex fast, especially for sites with mixed content.
- Standard fragmentation risk: Cloudflare's Pay-Per-Crawl is one approach. Competing standards may emerge, complicating compliance.
AI crawler controls· Content freshness signals· Entity authority· llms.txt· E-E-A-T signals· Schema.org structured data