AI visibility, answer engines, and the future of brand discovery.

Googlebot is Google Search’s documented crawler, while Google-Extended is a robots.txt control for specified AI training and grounding uses of crawled content, including documented Gemini-related applications.

GPTBot may support training-related use, OAI-SearchBot supports ChatGPT search, and ChatGPT-User retrieves pages after user requests. Robots.txt controls are documented for the first two, while ChatGPT-User policy remains unresolved.

AI crawlers are automated clients whose roles vary across training, search indexing, user-triggered retrieval, and product control; user-agent strings, robots.txt, logs, and IP checks provide useful but limited evidence.

Most robots.txt guides for AI crawlers are written for publishers who want to block the bots. This is the opposite: how to check if your site is accidentally invisible to ChatGPT, Claude and Perplexity — and the exact lines to paste to fix it.