# Are You Letting AI Crawlers In? URL: https://www.onsomble.ai/guide/ai-discoverability-foundations/ai-crawler-access Published: 2026-09-14 Updated: 2026-09-14 Read your website's robots.txt instructions and check whether the main search and AI discovery crawlers are allowed in. Level: Beginner · Reading time: 6 minutes ## What you'll learn - Understand User-agent, Allow and Disallow in plain language - Recognise the difference between a whole-site block and a selective rule - Run a short robots.txt assessment and share findings with your provider Your website appears in Google. Does that mean every AI discovery system is allowed to crawl it? Different services can have different instructions, so it is worth checking. ## The instructions at your website's front door A **crawler** is an automated visitor that finds and reads webpages. Your **robots.txt** file tells these visitors which parts of the website they may visit. Open a browser and put `/robots.txt` after your main website address. You might see a short text file with lines like these. ### Door closed ```text User-agent: * Disallow: / ``` `User-agent: *` means these are the default instructions for crawlers. `Disallow: /` tells them to stay out of every page on the website. ### Door open ```text User-agent: * Allow: / ``` Here, crawlers are being told they may visit pages across the website. The `*` group is the default for crawlers without their own named rules. If your file includes a named group, that crawler follows its matching group instead. This is why the exercise checks the complete file. ## Some rules apply only to particular visitors or paths A site can allow general crawling but give a particular crawler different instructions. It can also block a section, such as `/members/`, while allowing public experience pages. Search and training controls can be separate. For example, OpenAI uses `OAI-SearchBot` for search and `GPTBot` for content that may be used in training. The exercise focuses on search and discovery crawlers. [OpenAI crawler documentation](https://developers.openai.com/api/docs/bots). Look for blocks affecting the public pages you want people to discover. Give your provider the exact rule and page address so they can make a targeted change. ## What the exercise tells you Use the form below to fetch your robots.txt. You can check just your homepage or add specific offering and booking pages. Copy the completed prompt into your AI assistant. It will produce a table for Google, Bing, ChatGPT, Claude, Perplexity, Meta, Apple and Amazon discovery crawlers. Each result is **Allowed**, **Blocked** or **Unclear**, with the rule responsible. If a change or review is needed, it will produce a short message for your website provider. “Allowed” means your robots.txt gives that crawler permission to visit the page. If the file is missing or empty, the form adds that result to your prompt. You can also open the file yourself and paste its text into the form. ## Put the result to work If the assessment finds a block on a public page you want discovered, send the generated message to your provider. Ask them whether the rule is intentional and, if not, to correct it. Fetch the updated file and rerun the exercise after the change is published. ### Find the steps for your website - [Wix](/guide/ai-discoverability-foundations/wix) - [Squarespace](/guide/ai-discoverability-foundations/squarespace) - [WordPress.com](/guide/ai-discoverability-foundations/wordpress-com) - [Self-hosted WordPress](/guide/ai-discoverability-foundations/wordpress) - [Custom sites](/guide/ai-discoverability-foundations/custom-sites) The next article, [Website Platform Guides](/guide/ai-discoverability-foundations/website-platform-guides), explains how to work with your provider on the changes. Reference: [Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html). ## Check your crawler permissions Enter your website address and click Fetch robots.txt. Add any specific pages you want to check, then copy your completed prompt into your AI assistant. Enter your website address and fetch its robots.txt. You can also add specific pages to check. Then copy your completed prompt into your AI assistant. - Your website address (required) — e.g. yourbusiness.co.nz - Main experience, product or service page (optional) — e.g. yourbusiness.co.nz/experiences - Booking, enquiry or information page (optional) — e.g. yourbusiness.co.nz/contact - Your complete robots.txt text (required) — Paste the text from your robots.txt file here ```text Read the robots.txt instructions below and explain whether major search and AI discovery crawlers are allowed to access my three important public pages. You do not need to browse my website. This is a robots.txt permission check only. Website: [PASTE WEBSITE ADDRESS] Pages to check: 1. [PASTE HOMEPAGE ADDRESS] 2. [PASTE IMPORTANT EXPERIENCE, PRODUCT OR SERVICE PAGE] 3. [PASTE BOOKING, ENQUIRY OR PRACTICAL INFORMATION PAGE] Here is the complete content of my robots.txt file: --- START OF ROBOTS.TXT --- [PASTE ROBOTS.TXT CONTENT HERE] --- END OF ROBOTS.TXT --- Check these crawlers: - Googlebot — Google Search and Google's AI search features - bingbot — Bing Search and Microsoft Copilot - OAI-SearchBot — ChatGPT search - Claude-SearchBot — Claude search - PerplexityBot — Perplexity search - Meta-WebIndexer — Meta AI search - Applebot — Apple search experiences - Amzn-SearchBot — Amazon and Alexa search experiences For each crawler: 1. Look for instructions written specifically for that crawler. 2. If there are no crawler-specific instructions, apply the rules under User-agent: *. 3. Check the path of each of my three pages against the relevant Allow and Disallow rules. 4. Apply the most specific matching path rule. If equally specific Allow and Disallow rules conflict, treat Allow as taking precedence. 5. If multiple groups match the same crawler, combine their rules. If neither a named group nor a wildcard group applies, there is no robots.txt restriction. 6. Do not treat the absence of a crawler's name as a block if the wildcard rules allow access. Provider exceptions: if Applebot is not named but Googlebot is, apply Google's rules to Applebot. If Amzn-SearchBot is not named, Amazon says it can follow other search bots' rules; when those rules differ, mark its result Unclear rather than guessing which one it will use. If I supplied NO ROBOTS.TXT FILE FOUND or an error message rather than the file, report TECHNICAL CHECK NEEDED and ask the provider to confirm the response. Do not invent file contents. For pages on another hostname, report Unclear because this file does not govern that hostname. Use these result labels: - ALLOWED — robots.txt does not block this crawler from the page. - BLOCKED — robots.txt contains a rule that blocks this crawler from the page. - UNCLEAR — the supplied file is incomplete, conflicting or cannot be interpreted reliably. Use this exact output structure: 1. OVERALL RESULT Choose one: - ALLOWED BY ROBOTS.TXT — all eight crawlers are allowed to access all three page paths. - BLOCKED BY ROBOTS.TXT — at least one crawler is blocked from at least one page path. - TECHNICAL CHECK NEEDED — the result cannot be determined from the supplied information. Explain the result in no more than three plain-English sentences. 2. CRAWLER RESULTS Create one table with these columns: Search or AI product | Crawler | Homepage | Main offering page | Booking or information page | Rule applied Use only Allowed, Blocked or Unclear in the three page columns. Quote the exact robots.txt instruction in the final column, or say No matching restriction if no rule applies. 3. WHAT TO DO NEXT Give me no more than three priority actions. If nothing is blocked, say that no robots.txt change is required based on these pages. 4. MESSAGE FOR MY WEBSITE PROVIDER Only if something is blocked or unclear, write a short message I can copy and send to my website provider. Name the affected crawler, page path and robots.txt rule, and ask them to review it. Write for a non-technical business owner. Keep the entire answer concise. ``` [Download worksheet (PDF)](https://www.onsomble.ai/downloads/guides/crawler-access.pdf) [Download worksheet (Markdown)](https://www.onsomble.ai/downloads/guides/crawler-access.md) ## Key takeaways - robots.txt tells crawlers which paths they may visit. - Being found in Google does not establish the permissions of every AI crawler. - Check the actual file and the exact rules before requesting a change. ## Related resources - [Website Platform Guides](https://www.onsomble.ai/guide/ai-discoverability-foundations/website-platform-guides) - [OpenAI crawler documentation](https://developers.openai.com/api/docs/bots) ## Related guides - [Wix: AI Crawler Access](https://www.onsomble.ai/guide/ai-discoverability-foundations/wix) - [Squarespace: AI Crawler Access](https://www.onsomble.ai/guide/ai-discoverability-foundations/squarespace) - [WordPress.com: AI Crawler Access](https://www.onsomble.ai/guide/ai-discoverability-foundations/wordpress-com) - [Self-Hosted WordPress: AI Crawler Access](https://www.onsomble.ai/guide/ai-discoverability-foundations/wordpress) - [Custom Sites: AI Crawler Access](https://www.onsomble.ai/guide/ai-discoverability-foundations/custom-sites) [Return to this learning path](https://www.onsomble.ai/guide/ai-discoverability-foundations)