AI Discoverability Foundations·Beginner·Module 3 of 4

Are You Letting AI Crawlers In?

Read your website's robots.txt instructions and check whether the main search and AI discovery crawlers are allowed in.

6 minBeginner

What you'll learn

Understand User-agent, Allow and Disallow in plain language

Recognise the difference between a whole-site block and a selective rule

Run a short robots.txt assessment and share findings with your provider

The instructions at your website's front door

A crawler is an automated visitor that finds and reads webpages. Your robots.txt file tells these visitors which parts of the website they may visit.

Open a browser and put /robots.txt after your main website address. You might see a short text file with lines like these.

Door closed

User-agent: *
Disallow: /

User-agent: * means these are the default instructions for crawlers. Disallow: / tells them to stay out of every page on the website.

Door open

User-agent: *
Allow: /

Here, crawlers are being told they may visit pages across the website.

The * group is the default for crawlers without their own named rules. If your file includes a named group, that crawler follows its matching group instead. This is why the exercise checks the complete file.

Some rules apply only to particular visitors or paths

A site can allow general crawling but give a particular crawler different instructions. It can also block a section, such as /members/, while allowing public experience pages.

Search and training controls can be separate. For example, OpenAI uses OAI-SearchBot for search and GPTBot for content that may be used in training. The exercise focuses on search and discovery crawlers. OpenAI crawler documentation.

Look for blocks affecting the public pages you want people to discover. Give your provider the exact rule and page address so they can make a targeted change.

What the exercise tells you

Use the form below to fetch your robots.txt. You can check just your homepage or add specific offering and booking pages. Copy the completed prompt into your AI assistant. It will produce a table for Google, Bing, ChatGPT, Claude, Perplexity, Meta, Apple and Amazon discovery crawlers.

Each result is Allowed, Blocked or Unclear, with the rule responsible. If a change or review is needed, it will produce a short message for your website provider.

“Allowed” means your robots.txt gives that crawler permission to visit the page.

If the file is missing or empty, the form adds that result to your prompt. You can also open the file yourself and paste its text into the form.

Put the result to work

If the assessment finds a block on a public page you want discovered, send the generated message to your provider. Ask them whether the rule is intentional and, if not, to correct it. Fetch the updated file and rerun the exercise after the change is published.

Find the steps for your website

The next article, Website Platform Guides, explains how to work with your provider on the changes.

Reference: Robots Exclusion Protocol.

Key takeaways

robots.txt tells crawlers which paths they may visit.

Being found in Google does not establish the permissions of every AI crawler.

Check the actual file and the exact rules before requesting a change.

Try it yourself

Check your crawler permissions

Enter your website address and fetch its robots.txt. You can also add specific pages to check. Then copy your completed prompt into your AI assistant.

Your website address

Main experience, product or service page(optional)

Booking, enquiry or information page(optional)

Your complete robots.txt text

Fetch your robots.txt using the button above, or paste it here.

Your details are added to the prompt automatically.

Preview your prompt

Read the robots.txt instructions below and explain whether major search and AI discovery crawlers are allowed to access the public pages listed below.

You do not need to browse my website. This is a robots.txt permission check only.

Website: [PASTE WEBSITE ADDRESS]

Pages to check:
1. [PASTE HOMEPAGE ADDRESS]

Here is the complete content of my robots.txt file:

--- START OF ROBOTS.TXT ---
[PASTE ROBOTS.TXT CONTENT HERE]
--- END OF ROBOTS.TXT ---

Check these crawlers:

- Googlebot — Google Search and Google's AI search features
- bingbot — Bing Search and Microsoft Copilot
- OAI-SearchBot — ChatGPT search
- Claude-SearchBot — Claude search
- PerplexityBot — Perplexity search
- Meta-WebIndexer — Meta AI search
- Applebot — Apple search experiences
- Amzn-SearchBot — Amazon and Alexa search experiences

For each crawler:

1. Look for instructions written specifically for that crawler.
2. If there are no crawler-specific instructions, apply the rules under User-agent: *.
3. Check the path of each listed page against the relevant Allow and Disallow rules.
4. Apply the most specific matching path rule. If equally specific Allow and Disallow rules conflict, treat Allow as taking precedence.
5. If multiple groups match the same crawler, combine their rules. If neither a named group nor a wildcard group applies, there is no robots.txt restriction.
6. Do not treat the absence of a crawler's name as a block if the wildcard rules allow access.

Provider exceptions: if Applebot is not named but Googlebot is, apply Google's rules to Applebot. If Amzn-SearchBot is not named, Amazon says it can follow other search bots' rules; when those rules differ, mark its result Unclear rather than guessing which one it will use.

If I supplied NO ROBOTS.TXT FILE FOUND or an error message rather than the file, report TECHNICAL CHECK NEEDED and ask the provider to confirm the response. Do not invent file contents.
For pages on another hostname, report Unclear because this file does not govern that hostname.

Use these result labels:

- ALLOWED — robots.txt does not block this crawler from the page.
- BLOCKED — robots.txt contains a rule that blocks this crawler from the page.
- UNCLEAR — the supplied file is incomplete, conflicting or cannot be interpreted reliably.

Use this exact output structure:

1. OVERALL RESULT

Choose one:

- ALLOWED BY ROBOTS.TXT — all eight crawlers are allowed to access all listed page paths.
- BLOCKED BY ROBOTS.TXT — at least one crawler is blocked from at least one page path.
- TECHNICAL CHECK NEEDED — the result cannot be determined from the supplied information.

Explain the result in no more than three plain-English sentences.

2. CRAWLER RESULTS

Create one table with these columns:

Search or AI product | Crawler | Homepage | Rule applied

Use only Allowed, Blocked or Unclear in the page columns. Quote the exact robots.txt instruction in the final column, or say No matching restriction if no rule applies.

3. WHAT TO DO NEXT

Give me no more than three priority actions. If nothing is blocked, say that no robots.txt change is required based on these pages.

4. MESSAGE FOR MY WEBSITE PROVIDER

Only if something is blocked or unclear, write a short message I can copy and send to my website provider. Name the affected crawler, page path and robots.txt rule, and ask them to review it.

Write for a non-technical business owner. Keep the entire answer concise.

Prefer a worksheet?Download worksheet (PDF)Markdown version