Back to Making Your Website Accessible to AI: Platform Guides
AI Discoverability Foundations·Beginner·Supporting guide

Custom Sites: AI Crawler Access

Check the relevant Custom Sites settings and give your website provider a specific issue to review.

4 minBeginner
What you'll learn
01

Find the public access settings relevant to Custom Sites

02

Identify a specific issue to discuss with your website provider

03

Retest the published website after a change

Who owns this work

Use this page as an acceptance checklist for the web developer, hosting provider and CDN or firewall owner. A custom site has no single settings screen, so the final production responses are the evidence.

Technical requirements

  1. Serve a valid /robots.txt from every relevant hostname and subdomain. Allow desired search and retrieval agents on important public paths; record the training policy separately.
  2. Return a useful 200 response for intended public pages when requested anonymously.
  3. Do not send noindex in the HTML or an X-Robots-Tag header on pages intended for discovery.
  4. Serve an XML sitemap containing canonical, public, indexable URLs.
  5. Connect important pages with ordinary <a href> links.
  6. Give each meaningful experience, service, location and practical-information view its own URL.
  7. Put core facts in server-rendered or pre-rendered readable content. Do not require JavaScript, a pop-up, cookie choice, location selector or booking interaction before those facts exist.
  8. Return truthful status codes for missing, moved, rate-limited and failed pages.
  9. Review CDN, firewall, rate-limit and bot-management rules separately from robots.txt.
  10. Verify the production site externally and inspect server or CDN logs for blocked desired agents.

What healthy looks like

  • The intended crawler can read robots.txt, request the page and receive its meaningful content.
  • The page is not marked noindex and its canonical URL appears in the sitemap.
  • Important pages are connected by crawlable links.
  • Network security allows the same access policy as the application.

Let crawlers in while keeping security in place

Ask your developer to use the crawler provider's verification or IP-range guidance to identify its requests. Adjust the rule for that crawler while keeping the site's other firewall protections active.

Message for the development team

Please make [PAGE URLS] accessible for [DESIRED SEARCH OR RETRIEVAL AGENTS] while retaining our separate policy for [TRAINING AGENTS]. The current assessment found [EXACT ROBOTS RULE, NOINDEX/X-ROBOTS-TAG, HTTP STATUS, CAPTCHA OR RENDERING ISSUE]. Please identify the owning layer, implement the smallest production change, verify robots and indexing directives, confirm the sitemap and crawlable links, test an anonymous fetch from outside our authenticated network, and provide the before-and-after evidence.

Primary references

Return to all platform guides.