# Assess your crawler access

URL: https://www.onsomble.ai/guide/applied/readiness-crawler-access

Published: 2026-09-14

Updated: 2026-09-14

Review crawler access, gather evidence and score this component of your assessment.

Level: Intermediate · Reading time: 4 minutes

## What you'll learn

- Assess crawler access using evidence and the five-point scale.


This component asks whether crawler access supports an accurate, useful account of the business. Complete it for the offerings, locations and customer situations in your assessment scope.

## What to gather

Gather the live robots.txt file and the public paths that matter to your customers. Ask the website provider for relevant hosting and security checks.

## Carry out the review

Use the crawler-permissions exercise to inspect the published rules. Record relevant identities and affected paths. Keep permission-rule findings separate from evidence about hosting or security access. Intended restrictions on private areas can remain.

[Open the related practical guide or exercise](/guide/ai-discoverability-foundations/ai-crawler-access#try-it-yourself).

## Questions to answer

- **CA1** — Has the current `robots.txt` file been reviewed?
- **CA2** — Are rules applying to all crawlers understood?
- **CA3** — Have rules for the relevant search and AI crawler identities been checked?
- **CA4** — Are the homepage and primary navigation pages permitted?
- **CA5** — Are important offering, location, pricing and visitor-information pages permitted?
- **CA6** — Are booking, purchasing and enquiry paths permitted where they contain useful public information?
- **CA7** — Have path-level restrictions been checked for unintended effects?
- **CA8** — Have hosting, content-delivery and security controls been checked for unintended automated-access blocks?

## Choose your scores

Use a whole-number score for each question and record supporting evidence. The examples below show how the levels apply to this component.

| Score | What it looks like |
| --- | --- |
| 1 | Required public pages are confirmed blocked for relevant discovery crawlers, or the requested review activity does not happen. |
| 2 | Only some rules or important paths have been checked, or permissions are inconsistent and fixes occur after problems appear. |
| 3 | Checks cover relevant crawler identities, important paths and applicable security controls. Required public access works and the checks are repeatable. |
| 4 | An owner reviews access after relevant site or hosting changes and at an agreed interval, following up unintended blocks. |
| 5 | Repeated findings improve the access checks and release process, including how the team responds to changes in relevant crawler requirements. |

Start at 3 and check whether the important requirement is implemented and repeatable. Use the lower score if a material part of the higher level is missing. Use U for checks still outstanding and N/A only where the requirement does not apply.

## Record your result

In the [assessment workbook](/tools/ai-perception-assessment?workbook=Readiness), open **Readiness → Crawler access**. For each question, retain the page or report, date checked, finding and next improvement. The workbook calculates the component average and shows evidence coverage.

Highlight questions scored 1 or 2 even when the average is higher. Give outstanding checks an owner. Return to [the information foundations overview](/guide/applied/assess-information-foundations) to continue the other components.


## Related guides

[Return to this learning path](https://www.onsomble.ai/guide/applied)