This component asks whether crawler access supports an accurate, useful account of the business. Complete it for the offerings, locations and customer situations in your assessment scope.
What to gather
Gather the live robots.txt file and the public paths that matter to your customers. Ask the website provider for relevant hosting and security checks.
Carry out the review
Use the crawler-permissions exercise to inspect the published rules. Record relevant identities and affected paths. Keep permission-rule findings separate from evidence about hosting or security access. Intended restrictions on private areas can remain.
Open the related practical guide or exercise.
Questions to answer
- CA1 — Has the current
robots.txtfile been reviewed? - CA2 — Are rules applying to all crawlers understood?
- CA3 — Have rules for the relevant search and AI crawler identities been checked?
- CA4 — Are the homepage and primary navigation pages permitted?
- CA5 — Are important offering, location, pricing and visitor-information pages permitted?
- CA6 — Are booking, purchasing and enquiry paths permitted where they contain useful public information?
- CA7 — Have path-level restrictions been checked for unintended effects?
- CA8 — Have hosting, content-delivery and security controls been checked for unintended automated-access blocks?
Choose your scores
Use a whole-number score for each question and record supporting evidence. The examples below show how the levels apply to this component.
| Score | What it looks like |
|---|---|
| 1 | Required public pages are confirmed blocked for relevant discovery crawlers, or the requested review activity does not happen. |
| 2 | Only some rules or important paths have been checked, or permissions are inconsistent and fixes occur after problems appear. |
| 3 | Checks cover relevant crawler identities, important paths and applicable security controls. Required public access works and the checks are repeatable. |
| 4 | An owner reviews access after relevant site or hosting changes and at an agreed interval, following up unintended blocks. |
| 5 | Repeated findings improve the access checks and release process, including how the team responds to changes in relevant crawler requirements. |
Start at 3 and check whether the important requirement is implemented and repeatable. Use the lower score if a material part of the higher level is missing. Use U for checks still outstanding and N/A only where the requirement does not apply.
Record your result
In the assessment workbook, open Readiness → Crawler access. For each question, retain the page or report, date checked, finding and next improvement. The workbook calculates the component average and shows evidence coverage.
Highlight questions scored 1 or 2 even when the average is higher. Give outstanding checks an owner. Return to the information foundations overview to continue the other components.