# Self-Hosted WordPress: AI Crawler Access

URL: https://www.onsomble.ai/guide/ai-discoverability-foundations/wordpress

Published: 2026-09-14

Updated: 2026-09-14

Check the relevant Self-Hosted WordPress settings and give your website provider a specific issue to review.

Level: Beginner · Reading time: 4 minutes

## What you'll learn

- Find the public access settings relevant to Self-Hosted WordPress
- Identify a specific issue to discuss with your website provider
- Retest the published website after a change


## What you can check yourself

1. Under **Settings → Reading**, ensure **Discourage search engines from indexing this site** is unchecked on production.
2. Open representative pages while logged out. Confirm they return the actual page rather than maintenance mode, a security challenge or a login.
3. In the active SEO plugin, check important pages and page types for unintended `noindex` settings.
4. Inspect the final live `/robots.txt`. A physical file, SEO plugin, security plugin, host or CDN may alter it.
5. Check `/wp-sitemap.xml`, `/sitemap.xml` or the sitemap endpoint declared by the active SEO plugin. Confirm important pages are included.
6. Confirm essential information appears in readable page content rather than only in a slider, pop-up, image or third-party booking widget.
7. Ask the host or security provider to review `403`, `429`, CAPTCHA and bot-management behaviour if desired crawlers cannot fetch the site.
8. Clear relevant caches, publish the change and rerun the assessment.

## What healthy looks like

- Production search discouragement is off.
- One clearly identified system owns page indexing, sitemap and crawler rules.
- Desired crawlers are not blocked by the final `robots.txt`, host, CDN or security plugin.
- Important pages are public, readable and listed in the authoritative sitemap.

## Making a change

Ask your provider which plugin or hosting setting controls the rule, and make the change there. Keep one system responsible for each setting to avoid conflicting rules. Back up the configuration before editing it.

## Message for the WordPress provider

> Our crawler-access assessment found **[EXACT RULE, NOINDEX TAG, STATUS OR CHALLENGE]** on **[PAGE URL]** for **[CRAWLER NAME]**. We want this public page available for current search and AI-assisted retrieval. Please identify whether WordPress core, the active SEO/security plugin, the host, CDN or firewall owns the behaviour; make the smallest supported change; clear caches; and confirm the final live response.

## Official WordPress references

- [WordPress.org — Settings Reading screen](https://wordpress.org/documentation/article/settings-reading-screen/)
- [WordPress core — XML sitemaps](https://make.wordpress.org/core/2020/07/22/new-xml-sitemaps-functionality-in-wordpress-5-5/)
- [WordPress developer resources — wp_robots hook](https://developer.wordpress.org/reference/hooks/wp_robots/)


[Return to all platform guides](/guide/ai-discoverability-foundations/website-platform-guides).


## Related guides

[Return to this learning path](https://www.onsomble.ai/guide/ai-discoverability-foundations)