All posts Crawlers

How to check if AI crawlers can read your site

If an engine can't read a page, it can't cite it. Three places decide whether yours is readable — and all three are things you can check today.

On this page

An AI engine can only cite a page it can fetch and read. Before you worry about what a page says, it’s worth making sure the engine can get to it at all. Three places decide this, and none of them need a developer.

robots.txt

This file sits at the root of your site, at /robots.txt. It lists crawlers by name and says where each may go. A broad Disallow: / under an AI crawler’s name keeps it out of the whole site. Read it with your own eyes — a plugin or a host default may have added a rule you never chose.

Your CDN or firewall

Cloudflare, Sucuri and others can block bots before a request ever reaches your server. This happens above robots.txt, so your file can look open while the door is shut. If a page loads fine for you but an engine never cites it, the edge is the next place to look.

Pages that need JavaScript

If the words on a page only appear after a script runs, a crawler that reads raw HTML may see an empty shell. View the page source — not the inspector — and check the real text is there.

A check that asks the engines your own questions will show you which crawlers reached you and which didn’t. That’s a more honest answer than guessing from the settings alone.

Run a free check on your site No signup. About 30 seconds. One private report.
Run a free check