AI crawler robots.txt checker
The AI crawler robots.txt checker is a free tool that reads your site's robots.txt and shows, for each of 17 search and AI crawlers, whether it may fetch your home page (or any page you enter) and which rule decides.
One request to your /robots.txt. Nothing is stored.
Allowed isn't the same as visible
robots.txt is only permission. The free scan requests your pages as each AI crawler and shows what actually arrives.
Questions
What does this checker test?
It reads your robots.txt once and applies the standard matching rules (RFC 9309) for 17 crawler names: the group that names the crawler is used, otherwise the * group, and the longest matching rule wins.
If robots.txt allows a crawler, can AI see my pages?
Not necessarily. robots.txt only grants permission. A firewall or bot protection can still refuse the request, and a page that is built in the browser can reach a crawler nearly empty. The free scan requests your pages as each crawler to show what actually arrives.
What happens if I don't have a robots.txt?
If /robots.txt returns a 404, crawlers treat the whole site as allowed. If it returns a server error or your app's HTML page instead of a plain text file, crawlers may handle it differently, so the checker marks it as unreadable.
Should I block training crawlers?
That is your call. Several AI companies run separate agents for training, for search and for fetching a page when a user asks, so blocking a training crawler does not have to block the others. The table shows each one separately.