What robots.txt controls
robots.txt is the first file a crawler reads. It says which parts of your site each crawler may visit. One wrong line can remove your whole site from search results.
The AI crawlers
ChatGPT, Claude, Perplexity and Gemini each crawl with their own user agent. Blocking them keeps your business out of AI answers, which is where a growing share of buying research happens. Many sites block them by accident with a wildcard rule.
How this checker works
We fetch your robots.txt and evaluate it per crawler with the same parser Navlio uses in its site reports. The verdicts are exactly what each bot would conclude.