Can AI crawlers read your site?
Enter a domain to see what its robots.txt says about five AI crawlers. Free, with no sign-up.
The crawlers it checks
Each one is a separate choice. Some serve search, and others collect training data.
- GPTBot
- Collects content to improve OpenAI's foundation models. OpenAI guide
- OAI-SearchBot
- Surfaces websites in ChatGPT's search features. OpenAI guide
- ClaudeBot
- Collects web content that could help train Anthropic's models. Anthropic guide
- PerplexityBot
- Surfaces and links websites in Perplexity's search results. Perplexity guide
- Google-Extended
- A robots.txt token for how Google's AI models can use your content. It doesn't affect Google Search. Google guide
FAQs
How does the checker work?
It reads the robots.txt on the domain you enter and works out what each crawler's rules say about your homepage. It doesn't visit any other page.
What does Not specified mean?
Your robots.txt has no rule that blocks that crawler. Most crawlers treat that as permission.
Does Allowed mean the crawler has visited?
No. It shows what your robots.txt says, not what crawlers actually did. A firewall or CDN rule can also block a crawler that your robots.txt allows.
How fresh is the result?
Ocita keeps a copy of each robots.txt for up to a day, so a recent change may take that long to show.
Do I need an account?
No. There is a limit on how many checks one visitor can run in an hour.
See what AI says about you
The crawler check is one small part. Ocita tracks whether AI actually mentions you.