Is GPTBot blocked on your site?
Enter a domain to see whether its robots.txt lets GPTBot, OpenAI's training crawler, or any of 14 other AI crawlers read the homepage.
Ask about another crawler:
- OAI-SearchBot
- ChatGPT-User
- GPTBot
- Claude-SearchBot
- Claude-User
- ClaudeBot
- PerplexityBot
- Perplexity-User
- Google-Extended
- Applebot-Extended
- Amazonbot
- DuckAssistBot
- Meta-ExternalAgent
- CCBot
- Bytespider
About GPTBot
Crawls content that may be used to train OpenAI's generative AI foundation models. OpenAI says disallowing it indicates a site's content should not be used in that training. Read the operator's documentation.
How to block or allow it
robots.txt rules are read per crawler: GPTBot follows a group that names it, or else the User-agent: * group. To block it everywhere:
User-agent: GPTBot
Disallow: /The AI crawler robots.txt generator makes these lines for any mix of crawlers. The check reads only your homepage path, robots.txt, llms.txt and a HEAD of llms-full.txt; see exactly what is fetched.