Is Applebot-Extended blocked on your site?
Enter a domain to see whether its robots.txt lets GPTBot, OpenAI's training crawler, or any of 14 other AI crawlers read the homepage.
Ask about another crawler:
- OAI-SearchBot
- ChatGPT-User
- GPTBot
- Claude-SearchBot
- Claude-User
- ClaudeBot
- PerplexityBot
- Perplexity-User
- Google-Extended
- Applebot-Extended
- Amazonbot
- DuckAssistBot
- Meta-ExternalAgent
- CCBot
- Bytespider
About Applebot-Extended
Does not crawl: it controls whether pages Applebot crawled may be used to train Apple's foundation models. Apple says pages that disallow it can still appear in search results. Read the operator's documentation.
How to block or allow it
robots.txt rules are read per crawler: Applebot-Extended follows a group that names it, or else the User-agent: * group. To block it everywhere:
User-agent: Applebot-Extended
Disallow: /The AI crawler robots.txt generator makes these lines for any mix of crawlers. The check reads only your homepage path, robots.txt, llms.txt and a HEAD of llms-full.txt; see exactly what is fetched.