pigfox.ai

AI crawler robots.txt generator

What each AI crawler is for, in its operator's own words, and the robots.txt lines that allow or block it.

Search crawlers put your pages in AI search results. User fetchers open a page when someone asks an AI tool about it. Training crawlers collect pages to train AI models. Blocking training while allowing search is a common choice; each row links to the operator's documentation.

Quick choices: Block training only · Block all · Allow all

  • Search

    OpenAI says it is used to surface websites in search results in ChatGPT's search features, and recommends allowing it for sites that want to appear there. OpenAI's docs

  • User fetch

    Used for certain user actions in ChatGPT and Custom GPTs. OpenAI says it does not crawl the web automatically and that, because these actions are started by a user, robots.txt rules may not apply. OpenAI's docs

  • Training

    Crawls content that may be used to train OpenAI's generative AI foundation models. OpenAI says disallowing it indicates a site's content should not be used in that training. OpenAI's docs

  • Search

    Anthropic says it navigates the web to improve search result quality for users, and that blocking it may reduce your visibility in user search results. Anthropic's docs

  • User fetch

    Fetches pages when people ask Claude questions. Anthropic says blocking it prevents Claude retrieving your content in response to a user query. Anthropic's docs

  • Training

    Collects web content that could contribute to training Anthropic's models. Anthropic says blocking it signals that your future content should be excluded from training, and that its bots honour robots.txt. Anthropic's docs

  • Search

    Surfaces and links websites in Perplexity's search results. Perplexity says it is not used to crawl content for AI foundation models. Perplexity's docs

  • User fetch

    Visits pages to answer a Perplexity user's question. Perplexity says that because a user requested the fetch, it generally ignores robots.txt rules. Perplexity's docs

  • Training

    A robots.txt token, not a separate crawler: it controls whether content Google crawls may be used to train future Gemini models and for grounding. Google says it does not affect inclusion or ranking in Google Search. Google's docs

  • Training

    Does not crawl: it controls whether pages Applebot crawled may be used to train Apple's foundation models. Apple says pages that disallow it can still appear in search results. Apple's docs

  • Products and training

    Amazon says it is used to improve its products and services and may be used to train Amazon AI models, and that it respects robots.txt. Amazon's docs

  • Search

    Crawls pages in real time for DuckDuckGo's AI-assisted answers. DuckDuckGo says the data is not used to train AI models, and a disallow takes effect after 72 hours. DuckDuckGo's docs

  • Training

    Meta says it crawls the web for uses such as training foundation AI models and improving products by indexing content directly. Meta's docs

  • Training

    Common Crawl's crawler, which builds an open repository of web crawl data that anyone can use, including to train AI models. Common Crawl's docs

  • Undocumented

    ByteDance's crawler. We found no official documentation from ByteDance to cite, so we make no claims about what it is used for. The checker scores it with the training crawlers.

Your robots.txt lines

Add these to your site's /robots.txt. A crawler named in a group follows only that group, so copy any other Disallow lines it should obey into the group too. User fetchers may not follow robots.txt at all, as their operators say above.

Then check what your live robots.txt says to each crawler.