How the check works
pigfox.ai checks one site, the one you type, only when you ask. Here is exactly what it fetches and how it scores what it finds.
What is fetched
For a site such as example.com, a check makes four requests, plus any redirects the homepage makes:
- The homepage,
https://example.com/, reading at most the first 1 MB. If https can't connect, it trieshttp://once. It follows up to 5 redirects, and only within the same site (for example towww.example.com) or from http to https. /robots.txton the address the homepage ended up at, reading at most 500 KB./llms.txt, reading at most 64 KB./llms-full.txt, as a HEAD request only, to see whether it exists.
Each request identifies itself as PigfoxAIReadiness/1.0 (+https://pigfox.ai/about), must answer within 5 seconds, and a whole check stops after 12 seconds. Nothing else on the site is fetched, and no links are followed.
The checker connects only to public addresses on the standard web ports, 443 and 80. It refuses private, loopback, link-local and reserved addresses, and it checks the address again at the moment it connects. Each visitor can start six checks a minute, and a site's result is reused for 10 minutes, so checking the same site again doesn't fetch it again.
What is reported
- AI crawler access. robots.txt is read the way crawlers read it (RFC 9309): a crawler follows the group that names it, or else the
*group, and the longest matching rule wins. The report shows the answer for the homepage path for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot, DuckAssistBot and*. No robots.txt (a 4xx answer) means everything is allowed. A 5xx answer, or no answer, means access is unknown. - llms.txt. Whether it exists and starts with a
#heading, as the llms.txt format expects. - The homepage. HTTP status, the final address, the title and its length (30 to 60 characters is best), meta description, canonical link, language, meta robots and X-Robots-Tag noindex, whether there is an h1, and every JSON-LD block with its Schema.org types.
How the score adds up
| Check | Points |
|---|---|
| Search and assistant crawlers allowed (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Amazonbot, DuckAssistBot) | 30 |
| Training crawlers allowed (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, Bytespider) | 10 |
| Homepage loads with HTTP 200 | 10 |
| Homepage is not noindex | 10 |
| Valid JSON-LD | 8 |
| Title of 30 to 60 characters | 5 |
| Meta description | 5 |
| Canonical link | 5 |
| Organization and WebSite structured data | 5 |
| llms.txt with a heading | 5 |
| An h1 heading | 4 |
| Page language set | 3 |
Blocking training crawlers is a fair choice for many sites; it costs only a few points here. The score measures whether AI tools may read your homepage and can tell what it is. It doesn't predict rankings or citations.
Who runs it
pigfox.ai is built by Pigfox, the studio behind Triremes, which measures SEO and AEO gaps across 500,000+ business websites. Read the privacy notice.