Check AI crawler access

Last updated: 2026-10-19

POST /v1/access-check takes one page URL, reads the robots.txt of that site, and says for five AI crawlers whether they may fetch the page: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot. For each crawler it gives allowed, the User-agent group used, the line that decided it, the group's Crawl-delay and the reason for the verdict. It also lists the robots.txt's Sitemap lines, reads its Content-Signal line, and says whether the site has an llms.txt. Price: US$0.002 per call.

Use it before you publish or link a page and want to know whether AI crawlers are allowed to read it, or before you tell a user why an AI tool could not fetch a page.

Only the site's /robots.txt and /llms.txt are fetched. The page itself is not fetched, and nothing is stored.

Request

curl -s -X POST https://vehcdj664efetfrsolne5umanq.srv.us/v1/access-check \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://www.iana.org/help/example-domains"}'

The call is paid: the server answers 402 with the offer, and the call is charged US$0.002 once it is answered below 400.

Unknown fields are refused with 400, and the call is not charged.

Response

{
  "url": "https://news.example.com/articles/1",
  "robots": { "url": "https://news.example.com/robots.txt", "http_status": 200, "served": true, "truncated": false },
  "crawlers": [
    { "crawler": "GPTBot", "allowed": false, "user_agent": "gptbot", "deciding_line": { "line": 3, "text": "Disallow: /" }, "crawl_delay": null, "reason": "rule_matched" },
    { "crawler": "ClaudeBot", "allowed": true, "user_agent": "*", "deciding_line": null, "crawl_delay": 10, "reason": "no_rule_matched" }
  ],
  "sitemaps": ["https://news.example.com/sitemap.xml"],
  "content_signal": { "search": "yes", "ai_input": "no", "ai_train": "no" },
  "llms_txt": { "url": "https://news.example.com/llms.txt", "http_status": 404, "served": false },
  "tier": "paid",
  "rail": "x402",
  "network": "eip155:8453",
  "receipt": "eyJhbGciOiJFZERTQSIs…"
}

The example above is illustrative. Another site gives other values.

• url: the page URL as normalised by the server.
• robots.url: the robots.txt that was read.
• robots.http_status: the status the robots.txt returned. A robots.txt that could not be read at all is refused with 502 instead.
• robots.served: true for a 2xx answer.
• robots.truncated: true when the robots.txt is longer than the 500 KB this check parses, so only its start was judged.
• crawlers: one verdict per crawler, in the order above.
• crawlers[].allowed: true or false for that crawler.
• crawlers[].user_agent: the User-agent group used, in lowercase (gptbot, claudebot, perplexitybot, google-extended, ccbot) when a group names the crawler, * when only the wildcard group applies, and null when no group applies (reason no_group, server_error_disallow or unreachable_allow).
• crawlers[].deciding_line: { line, text } for the winning rule, or null when no rule matches.
• crawlers[].crawl_delay: the Crawl-delay in seconds of the group used, or null when that group sets none.
• crawlers[].reason: which rule decided the verdict:
• rule_matched: a rule of the group used matches the path.
• no_rule_matched: a group applies, but no rule matches the path, so the crawler is allowed.
• no_group: no group names the crawler or the * wildcard, so the crawler is allowed.
• server_error_disallow: the robots.txt answered 5xx, so every crawler is disallowed.
• unreachable_allow: the robots.txt answered with another non-2xx status (for example 404), so every crawler is allowed.
• sitemaps: the absolute http(s) URLs of the Sitemap lines in the robots.txt, in file order, without repeats, up to 100. Empty when the robots.txt is not 2xx or names none.
• content_signal: the Content-Signal line of the robots.txt, read as { search, ai_input, ai_train }. In the file these are written search, ai-input and ai-train. Each is yes, no, or null when the file does not name that signal. Every line of the file is read (up to the 500 KB cap), and the first value given for each signal wins. Case and spaces are ignored, and values other than yes or no are ignored. content_signal is null when the robots.txt is not 2xx, or when it has no Content-Signal line with a yes or no value.
• llms_txt: the site's /llms.txt, as { url, http_status, served }. http_status is null when the file could not be read, and served is true for a 2xx answer. It is reported only: a missing or unreadable llms.txt never refuses the call and never changes a crawler verdict. sitemaps, content_signal and llms_txt do not change any allowed value either.
• tier: paid.
• rail and network: the payment rail and network used.
• receipt: a signed receipt of the answer, kind access_check. Keep it to show later what the robots.txt said on that day; check it offline with the receipt verifier.

How the verdict is reached

• A 2xx robots.txt is read with the usual robots.txt rules: the group that names the crawler is used, otherwise the * group. Among the rules that match the path, the longest one decides; on a tie, an allow rule wins.
• A 5xx answer disallows the whole site for every crawler. user_agent, deciding_line and crawl_delay are null in that case, and reason is server_error_disallow.
• Any other non-2xx answer, for example a 404, allows every crawler, with user_agent, deciding_line and crawl_delay null and reason unreachable_allow.
• A robots.txt that cannot be read at all (a timeout, a DNS or connection error, or a redirect loop) gives no verdict. The call is answered 502 with robots_unreachable and is not charged.
• A robots.txt larger than 2 MB is refused with 422 and robots_too_large, and is not charged, because its rules cannot be judged in full.

Errors

See Errors for every code and what it means. The codes specific to this route:

Every answer with status 400 or above is not charged.