# Check AI crawler access

Last updated: 2026-10-19

`POST /v1/access-check` takes one page URL, reads the robots.txt of that site, and says for five AI crawlers whether they may fetch the page: `GPTBot`, `ClaudeBot`, `PerplexityBot`, `Google-Extended` and `CCBot`. For each crawler it gives `allowed`, the User-agent group used, the line that decided it, the group's Crawl-delay and the reason for the verdict. It also lists the robots.txt's Sitemap lines, reads its Content-Signal line, and says whether the site has an llms.txt. Price: US$0.002 per call.

Use it before you publish or link a page and want to know whether AI crawlers are allowed to read it, or before you tell a user why an AI tool could not fetch a page.

Only the site's `/robots.txt` and `/llms.txt` are fetched. The page itself is not fetched, and nothing is stored.

## Request

```bash
curl -s -X POST https://vehcdj664efetfrsolne5umanq.srv.us/v1/access-check \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://www.iana.org/help/example-domains"}'
```

The call is paid: the server answers `402` with the offer, and the call is charged US$0.002 once it is answered below 400.

| Field | Required | Meaning |
|---|---|---|
| `url` | yes | The http(s) page URL to judge, up to 2,048 characters. Its site's `/robots.txt` and `/llms.txt` are read; the page itself is not. |

Unknown fields are refused with 400, and the call is not charged.

## Response

```json
{
  "url": "https://news.example.com/articles/1",
  "robots": { "url": "https://news.example.com/robots.txt", "http_status": 200, "served": true, "truncated": false },
  "crawlers": [
    { "crawler": "GPTBot", "allowed": false, "user_agent": "gptbot", "deciding_line": { "line": 3, "text": "Disallow: /" }, "crawl_delay": null, "reason": "rule_matched" },
    { "crawler": "ClaudeBot", "allowed": true, "user_agent": "*", "deciding_line": null, "crawl_delay": 10, "reason": "no_rule_matched" }
  ],
  "sitemaps": ["https://news.example.com/sitemap.xml"],
  "content_signal": { "search": "yes", "ai_input": "no", "ai_train": "no" },
  "llms_txt": { "url": "https://news.example.com/llms.txt", "http_status": 404, "served": false },
  "tier": "paid",
  "rail": "x402",
  "network": "eip155:8453",
  "receipt": "eyJhbGciOiJFZERTQSIs…"
}
```

The example above is illustrative. Another site gives other values.

- `url`: the page URL as normalised by the server.
- `robots.url`: the robots.txt that was read.
- `robots.http_status`: the status the robots.txt returned. A robots.txt that could not be read at all is refused with 502 instead.
- `robots.served`: `true` for a 2xx answer.
- `robots.truncated`: `true` when the robots.txt is longer than the 500 KB this check parses, so only its start was judged.
- `crawlers`: one verdict per crawler, in the order above.
- `crawlers[].allowed`: `true` or `false` for that crawler.
- `crawlers[].user_agent`: the User-agent group used, in lowercase (`gptbot`, `claudebot`, `perplexitybot`, `google-extended`, `ccbot`) when a group names the crawler, `*` when only the wildcard group applies, and `null` when no group applies (reason `no_group`, `server_error_disallow` or `unreachable_allow`).
- `crawlers[].deciding_line`: `{ line, text }` for the winning rule, or `null` when no rule matches.
- `crawlers[].crawl_delay`: the Crawl-delay in seconds of the group used, or `null` when that group sets none.
- `crawlers[].reason`: which rule decided the verdict:
  - `rule_matched`: a rule of the group used matches the path.
  - `no_rule_matched`: a group applies, but no rule matches the path, so the crawler is allowed.
  - `no_group`: no group names the crawler or the `*` wildcard, so the crawler is allowed.
  - `server_error_disallow`: the robots.txt answered 5xx, so every crawler is disallowed.
  - `unreachable_allow`: the robots.txt answered with another non-2xx status (for example 404), so every crawler is allowed.
- `sitemaps`: the absolute http(s) URLs of the `Sitemap` lines in the robots.txt, in file order, without repeats, up to 100. Empty when the robots.txt is not 2xx or names none.
- `content_signal`: the Content-Signal line of the robots.txt, read as `{ search, ai_input, ai_train }`. In the file these are written `search`, `ai-input` and `ai-train`. Each is `yes`, `no`, or `null` when the file does not name that signal. Every line of the file is read (up to the 500 KB cap), and the first value given for each signal wins. Case and spaces are ignored, and values other than `yes` or `no` are ignored. `content_signal` is `null` when the robots.txt is not 2xx, or when it has no Content-Signal line with a `yes` or `no` value.
- `llms_txt`: the site's `/llms.txt`, as `{ url, http_status, served }`. `http_status` is `null` when the file could not be read, and `served` is `true` for a 2xx answer. It is reported only: a missing or unreadable llms.txt never refuses the call and never changes a crawler verdict. `sitemaps`, `content_signal` and `llms_txt` do not change any `allowed` value either.
- `tier`: `paid`.
- `rail` and `network`: the payment rail and network used.
- `receipt`: a signed receipt of the answer, kind `access_check`. Keep it to show later what the robots.txt said on that day; check it offline with the receipt verifier.

## How the verdict is reached

- A 2xx robots.txt is read with the usual robots.txt rules: the group that names the crawler is used, otherwise the `*` group. Among the rules that match the path, the longest one decides; on a tie, an allow rule wins.
- A 5xx answer disallows the whole site for every crawler. `user_agent`, `deciding_line` and `crawl_delay` are `null` in that case, and `reason` is `server_error_disallow`.
- Any other non-2xx answer, for example a 404, allows every crawler, with `user_agent`, `deciding_line` and `crawl_delay` `null` and `reason` `unreachable_allow`.
- A robots.txt that cannot be read at all (a timeout, a DNS or connection error, or a redirect loop) gives no verdict. The call is answered `502` with `robots_unreachable` and is not charged.
- A robots.txt larger than 2 MB is refused with `422` and `robots_too_large`, and is not charged, because its rules cannot be judged in full.

## Errors

See [Errors](/docs/errors) for every code and what it means. The codes specific to this route:

| Status | Code | Meaning | Charged |
|---|---|---|---|
| 400 | `invalid_url` | `url` is missing, is not an http(s) URL, or is refused by the URL guard. `details.field` is `url`. | No |
| 400 | `bad_request` | The body is not a JSON object, or a field has the wrong type. | No |
| 422 | `robots_too_large` | The site's robots.txt is over 2 MB, so its rules cannot be judged in full. `details.robots_url` names the file. | No |
| 502 | `robots_unreachable` | The robots.txt could not be read (timeout, DNS or connection error, redirect loop, or blocked). `details.outcome` says which. | No |

Every answer with status 400 or above is not charged.