# Our fetcher

Last updated: 2026-10-10

AttestPage fetches web pages when an agent asks it to check a quote or a list of citations, fetch a page or check links. This page describes that fetcher and how site owners can stop it.

## How to recognise it

- User agent: `QuoteproofBot/1.0 (+https://vehcdj664efetfrsolne5umanq.srv.us/bot)`
- robots.txt token: `QuoteproofBot` (matched without regard to case)
- It fetches only the URL a caller sent, plus that site's `/robots.txt`. It does not crawl or follow links on its own.
- For `check/packages` it also reads the public JSON APIs of registry.npmjs.org, pypi.org and crates.io for the names a caller sent, with the same user agent plus our contact address, and at most one crates.io request a second.

## What it does

- GET requests only, without cookies, logins or any header passed through from the caller.
- http and https on ports 80 and 443 only. It refuses private, loopback and other internal addresses, and checks again after every redirect.
- Up to 5 redirects. It gives up after 5 seconds to connect and 10 seconds in all.
- Reads at most 2 MB, and only HTML, plain text and JSON.
- No JavaScript, no proxies, no IP rotation, and no attempt to get past CAPTCHAs or bot checks. A challenge page is reported to the caller as `bot_wall`.
- At most 1 request a second and 30 a minute to any one host.
- Page content is not stored. Receipts keep hashes of what was read, not the text.

## robots.txt

The fetcher reads robots.txt before every fetch and keeps it for up to one hour. If there is a group for `QuoteproofBot`, only its rules apply; otherwise the `*` group applies. If your robots.txt answers with a server error (5xx), we treat the whole site as disallowed for an hour. To block it completely:

```
User-agent: QuoteproofBot
Disallow: /
```

A disallowed page is refused to the caller with `403 robots_disallowed`, and the caller is not charged.

## Opt-out list

We also keep a list of hosts that asked not to be fetched. Hosts on it are refused in the same way as a robots.txt disallow.

## Contact

To ask about the fetcher or report a problem: no contact address is set on this server; the robots.txt opt-out works without one.

Security reports: see [security.txt](https://vehcdj664efetfrsolne5umanq.srv.us/.well-known/security.txt).