Our fetcher

Last updated: 2026-10-10

AttestPage fetches web pages when an agent asks it to check a quote or a list of citations, fetch a page or check links. This page describes that fetcher and how site owners can stop it.

How to recognise it

• User agent: QuoteproofBot/1.0 (+https://vehcdj664efetfrsolne5umanq.srv.us/bot)
• robots.txt token: QuoteproofBot (matched without regard to case)
• It fetches only the URL a caller sent, plus that site's /robots.txt. It does not crawl or follow links on its own.
• For check/packages it also reads the public JSON APIs of registry.npmjs.org, pypi.org and crates.io for the names a caller sent, with the same user agent plus our contact address, and at most one crates.io request a second.

What it does

• GET requests only, without cookies, logins or any header passed through from the caller.
• http and https on ports 80 and 443 only. It refuses private, loopback and other internal addresses, and checks again after every redirect.
• Up to 5 redirects. It gives up after 5 seconds to connect and 10 seconds in all.
• Reads at most 2 MB, and only HTML, plain text and JSON.
• No JavaScript, no proxies, no IP rotation, and no attempt to get past CAPTCHAs or bot checks. A challenge page is reported to the caller as bot_wall.
• At most 1 request a second and 30 a minute to any one host.
• Page content is not stored. Receipts keep hashes of what was read, not the text.

robots.txt

The fetcher reads robots.txt before every fetch and keeps it for up to one hour. If there is a group for QuoteproofBot, only its rules apply; otherwise the * group applies. If your robots.txt answers with a server error (5xx), we treat the whole site as disallowed for an hour. To block it completely:

User-agent: QuoteproofBot
Disallow: /

A disallowed page is refused to the caller with 403 robots_disallowed, and the caller is not charged.

Opt-out list

We also keep a list of hosts that asked not to be fetched. Hosts on it are refused in the same way as a robots.txt disallow.

Contact

To ask about the fetcher or report a problem: no contact address is set on this server; the robots.txt opt-out works without one.

Security reports: see security.txt.