How to tell a bot wall from the real page when an agent fetches a URL

Last updated: 2026-10-10

A 200 status does not mean the agent got the page. Check what came back before you summarise or quote it. POST /v1/fetch (US$0.002 per call) fetches one URL and returns a content_kind verdict (real, bot_wall, js_shell, paywall, empty, http_error and others), the clues behind it, the final URL, hashes and a signed receipt.

What a fetch can return instead of the page

• Bot challenge. A "checking your browser" page or a CAPTCHA. We report it as bot_wall and never try to get past it.
• JavaScript shell. An almost empty HTML page whose content is drawn by scripts. Reported as js_shell.
• Paywall or login teaser. The first paragraph, then a sign-in prompt. Reported as paywall.
• Redirect to another site. A consent page or a home page on a different host. Reported as off_site.
• Error or empty page. Reported as http_error or empty.

Only real means the text is the page's content. On any other verdict, a summary or a missing quote says nothing about the page.

Fetch a page

{ "url": "https://www.iana.org/help/example-domains", "return_text": true }

With return_text, you also get up to 20,000 characters of clean text. The answer lists injection_flags when the text holds patterns often used for prompt injection. Try the route free on demo data: [GET /v1/sample/fetch](https://vehcdj664efetfrsolne5umanq.srv.us/v1/sample/fetch). Details: Fetch a page and How it works.

Questions

How is the verdict decided?

By automated rules over the status, headers and text. Rules can be wrong, so every answer lists the signals behind its verdict.

Does it render JavaScript?

No. A page that needs JavaScript is reported as js_shell rather than summarised as empty text, so the agent knows to try another source.

Is a failed fetch charged?

A result about the page is charged, whatever the verdict, including 404s, bot walls and timeouts: the verdict is the answer. A bad URL, a blocked address or a host name that does not resolve is refused before payment and not charged. See Payments and Errors in the docs.

Can I show later what the page said?

Yes. The receipt records the URL, final URL, status, verdict, text hash and time, signed with Ed25519, and anyone can check it offline. See How to get a signed receipt of a web page.