crawl-preflight.postnov01.workers.dev

As advertised

Read from a directory and written down with its source and date. Not measured. provenance: recorded

url

https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain

hostname

crawl-preflight.postnov01.workers.dev

operator

Crawl Pre-Flight

capability

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.

capability tag

web-search

source directory

cdp-bazaar

Seen

first seen

4 September 2026 2026-09-04T11:24:51Z

last seen

4 September 2026 2026-09-04T11:24:51Z

current status

active

observations

31

observation cost, USD

$0.46

Price history

1 rows. 0 observed on a call Crosspeel made, 1 recorded from an advertised figure. The provenance column says which, on every row, and the two are never added together.

Cluster membership

This endpoint is not a member of any published cluster. That means either no match was found at the current threshold, or the observations behind a candidate match do not yet reach the depth the method requires. Neither is a statement about the endpoint. The thresholds are at /method.

Observations