https://crawl-preflight.postnov01.workers.dev/crawl-check...
Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.
#54
$0.02
2
1
Points earned against points available, biggest lever first.
| #3 | 57 | 6.3K txns | $0.0010 |
| #4 | 55 | 240 txns | $1.00 |
| #5 | 54 | 455 txns | $0.01 |