0.4.0: a free allowance that answers with a price, not a 429

[ralyodio]

Named crawlers were the only thing this could charge. The other half of the traffic is an ordinary client hammering a site far past what a reader would, and the usual answer to that is 429 Too Many Requests: go away and try later. That is the right answer when you have nothing to sell. Here we do.

createGateway({
  siteUrl: 'https://your-site.com',
  coinpay: { apiKey: process.env.COINPAY_X402_KEY },
  payTo: process.env.CRAWL_PAY_TO,

  freeQuota: 100,                    // 100 requests a minute, free, no account
  benefits: ['No rate limit', 'Bulk export endpoint', 'Every field, not just the summary'],
});

Past the allowance the gateway answers 402 with the offer, the RateLimit headers, and a body naming what ran out, when it comes back, and what a pass costs. The moment a caller runs out of free requests is the best sales pitch a site will ever get: it has just demonstrated it wants more than the free tier and is still holding the request.

On rotating addresses, which defeats this

The default identity is the caller's address, so a proxy rotation walks straight past it. That is deliberate and documented rather than patched. The arithmetic already argues for paying:

• Residential proxy bandwidth is sold by the gigabyte, and a crawl big enough to be worth rotating for passes a dollar on the first day.
• A rotation still fetches every page one at a time. It buys no speed.
• A pass is flat, with nothing to maintain and nothing to keep working.

Someone who spends more on proxies than the pass costs, in order to avoid the pass, has not been defeated by cleverness. They have made an arithmetic mistake, and the throttled page is where we point it out. Detection is a race that re-runs every time someone changes tactics. Price is not.

Set benefits to the things a rotation genuinely cannot get, a bulk endpoint above all, and evasion stops being worth the effort rather than being blocked. Where a caller can be identified properly, identify keys the allowance on an API key or account and it becomes exact.

Two ordering changes that matter

A valid pass is now read before anything that can refuse, so a pass holder is neither charged as a crawler nor metered. It used to be read only after the crawler lists matched, which was correct while the lists were the sole reason to refuse and is not once an allowance exists.

The sales page, robots.txt and .well-known/ stay reachable with the allowance spent. Being unable to reach the page that sells the fix would be the worst possible failure of a throttle whose entire purpose is to sell something. There is a test for it.

A contract I deliberately did not grow

handle() still answers with a Response or nothing. I first had the pass-through path return a bare object carrying RateLimit headers, which would have broken every adapter that checks the return value for truth. The allowance is advertised on the 402 instead, which is where it gets read anyway.

Tests

21 new, 76 total, all green. The ones worth naming:

• A pass lifts the throttle: 25 requests against an allowance of 1, all through.
• A named training crawler is still charged on its first request, so a generous free tier does not become a loophole for the thing this package was built to charge.
• An unidentifiable caller counts as within the allowance, because charging someone for our own edge not handing us an address would be charging for our gap, not their abuse.
• A throttled reader is not called a training crawler. Same price, different argument, and getting that wrong is both incorrect and insulting.
• Each address is counted separately, asserted as a test rather than hidden, because it is the known limit of the mechanism and not a bug.

🤖 Generated with Claude Code

https://claude.ai/code/session_01TKuF2jCbRj3GZTQVtwhi5m