Skip to content

Add BlockPrivateAddressesPlugin to reject requests to non-public addresses - #670

Draft
aivus wants to merge 1 commit into
masterfrom
block-private-addresses-plugin
Draft

aivus wants to merge 1 commit into
masterfrom
block-private-addresses-plugin

Conversation

@aivus

@aivus aivus commented Nov 8, 2026

Copy link
Copy Markdown
Member

Opt-in built-in plugin for applications which scrape urls supplied by untrusted users, where a downloaded page can otherwise make the scraper request internal services (server-side request forgery). The default behaviour is unchanged: the scraper does not restrict destinations unless the plugin is added, because mirroring intranet, staging or localhost sites is a legitimate use.

import scrape, { BlockPrivateAddressesPlugin } from 'website-scraper';

await scrape({ urls: [untrustedUrl], directory, ignoreErrors: true, plugins: [new BlockPrivateAddressesPlugin()] });

How it works

The plugin registers a single beforeRequest action which

  • rejects a resource url whose host is an ip literal in a loopback, link-local, private, CGNAT, multicast, reserved or documentation range, including the initial urls (the WHATWG URL parser normalizes forms such as 0x7f000001, 2130706433 and 127.1);
  • appends a got beforeRedirect hook which rejects redirects to such literals, keeping any hooks the user configured;
  • wraps the dns lookup used for the connection (dnsLookup, or dnsCache.lookup when a dns cache is configured, else dns.lookup) so that a hostname resolving to any blocked address fails the request. Node skips the lookup for ip literals, which is why those are checked separately. The validated address is the one the socket connects to.

Ranges are checked with Node's net.BlockList, which also handles IPv4-mapped IPv6 addresses (::ffff:a.b.c.d). No new dependency.

A blocked request is reported as an ordinary resource error, so onResourceError is called and ignoreErrors decides whether the scrape continues.

Exports and docs

Exported as a named export from the package entry point and from lib/plugins/index.js. Documented in the README under "Built-in plugins" with a dedicated subsection.

Tests

  • Unit tests for the range check (blocked and allowed samples for IPv4, IPv6 and mapped addresses), for the beforeRequest action (ip literal urls, preserved request options and hooks), for the redirect hook, and for the dns wrapper (single and all: true results, lookup errors, dnsCache fallback).
  • A functional test with a real loopback http server asserting it receives zero hits across the direct, dns (localhost) and redirect vectors, and that a blocked initial url fails the scrape when ignoreErrors is false.
  • 293 tests passing, eslint clean, 100% coverage on the new file.

🤖 Generated with Claude Code

…esses

Opt-in built-in plugin for applications which scrape urls supplied by
untrusted users. It registers a beforeRequest action which rejects resource
urls with an ip literal in a loopback, link-local, private, CGNAT, multicast,
reserved or documentation range, adds a beforeRedirect hook which rejects
redirects to such literals, and wraps the dns lookup used for the connection
(dnsLookup, or dnsCache.lookup when configured) so that a hostname resolving to
any blocked address fails the request. Ranges are checked with net.BlockList,
which also covers IPv4-mapped IPv6 addresses.

The plugin is exported from the package entry point and from lib/plugins and
documented in the README plugins section.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@aivus
aivus marked this pull request as draft November 8, 2026 22:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant