Skip to content

nstun: cap aggregate host-to-guest TCP buffering - #380

Open
PedroACFerreira wants to merge 2 commits into
google:masterfrom
PedroACFerreira:codex/nstun-host-to-guest-tcp-budget
Open

PedroACFerreira wants to merge 2 commits into
google:masterfrom
PedroACFerreira:codex/nstun-host-to-guest-tcp-budget

Conversation

@PedroACFerreira

Copy link
Copy Markdown

NSTUN stops reading from an individual TCP connection when it has buffered
about 256 KiB of response data. That limit is per connection. In an
NSTUN-enabled jail, a program can open many connections to a cooperating
server and leave the replies unread. The buffers then add up in the nsjail
parent process.

This PR puts a 64 MiB shared limit on that host-to-guest TCP data. When the
limit is full, nsjail pauses reads from the affected connections and resumes
them as space becomes available. PR #300 added a similar limit for data
going in the other direction.

I split the fix into two commits. The first tags queued epoll socket
notifications so events from a closed connection are ignored when Linux
reuses its file descriptor. The second adds the shared limit, the pause
and resume logic, and tests. It also updates the close and HTTP CONNECT
paths that need to account for the new limit.

What I tested

I built the unmodified d8415bc8eea43cdd413daba25af63127019a1189 and
the patched version in network-isolated containers. In the main test, the
guest opened 400 TCP connections to a cooperating sender and did not read
the replies. Both senders accepted all 400 connections and sent 131,072,000
bytes. Both guests reported 400 connections and the same final queued byte
count (25,788,800). I checked those counts so the memory comparison was
between the same amount of work.

With a 384 MiB limit on the runner container, I measured:

Memory measured Unmodified With this patch
Runner container peak 338,907,136 bytes 278,528,000 bytes
nsjail parent peak, first matched run 173,682,688 bytes 84,021,248 bytes
nsjail parent peak, repeat 173,174,784 bytes 84,598,784 bytes

The parent numbers are its peak resident memory. They include all of the
parent's allocations, not just TCP buffers.

In a separate 256-connection test with a 200 MiB runner limit, the
unmodified runner was killed for exceeding that limit and the patched
runner finished. The unmodified run ended before it could report its final
guest queue, so I treat this as a local memory-limit example rather than a
general promise that the patch prevents OOM.

The patched build passed six focused native tests. It also received 1 MiB
of generated data byte-for-byte through a real TUN device using a direct
REDIRECT rule. I can provide the network-isolated reproducer and raw
receipts if useful.

The 64 MiB limit covers the TCP response bytes held by NSTUN. It does not
cap the parent process's total memory use or kernel socket buffers. I have
not run the full test suite.

@google-cla

google-cla Bot commented Oct 29, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Tag TCP epoll registrations with a generation and ignore events for a prior owner of a reused file descriptor. This prepares the event loop for shared transmit-budget waiters.
Cap the parent-owned TCP response payload across flows at 64 MiB. Pause and resume host reads when the shared budget is exhausted, and account for EOF, HUP, proxy negotiation, and blocked-flow wakeups. Add focused regression coverage.
@PedroACFerreira
PedroACFerreira force-pushed the codex/nstun-host-to-guest-tcp-budget branch from 91348cb to b275a78 Compare October 29, 2026 16:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant