Skip to content

fix(frontend): promptly cancel requests after client disconnect - #27634

Merged
XuPeng-SH merged 7 commits into
mainfrom
fix/27595-prompt-direct-disconnect
Aug 27, 2026
Merged

fix(frontend): promptly cancel requests after client disconnect#27634
XuPeng-SH merged 7 commits into
mainfrom
fix/27595-prompt-direct-disconnect

Conversation

@aptend

@aptend aptend commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

issue #27595

What this PR does / why we need it:

Probe active frontend requests for a disconnected client from the first one-second monitor tick. This promptly cancels the request context and lets the existing connection cleanup roll back an open transaction, so locks are not retained after a direct client disconnect.

The liveness hot path is bounded by the configured max_connections limit (100k): it reuses the request snapshot and a cached non-owning RawConn, performs only a zero-timeout poll for the common no-event case, and calls recv(MSG_PEEK | MSG_DONTWAIT) only when the socket is readable. Probe errors use a process-wide rate-limited event. TLS and unread protocol bytes remain covered.

Steady-state benchmark medians (-benchtime=5x -count=5) show zero allocations for the new path. A realistic 10k-distinct-socket scan at 100% active takes about 7.36 ms per one-second tick (0.74% of one core); the 100k upper-bound shared-fd case takes about 55.1 ms (5.5% of one core). At 100% active, the reviewed implementation took about 5.61 ms / 2.5 MB / 50k allocs for 10k connections and 70.3 ms / 27.3 MB / 500k allocs for 100k connections; this update takes about 2.98 ms / 0 allocs and 55.1 ms / 0 allocs in the same shared-fd cases.

Add policy, socket, allocation, TLS, unread-byte, and probe-refresh coverage, plus a reproducible end-to-end scenario covering an explicit transaction that holds a FOR UPDATE lock while a long-running statement is active.

Validation:

  • make build
  • go build, go vet, and full tests for ./pkg/frontend with the required CGO environment
  • full frontend -race run; seven lifecycle/probe tests also pass individually with -race -count=100
  • Python scenario self-tests, compile check, and Ruff
  • issue regression scenario against the locally built server (2 runs; lock released in 0.412s and 0.663s)

@aptend
aptend requested a review from XuPeng-SH as a code owner August 26, 2026 07:23
@qodo-code-review

Copy link
Copy Markdown

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

@XuPeng-SH XuPeng-SH left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep review of exact head fd50bd36c1f93b253e59bbefffe12e364fe50382 against merge base 1891117aedce0419a81630b3011906b560b93d0a completed.

The correctness/lifecycle direction is sound: only active requests are considered, the Unix probe is non-consuming, beginClose seals admission before cancellation, and the existing cleanup remains the sole rollback/resource owner. The real disconnect scenario demonstrates the lock is released promptly. One production performance blocker remains.

[P1 performance] Do not run two socket syscalls per active request every second without a bounded-capacity design and evidence.

This changes the monitor from once per 5 seconds and only after a 30-second grace period to once per second for every active request. For the common workload of many 1–30 second queries, the old path performed zero socket probes while this head serially performs poll(2) plus recvfrom(MSG_PEEK|MSG_DONTWAIT) for each request on one monitor goroutine, in addition to scanning every connection and allocating the active slice. A temporary local macOS probe using the same live-TCP SyscallConn.Control / Poll / Recvfrom sequence took 31–32ms per pass for 2,000 live connections; 10,000 active connections is roughly 160ms of one core per tick before manager overhead, and the frontend allows max_connections up to 100,000. At higher occupancy the monitor can consume material CPU or fall behind its own interval.

Please make the no-event live path materially cheaper (for example, avoid Recvfrom when Poll reports no relevant event, and/or batch/stagger/cap probes rather than issuing two syscalls for every request in one tick), then attach before/after benchmarks for realistic 10k and upper-bound connection populations with active ratios, allocations, and a stated CPU/tick budget. Retain the first-tick disconnect latency regression and false-positive/TLS/unread-byte coverage. Also rate-limit/deduplicate persistent probe-error diagnostics if the new cadence can emit the same error every second.

Exact-head CI is green; this is independent of CI correctness.

@matrix-meow matrix-meow added size/L Denotes a PR that changes [500,999] lines and removed size/M Denotes a PR that changes [100,499] lines labels Aug 27, 2026

@XuPeng-SH XuPeng-SH left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deep review of exact head 92e43943f35e7687700ced7d2f83d10d6e7c373f against merge base b0ae66c04d9a67175d2dfb8bf1c777fb888e1430 completed.

The previous performance blocker is closed. The common live-socket path now performs one zero-timeout poll, calls recv(MSG_PEEK | MSG_DONTWAIT) only for readable events, reuses both per-connection callback state and the manager snapshot, and has a zero-allocation regression test. The supplied 10k distinct-FD and 100k upper-bound measurements establish a bounded per-tick CPU cost under the existing connection cap, while persistent probe errors use a process-wide bounded event.

I also re-audited correctness and lifecycle behavior: the probe is non-consuming; Linux POLLRDHUP and Darwin POLLHUP cover disconnects even with unread bytes; syscall.RawConn.Control keeps the descriptor valid during the callback; TLS replacement refreshes the cached probe; stale/closing routines are not re-probed; and beginClose only seals/cancels lifecycle work while the existing single cleanup owner performs rollback and connection teardown. The first-tick policy and real explicit-transaction FOR UPDATE regression demonstrate that issue #27595 is addressed without introducing a new wait cycle or unbounded logging path.

Exact-head CI is green, including frontend UT and BVT. No blocking correctness, performance, or unhappy-path issue found.

@mergify mergify Bot added the queued label Aug 27, 2026
@mergify

mergify Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Merge Queue Status

  • Entered queue2026-08-27 12:26 UTC · Rule: main · triggered by rule Automatic queue on approval for main
  • 🟠 Checks running · in-place
  • 🚫 Left the queue2026-08-27 12:31 UTC · at 2b6eaa6c0befea2d48c0a7f9cb650ffcb370648e

This pull request spent 4 minutes 54 seconds in the queue, with no time running CI.

Reason

The pull request can't be updated

head ref does not exist

Hint

You should update or rebase your pull request manually. If you do, this pull request will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue, you can requeue the pull request, without updating it, by posting a @mergifyio queue comment.

Tick the box to put this pull request back in the merge queue (same as @mergifyio queue).

  • Requeue this pull request

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dequeued kind/bug Something isn't working size/L Denotes a PR that changes [500,999] lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants