Skip to content

Fix client shutdown deadlock - #142

Merged
AlexStocks merged 6 commits into
apache:masterfrom
222twotwotwo:fix-client-shutdown-deadlock
Aug 29, 2026
Merged

AlexStocks merged 6 commits into
apache:masterfrom
222twotwotwo:fix-client-shutdown-deadlock

Conversation

@222twotwotwo

Copy link
Copy Markdown

What this PR does:

Prevents client shutdown from deadlocking or hanging when session teardown can trigger reconnect logic.

This PR:

  • closes client sessions outside the client map lock by snapshotting ssMap during shutdown
  • introduces runReconnect() to start reconnect loops only while the client is open and track them with the client WaitGroup
  • routes session-triggered reconnect through runReconnect() instead of synchronously calling reConnect()
  • ensures reconnect backoff is cancelled by Client.Close()
  • adds regression coverage for shutdown/reconnect interleavings, backoff cancellation, and no-op reconnect after close

Which issue(s) this PR fixes:

Fixes #132

Special notes for your reviewer:

The branch has been merged with the latest origin/master

Additional tests cover:

  • session error triggering reconnect while Client.Close() is running
  • Client.Close() cancelling reconnect backoff
  • runReconnect() becoming a no-op after client shutdown
  • Client.Close() waiting for an in-flight session-triggered reconnect

Does this PR introduce a user-facing change?:

NONE

Comment thread client.go Outdated
close(done)
return done
default:
c.wg.Add(1)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] 这里把整个重连 goroutine 计入 wg 后,NewSessionCallback 内调用公开的 Client.Close() 会等待当前 goroutine 自己,形成确定性死锁。固定 Base/Head 的公开入口探针中,Base 在回调调用 Close() 并返回错误后正常退出;当前 Head 在 500ms 内始终无法返回。回调文档没有禁止关闭所属 endpoint,这类连接校验失败后主动停机的用法不能被锁死。请避免让 Close() 等待正在执行该回调的 worker(例如把停止信号与外部 join 分离,或调整重连任务的生命周期登记),并补一个回调内关闭客户端的回归测试。

@222twotwotwo 222twotwotwo Aug 16, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

收到, NewSessionCallback内调用 Client.Close() 时不会再等待当前正在执行 callback 的 goroutine 自己。

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses a client lifecycle deadlock during shutdown by decoupling session teardown from reconnect logic and ensuring reconnect work is coordinated and cancellable via client shutdown.

Changes:

  • Routes session-triggered reconnect through a new client.runReconnect() helper instead of calling reConnect() inline from session teardown.
  • Updates client.stop() to snapshot and close sessions outside the client lock to avoid lock re-entrancy during shutdown.
  • Adds regression tests covering shutdown/reconnect interleavings and reconnect backoff cancellation behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
session.go Switches session teardown from synchronous reConnect() to asynchronous, tracked runReconnect() to avoid shutdown lock re-entry.
client.go Introduces runReconnect() (tracked via WaitGroup), waits for initial reconnect in RunEventLoop, and closes sessions outside the client lock during stop().
client_test.go Adds regression tests for reconnect/backoff cancellation and shutdown/reconnect interleavings.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread client.go Resolved
@AlexStocks
AlexStocks merged commit 1fff403 into apache:master Aug 29, 2026
2 checks passed
222twotwotwo pushed a commit to 222twotwotwo/dubbo-getty that referenced this pull request Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] transport review findings: shutdown deadlock, nil-listener panic, websocket leak, and reconnect lifecycle issues

3 participants