[Bug]: Ubuntu UT job cascades across packages after cluster and checkpoint timeouts

[gouhongshen]

Is there an existing issue for the same bug?

• I searched matrixorigin/matrixone and matrixorigin/CI for waitAnyShardReadyLocked, base-cluster deadline failures, the observed checkpoint tests, and multi-test Ubuntu timeout failures; no matching issue was found.

Branch Name

main

Commit ID

0b8b53d

Other Environment Information

• GitHub Actions workflow: MatrixOne ALL CI
• Job: Matrixone CI / UT Test on Ubuntu/x86
• Runner: Ubuntu 22.04 x86
• PR head tested: 6961ff16d217e947618f43b92216c944f4559181

Actual Behavior

The Ubuntu UT job reported 20+ failures across unrelated ownership boundaries. The common symptoms include:

• embedded-cluster tests failing during setup with context deadline exceeded / internal error: waitAnyShardReadyLocked;
• HAKeeper heartbeats and proposals timing out, Dragonboat processing multiple delayed local ticks, and connections being reset;
• TestHandleBackup timing out while forcing a checkpoint;
• TestReadCheckpoint first observing an unexpected checkpoint count and then panicking after dereferencing a nil CheckpointEntry;
• later authenticated-cluster tests failing after the shared service connection became unavailable.

The final failure list includes engine lease, TAE checkpoint, Iceberg, embed cluster, lock service, and data-branch tests:

TestCheckLeaseFailed, TestReadCheckpoint, TestHandleBackup,
TestIcebergSQLEngineEmbeddedSecurityErrors, TestRunSQLWithFrontend,
TestRowCountOverMySQLProtocol, TestBaseClusterCanWorkWithNewCluster,
TestCreateDB, TestBaseClusterOnlyStartOnce, TestClusterCanStartNewCNServices,
TestCommitWithLockTablesChanged, TestKeepRemoteLockPeerCursorIsFairAcrossSlowRounds,
TestIssue26111DataBranchDatabaseWithCyclicForeignKeys,
TestIssue26121DatabaseOperationsKeepOrdinaryInternalLookingTables,
TestIssue26087ConcurrentDataBranchQuota, ...

Failing job: https://github.com/matrixorigin/matrixone/actions/runs/30620241711/job/91122944038

This was observed while validating PR #26502, but that PR only changes pkg/logservice/snapshot.go and its focused unit test. The same CI job reports the entire pkg/logservice package as PASS (181.18s), including:

TestInitSnapshotMgrReplacesInMemoryView PASS
TestInitSnapshotMgr PASS
TestAddSnapshot PASS

Expected Behavior

The Ubuntu UT job should complete deterministically. A timeout or crash in one package/shared embedded cluster should not cascade into many unrelated test failures, and checkpoint tests should not panic after a failed prerequisite assertion.

Steps to Reproduce

1. Run the full Ubuntu/x86 make ut workflow using the same package parallelism as MatrixOne ALL CI.
2. Observe delayed Dragonboat ticks and HAKeeper heartbeat/proposal timeouts during concurrent package execution.
3. Observe embedded-cluster initialization failing with waitAnyShardReadyLocked and subsequent tests failing immediately or losing their service connection.
4. Inspect the final FAIL_UT_CASES summary and the TAE checkpoint panic.

Additional information

The run's SCA, BVT, and coverage jobs succeeded; the failure was isolated to the full Ubuntu UT job. Investigation should determine whether runner resource contention, shared embedded-cluster lifecycle, and checkpoint test cleanup/error handling each require separate follow-up fixes.

[gouhongshen]

A second independent Ubuntu/x86 UT run shows the same embedded-cluster failure family on current main base 0b8b53dff3869dcd5e3ab52992852c9ea6545026 (tested PR head 1fd0b38f5c31b681acca920cd983b05779095b08):

https://github.com/matrixorigin/matrixone/actions/runs/30623452935/job/91133128589

New concrete evidence from this run:

• TestBasicCluster timed out in waitAnyShardReadyLocked after HAKeeper propose tick failed: timeout and raft shard already closed.
• TestTableFeatures lost HAKeeper while allocating an ID (127.0.0.1:12020: connection reset by peer).
• TestPitrCases, TestAlterRoleCases, and TestCreateAndDropPitr failed to initialize the base cluster because multiple processes attempted the fixed gossip listener 0.0.0.0:11060 (bind: address already in use).
• TestIcebergSQLEngineEmbeddedDeleteApply timed out creating its embedded tenant account.

The same job also found a separate TAE transfer-page data race; that independent root cause is tracked in #26523 rather than folded into this cluster issue. TestInvalidTimestamp already has #26491.

[gouhongshen-gmail]

The single permitted rerun also failed, confirming this is not an isolated first-attempt result.

• Workflow run: https://github.com/matrixorigin/matrixone/actions/runs/30623452935
• Attempt: 2
• Failed job: https://github.com/matrixorigin/matrixone/actions/runs/30623452935/job/91173024057
• Tested head: 1fd0b38f5c31b681acca920cd983b05779095b08
• Failed cases: TestPartitionBasedShardCanBeCreated, TestPartitionBasedShardCanBeDeleted

Attempt 2 failed in pkg/tests/shard, a different package/test surface from attempt 1, but with the same embedded-cluster symptoms:

failed to get checker state: timeout
failed to send tn/log service heartbeat: context deadline exceeded
HAKeeper backend: broken pipe / backend connection closed
wait TN ready timeout: context deadline exceeded
internal error: waitAnyShardReadyLocked

It also exposed a concrete cascade in pkg/tests/shard/shard_test.go: cluster startup fails inside sync.Once via require.NoError(t, new.Start()); the shared cluster is never assigned, but the Once is consumed. A subsequent test obtains the nil shareCluster and panics at c.GetCNService(0) (runShardClusterTestWithReuse, line 105 in this revision). Thus the job includes both the underlying embedded-cluster/HAKeeper instability and missing failure isolation in the shared test-cluster helper.

Per the PR CI policy, run attempt 2 will not be rerun again. PR #26504 still has no code overlap with pkg/embed, pkg/tests/shard, HAKeeper, or shared test-cluster lifecycle code.

[XuPeng-SH]

Latest-main audit on 2026-08-01:

• Current main: 3e9ad81b9a13732305b168e0f443fd76cca62a81.
• Since the original base 0b8b53dff3869dcd5e3ab52992852c9ea6545026, fix: join HAKeeper task scheduler before log store close #26507 merged a related but narrower fix: it makes LogService shutdown join the HAKeeper task-scheduler goroutine before NodeHost.Close. This can remove one shutdown race, but it does not address the complete failure family recorded here.
• No post-report commit changed pkg/embed, pkg/tests/shard, or pkg/vm/engine/tae/db/checkpoint, where the base-cluster startup cascade, consumed sync.Once/nil shared cluster, and nil CheckpointEntry panic were observed.
• fix(gossip): disable unsupported push-pull route snapshots #26538 passed one Ubuntu/x86 full UT run after the later main changes, but [Bug]: Ubuntu UT job cascades across packages after cluster and checkpoint timeouts #26519 is intermittent and already reproduced across an independent run and its rerun. A single green run is not sufficient closure evidence.
• No merged or open PR references [Bug]: Ubuntu UT job cascades across packages after cluster and checkpoint timeouts #26519 as fixed.

Conclusion: latest main contains a partial LogService shutdown improvement, but the broader Ubuntu/x86 embedded-cluster/checkpoint cascade is not demonstrated to be fully fixed. Keep this issue open until repeated full-UT validation covers the recorded failure surfaces.