You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Ubuntu/x86 UT job fails in pkg/tests/partition::TestAlterTableDropPartition while executing CREATE TABLE, before the partition-drop assertion:
ERROR cannot find lockservice address
target: 2-cn-16001
cns: []
ERROR failed to lock on remote
error: cannot connect to backend: none
Failure evidence: UT job 98235767659. The test fails after 0.40s with Received unexpected error: cannot connect to backend: none.
Root cause hypothesis
This is an admission/readiness ordering bug introduced or exposed by PR #27430, not by the DISTINCT aggregation changes in PR #27693.
clusterservice.GetCNServiceWithoutWorkingStateWithContext now excludes every CN while view-metadata admission is Preparing/Enabled unless ViewMetadataAdmissionReady is true.
lockservice uses that API to resolve a remote CN lockservice address.
CN startup publishes ingress readiness only after listeners are started and a heartbeat updates HAKeeper/cluster membership.
A DDL path creates the fake auto-increment metadata and immediately requests a remote lock. During the heartbeat gap, the filtered inventory is empty (cns: []), so lockservice sends no usable backend address and returns cannot connect to backend: none.
The physical CN processes are not necessarily absent; the public service snapshot is filtered to zero during the transition.
The mo_table_stats duplicate-entry log in the same window is secondary initialization noise; the failing assertion is the empty lockservice backend.
Impact
Flaky or deterministic failures of embedded multi-CN partition/DDL UTs.
Potential transient CREATE TABLE/metadata-operation failures while a CN is not yet admission-ready.
Lockservice errors are misleading because no backend address is available, rather than a real network refusal.
Start/refresh the cluster while CNs are still admission-pending.
Execute TestAlterTableDropPartition.
Observe target=2-cn-16001, cns=[], and cannot connect to backend: none.
Expected behavior
Public SQL routing must remain fail-closed for non-ready generations, but lockservice backend discovery must have a bounded, readiness-aware path that does not immediately convert a temporary admission transition into an empty backend. Cluster startup/test admission must not expose a usable SQL service before the required lockservice routing state is available.
Acceptance criteria
Add a deterministic regression test covering a lock request during the admission/readiness transition.
Ensure lockservice either waits with a bounded deadline for a ready backend or returns an explicit not-ready error without sending an empty address.
Repeated 3-CN pkg/tests/partition runs, including TestAlterTableDropPartition, pass under normal and race execution.
Preserve generation fencing and fail-closed public routing semantics.
No regression in lockservice cancellation, CN restart, or admission revocation paths.
Workaround
Rerun the failed UT or run the partition package after all CN readiness heartbeats have converged. Serializing the embedded-cluster startup reduces the probability but does not fix the ordering bug.
Problem
The Ubuntu/x86 UT job fails in
pkg/tests/partition::TestAlterTableDropPartitionwhile executingCREATE TABLE, before the partition-drop assertion:Failure evidence: UT job 98235767659. The test fails after 0.40s with
Received unexpected error: cannot connect to backend: none.Root cause hypothesis
This is an admission/readiness ordering bug introduced or exposed by PR #27430, not by the DISTINCT aggregation changes in PR #27693.
clusterservice.GetCNServiceWithoutWorkingStateWithContextnow excludes every CN while view-metadata admission is Preparing/Enabled unlessViewMetadataAdmissionReadyis true.lockserviceuses that API to resolve a remote CN lockservice address.cns: []), so lockservice sends no usable backend address and returnscannot connect to backend: none.The physical CN processes are not necessarily absent; the public service snapshot is filtered to zero during the transition.
The
mo_table_statsduplicate-entry log in the same window is secondary initialization noise; the failing assertion is the empty lockservice backend.Impact
CREATE TABLE/metadata-operation failures while a CN is not yet admission-ready.Reproduction
pkg/tests/partitionUT package with the admission-aware code from feat(view): add durable lifecycle admission fence #27430 and a 3-CN embedded cluster.TestAlterTableDropPartition.target=2-cn-16001,cns=[], andcannot connect to backend: none.Expected behavior
Public SQL routing must remain fail-closed for non-ready generations, but lockservice backend discovery must have a bounded, readiness-aware path that does not immediately convert a temporary admission transition into an empty backend. Cluster startup/test admission must not expose a usable SQL service before the required lockservice routing state is available.
Acceptance criteria
pkg/tests/partitionruns, includingTestAlterTableDropPartition, pass under normal and race execution.Workaround
Rerun the failed UT or run the partition package after all CN readiness heartbeats have converged. Serializing the embedded-cluster startup reduces the probability but does not fix the ordering bug.
Related