Skip to content

[Bug]: admission-ready CN filtering can starve lockservice backend discovery during startup #27699

Description

@XuPeng-SH

Problem

The Ubuntu/x86 UT job fails in pkg/tests/partition::TestAlterTableDropPartition while executing CREATE TABLE, before the partition-drop assertion:

ERROR cannot find lockservice address
target: 2-cn-16001
cns: []
ERROR failed to lock on remote
error: cannot connect to backend: none

Failure evidence: UT job 98235767659. The test fails after 0.40s with Received unexpected error: cannot connect to backend: none.

Root cause hypothesis

This is an admission/readiness ordering bug introduced or exposed by PR #27430, not by the DISTINCT aggregation changes in PR #27693.

  1. clusterservice.GetCNServiceWithoutWorkingStateWithContext now excludes every CN while view-metadata admission is Preparing/Enabled unless ViewMetadataAdmissionReady is true.
  2. lockservice uses that API to resolve a remote CN lockservice address.
  3. CN startup publishes ingress readiness only after listeners are started and a heartbeat updates HAKeeper/cluster membership.
  4. A DDL path creates the fake auto-increment metadata and immediately requests a remote lock. During the heartbeat gap, the filtered inventory is empty (cns: []), so lockservice sends no usable backend address and returns cannot connect to backend: none.

The physical CN processes are not necessarily absent; the public service snapshot is filtered to zero during the transition.

The mo_table_stats duplicate-entry log in the same window is secondary initialization noise; the failing assertion is the empty lockservice backend.

Impact

  • Flaky or deterministic failures of embedded multi-CN partition/DDL UTs.
  • Potential transient CREATE TABLE/metadata-operation failures while a CN is not yet admission-ready.
  • Lockservice errors are misleading because no backend address is available, rather than a real network refusal.

Reproduction

  1. Run the pkg/tests/partition UT package with the admission-aware code from feat(view): add durable lifecycle admission fence #27430 and a 3-CN embedded cluster.
  2. Start/refresh the cluster while CNs are still admission-pending.
  3. Execute TestAlterTableDropPartition.
  4. Observe target=2-cn-16001, cns=[], and cannot connect to backend: none.

Expected behavior

Public SQL routing must remain fail-closed for non-ready generations, but lockservice backend discovery must have a bounded, readiness-aware path that does not immediately convert a temporary admission transition into an empty backend. Cluster startup/test admission must not expose a usable SQL service before the required lockservice routing state is available.

Acceptance criteria

  • Add a deterministic regression test covering a lock request during the admission/readiness transition.
  • Ensure lockservice either waits with a bounded deadline for a ready backend or returns an explicit not-ready error without sending an empty address.
  • Repeated 3-CN pkg/tests/partition runs, including TestAlterTableDropPartition, pass under normal and race execution.
  • Preserve generation fencing and fail-closed public routing semantics.
  • No regression in lockservice cancellation, CN restart, or admission revocation paths.

Workaround

Rerun the failed UT or run the partition package after all CN readiness heartbeats have converged. Serializing the embedded-cluster startup reduces the probability but does not fix the ordering bug.

Related

  • PR #27430 — admission-aware CN inventory filtering.
  • Issue #25948 — older, different lockservice address-registration test flake.
  • Issue #26704 — related multi-CN partition cron-task initialization race.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions