Cross-node deploy-transaction ordering: config write + install + replication not serialized across concurrent same-component deploys

[kriszyp]

Summary

deployComponent (components/operations.js) writes the component's YAML config, then extracts/installs, then replicates — but nothing serializes that whole per-component sequence across concurrent deploys of the same component. PR #1991 (#1973) added a filesystem lock that closes the node_modules corruption window during extract/install, but the wider per-component transaction (config write → install → replicate) can still interleave across two concurrent deploys, leaving disk, config, and peer state disagreeing with each other.

Repro scenario

1. Deploy A writes applicationConfig A via configUtils.addConfig (components/operations.js:416), then pauses (e.g. in recorder/payload/credential work) before reaching prepareApplication.
2. Deploy B writes applicationConfig B, acquires the withComponentPreparationLock mutex (components/Application.ts:1070), and installs B.
3. Deploy A then acquires the lock and installs A.
4. Both calls succeed. Component A ends up on disk, but the YAML config on disk is B's. Replication to peers (server.replication.replicateOperation, components/operations.js:613) then propagates whichever order the two deploys happened to replicate in, which is not guaranteed to match the local disk/config order — peers can end up with a third, different final config/version than the origin.

Why the existing lock doesn't cover this

withComponentPreparationLock in #1991 intentionally scopes serialization to the destructive extract/install filesystem transaction only (the corruption class that caused #1973). It does not cover the config write (which happens before the lock is acquired) or replication (which happens after the lock is released).

Naively widening the lock to span replicateOperation was considered and rejected: extending the local filesystem mutex across replication would introduce a distributed deadlock when two nodes originate a same-component deploy concurrently — each origin would hold its local mutex while waiting for the other node to apply the replicated operation.

What's needed

An explicit cross-node deployment generation/ordering protocol — not a wider local critical section. Candidates to evaluate:

• An ordered commit/version protocol for per-component deploys (e.g. a monotonic deployment generation number that config writes, installs, and replication all key off, with later generations winning and earlier ones detecting/aborting on conflict).
• This may be substantially covered by the hdb_deployment tracking work in Deployment tracking, replicated payload delivery, and rollback #641 (replicated deployment records with ordering/rollback) — worth confirming whether that design's generation/ordering story closes this gap, or whether it needs its own explicit treatment.

Suggested test coverage for a fix

A two-package (or same-package, two concurrent deploy) test that asserts disk contents, on-disk config, and peer-replicated config/version all agree on the same final state after two overlapping deploys.

Provenance

Surfaced during review of #1991 (in-thread discussion on components/Application.ts); confirmed as real and well-characterized by @Ethan-Arrowood's approval on that PR, with a request to track it separately rather than let it evaporate. Deliberately out of scope for #1991, which fixes only the narrower node_modules corruption race from #1973.