doc: add centralized chassis aggregate ACK HLD

[amitg90]

Summary

This PR adds a High-Level Design (HLD) document for the aggregate ACK mechanism on a SONiC centralized chassis — where multiple line-card consumers (for example Route Orch on each NPU) must each program a system object, but the control-plane producer on the Supervisor needs one authoritative ACK per object.

The document is intended for community review as part of the centralized chassis design effort, complementing the routing HLD.

What this HLD covers

Problem and solution

• Premature ACK problem — per-consumer ACKs on a multi-NPU chassis can complete before all NPUs finish programming
• Aggregate ACK — vendor-agnostic mechanism in sonic-swss-common that combines consumer ACKs into a single status for fpmsyncd (routes) or other ACK consumers (for example vrfmgrd)

Architecture

• Supervisor vs line card — AckProducerTracker on the Supervisor; Route Orch as ACK producer on each line card
• Non-centralized ACK flow (Figure 2) — existing fire-and-forget publish plus APPL_DB_ROUTE_TABLE_RESPONSE_CHANNEL status path on a single CPU
• Aggregate flow (Figure 1) — PST set/del → ACK_TABLE snapshot → per-producer ACK → single aggregated ACK to fpmsyncd

Acknowledge management

• Line-card boot and producer_id registration via Tracking Service API
• Service restart and line-card removal — master ACK_PRODUCER_TABLE:0 bitlist updates and cleanup

Database schema

• ACK_TABLE, ACK_PRODUCER_TABLE:0, ACK_STATUS_TABLE, WAITING_PRODUCER (when used)
• ROUTE_TABLE timestamp matching to prevent stale ACK false positives

Operational flows

• Four producer-loss cleanup approaches (full walk, reverse map, hybrid, producer-scoped keys) with scenario comparison
• Race conditions during ACK_TABLE bit cleanup
• System object delete trigger (for example VRF coordinated delete)

Future work

• Route ACK aggregation at scale — comparison and recommendation (Approach 3 hybrid as leading candidate)

Files added

Path	Description
doc/centralized-chassis/centralized_chassis_ack_hld.md	Main HLD document
images/centralized-chassis/ack/*	Block diagram, baseline flow, producer-loss walkthroughs, and supporting figures

Reference documents

Document	Link
Central Chassis Architecture	voq_chassis_hld.md
Centralized chassis routing HLD	PR #2382
VOQ architecture	architecture.md
VOQ HLD	voq_hld.md

Test plan

• Review HLD scope, terminology (Supervisor / line card), and Redis schemas against routing HLD and VOQ docs
• Verify diagram links and image assets render correctly
• Confirm cross-references to related HLDs are accurate
• Community review in routing / centralized chassis working group

[mssonicbld]

/azp run

[azure-pipelines]

No pipelines are associated with this pull request.

[mssonicbld]

/azp run

[azure-pipelines]

No pipelines are associated with this pull request.

[mssonicbld]

/azp run

[azure-pipelines]

No pipelines are associated with this pull request.

[mssonicbld]

/azp run

[azure-pipelines]

No pipelines are associated with this pull request.

[mssonicbld]

/azp run

[azure-pipelines]

No pipelines are associated with this pull request.

[mssonicbld]

/azp run

[azure-pipelines]

No pipelines are associated with this pull request.

[mssonicbld]

/azp run

[azure-pipelines]

No pipelines are associated with this pull request.

[eddieruan-alibaba]

1. Would you put ACK in APPL_DB or in APP_STATE_DB ?
2. For "Non-centralized ACK flow", the response is via APP_STATE_DB, can you fix the diagram?
3. In centralized chassis case, can you list out which DB you are using, since you will use Multi DB instance due to scale consideration. Also, can you explicit draw all redis in supervisor to make it clear on hardware carrier for these redis. The main concern I have is how much extra DB bandwidth you would use for this ACK system, and what the performance impact would be.
4. Do you expect none chassis system use this system or not? Can you make it clear in this doc?
   5 "ACK processing compares the supplied timestamp against the value stored with the ROUTE_TABLE snapshot for that route. An ACK is accepted only when it matches the latest update — stale ACKs from superseded programming attempts are ignored." where does this step get handled, lua script or application in supervisor?

[eddieruan-alibaba]

5. what's the main difference between approach 3 and approach 4.
6. During routes downloading, initially we have A, B, one LC (B) OIR and come back, how do you plan to redownload routes to LC B in approach 1, 2 and 3? For approach 4, it would be clean. You may want to add some explaination on how you plan to handle reinstall, who triggers it.
7. Why do you use redis for ack system, why not make a client in supervior to handle it in in memory ? I am worry about redis db bandwidth impact to route downloading performance.

[amitg90]

1. Would you put ACK in APPL_DB or in APP_STATE_DB ?
2. For "Non-centralized ACK flow", the response is via APP_STATE_DB, can you fix the diagram?
3. In centralized chassis case, can you list out which DB you are using, since you will use Multi DB instance due to scale consideration. Also, can you explicit draw all redis in supervisor to make it clear on hardware carrier for these redis. The main concern I have is how much extra DB bandwidth you would use for this ACK system, and what the performance impact would be.
4. Do you expect none chassis system use this system or not? Can you make it clear in this doc?
   5 "ACK processing compares the supplied timestamp against the value stored with the ROUTE_TABLE snapshot for that route. An ACK is accepted only when it matches the latest update — stale ACKs from superseded programming attempts are ignored." where does this step get handled, lua script or application in supervisor?

Thanks @eddieruan-alibaba for your comments, Please see my response. I will post another update on this PR to take care of your suggestions.

#1 ACK tracking APPL_DB on Supervisor

Final route status to fpmsyncd → **unchanged** APPL_STATE_DB

#2 Agreed,

i will fix the diagram (BTW, This HLD is not proposing any changes to how response is sent back to fpmsyncd, but yes given digram needs to be fixed anyway).

#3 For ACK specifically, We are only using APPL_DB which will run on RP / Supervisor.

 Extra bandwidth / performance
Yes, there is additional Redis operations 

-> PST set/del side (Producer side), existing Lua script will be extended to also create entry in ACK_TABLE.
-> on consumer side new lua script added to clear the bit for each route (ACK) from each ACK producer.

#4 Sure i will make it clear, Our goal is make it generic across all platforms, BUT for now it is only for centralized chassis based platforms.

#5 This step of handling timestamp mismatch is handled in the lua script (for atomicity)
RouteOrch on the LC invokes it, but the script executes on Supervisor APPL_DB Redis where ACK_TABLE resides.

[amitg90]

5. what's the main difference between approach 3 and approach 4.
6. During routes downloading, initially we have A, B, one LC (B) OIR and come back, how do you plan to redownload routes to LC B in approach 1, 2 and 3? For approach 4, it would be clean. You may want to add some explaination on how you plan to handle reinstall, who triggers it.
7. Why do you use redis for ack system, why not make a client in supervior to handle it in in memory ? I am worry about redis db bandwidth impact to route downloading performance.

#5 Approach 3 vs 4 — main difference is data model and cleanup path:

Approach 3: per-route ACK_TABLE:<sys_obj_key> (producer_tracking_list bitlist)
           + WAITING_PRODUCER:<producer_id> (reverse map producer → routes).
           Normal ACK: clear bit + update waiting list.
           Producer loss: hybrid — walk waiting list(s) or full ACK_TABLE walk (cost-based).
Approach 4: single structure — ACK_TABLE:<producer_id>|<sys_obj_key> (one key per producer×route).
           Normal ACK: O(1) delete of that producer’s key.
           Producer loss: prefix scan ACK_TABLE:<lost_id>|* (no WAITING_PRODUCER).
Approach 3 optimizes mixed producer-loss patterns; Approach 4 simplifies per-ACK
lookup/delete but uses more keys (producers × in-flight routes).

**Main difference**
**Approach 3** keeps a shared per-route bitlist plus a reverse index and picks the cheapest cleanup path at loss time. **Approach 4** denormalizes into per-(producer, route) keys so normal ACK and producer teardown are direct key delete/scan operations, at the cost of more keys in Redis.

#6

**On OIR:** 
AckProducerTracker clears B from ACK_PRODUCER_TABLE:0 and runs producer-loss cleanup (approach-specific). In-flight ACK state for B is removed; fpmsyncd is notified when B is no longer an outstanding dependency.

**On re-insert:** 
LC boot → RouteOrch re-registers → B’s bit restored in master bitlist.

Route reinstall is triggered by LC/swss startup (RouteOrch sync from central
APPL_DB ROUTE_TABLE / ZMQ resync per routing HLD), not by ACK.

New ACK tracking applies only to routes published via PST set/del after B is back
(snapshot includes B). fpmsyncd does not re-ACK routes already completed for the
prior producer set.

Approach 4 is “cleaner” on OIR because scoped keys ACK_TABLE:B|* are removed directly; but if multiple bits are reset (OIR case) i have scan the table 3 times.and then walk those entry for bit cleanup
in Option 3 i can use smart algorithm to decide if i should walk ACK_TABLE (once and reset all 3 bits in 1 shot) or use reverse map table (WAITING_PRODUCER) Table
I will add a sequence diagram for this.

#7 Redis vs in-memory on Supervisor:

Primary reasons for Redis:
- ACK snapshot must be created atomically with PST set/del in the same APPL_DB Lua script.
- LC AckProducers submit ACKs to central APPL_DB; Redis already provides the shared,
  recoverable coordination layer.
- Crash/restart recovery and future debuggability (per-route ACK status, failures).

In-memory would need a new RPC + persistence model and would break atomic PST+ACK updates.

Bandwidth impact is real but bounded: ~one extra Lua op per route per NPU on steady path;
producer-loss cleanup is the scale concern (§8.2/§13). Mitigations: transient keys,
failure-only ACK_STATUS_TABLE (and we can avoid creating this also if not needed), TTL/hybrid cleanup.