feat(mls/database): add KpLiveness Task variant - #342
Merged
Conversation
Adds a KpLiveness variant to the Task oneof: a recurring singleton that verifies this installation still has a usable key package published on the network, and queues a rotation when it does not. Rotation (KpRotation) is driven purely by a local deadline column. If that column is ever wrong, the client silently stops rotating, its published key package expires, and it becomes permanently unreachable: anyone adding it records it as a failed installation, so it receives no welcome and therefore never gets the welcome-driven rotation nudge either. Nothing in that loop produces a local error. KpLiveness is the independent watchdog — separate schedule, separate retry/backoff, and a network probe rather than a local deadline as its source of truth. Consumed by libxmtp (xmtp/libxmtp). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
Contributor
ApprovabilityVerdict: Approved 913a1d2 Additive protobuf schema change that introduces a new empty Task variant (KpLiveness) with no impact on existing behavior. The author is the designated code owner of this file, and the change is backwards-compatible with comprehensive documentation. You can customize Macroscope's approvability policy. Learn more. |
|
🎉 This PR is included in version 3.95.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
insipx
added a commit
to xmtp/libxmtp
that referenced
this pull request
Aug 7, 2026
Key-package rotation is driven purely by a local deadline column (`identity.next_key_package_rotation_ns`). Nothing ever verified that a usable key package is actually published. When that column is wrong the failure is silent and terminal: - the published key package expires and is never rotated; - anyone adding the installation records it in `failed_installations` instead of emitting an Add proposal, so it is a group member with no MLS leaf; - with no leaf it receives no welcome, so the existing welcome-driven rotation nudge never fires either; - the client sees no local error. It syncs, sends messages, and creates groups. Closed loop, no exit, no logs. A dev-network installation sat unreachable for weeks this way, and its only symptom was a missing INFO line. Add `KpLiveness`: a recurring TaskRunner singleton that probes the network for this installation's own key package and queues a rotation when it is absent, unverifiable, or close to expiry. It is seeded next to `KpRotation` and `KpDeletion`, so the first client build after upgrade checks at once. That is what rescues installations that are already broken. The watchdog is deliberately a separate task, not a branch inside `KpRotation`. It needs its own schedule (daily, against rotation's 30 days), its own retry budget (a network probe fails often; rotation must not share its backoff), and its own pull-in target. A watchdog that shares a schedule and a retry budget with its subject fails with its subject. Details the design gets right, and the reasons: - "Absent" has two wire shapes. A short response is `MismatchedKeyPackages`, but the network actually returns a present-but-empty payload. Both mean absent. - The response is matched to the request by position, so a backend defect can return a valid key package for another installation. The probe compares the leaf signature key, and reports `Inconclusive`, because an upload cannot repair a mapping defect and rotating would churn the whole fleet. - Only "asked for 1, got 0" proves absence. Any other count is a backend defect. - The throttle column is never trusted blindly. A future stamp, from clock skew or corruption, would disable the watchdog for months. That is the same failure this check exists to catch. - Startup reconciliation pulls the task in unconditionally, because a clock jump can also strand the task row where the handler's clamp cannot reach it. - Probe failures return a deadline, never `Err`, so an offline client does not drive the task into the TaskRunner's backoff. Also log the rotation deadline at INFO on both branches. A silent "not due" branch is what hid the incident. Requires xmtp/proto#342 (merged as f53e21af), which `proto_version` now pins. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a
KpLivenessvariant to theTaskoneof inproto/mls/database/task.proto(field 8), plus an emptyKpLivenessmessage.Why
libxmtp's key-package rotation (
KpRotation) is driven purely by a local deadline column. Nothing ever verifies that the client actually has a usable key package published on the network.When that column is wrong, the failure is silent and terminal:
Addproposal is produced and no welcome is ever delivered;Closed loop, no exit, no logs. We recently spent hours diagnosing exactly this on a dev-network installation that had been unreachable for ~2 weeks.
KpLivenessis the independent watchdog for that loop: it probes the network for its own key package on its own throttle and queues a rotation when the key package is absent, unverifiable, or close to expiry.Why a distinct variant instead of folding it into
KpRotationThe alternative — doing the liveness probe inside the existing
KpRotationtask arm — was evaluated and rejected:KpRotation's deadline followsnext_key_package_rotation_ns(~30 days). A liveness check needs a ~24h cadence, so the two would have to be multiplexed onto one row viamin()— the watchdog's schedule would then be derived from the same column whose corruption it exists to detect.PullInDeadlinetargets a task bydata_hash. With one row there is no way to express "run the liveness check now" as distinct from "rotate now".A separate variant keeps the watchdog structurally independent of the thing it watches, which is the whole point.
Compatibility
Additive: new oneof field number, no existing field changed or removed. Older clients decode it as an unknown oneof variant and skip the task, which is the existing behavior for any unrecognized
Task(Nonearm deletes the row).Downstream
Consumed by a companion libxmtp PR, which is blocked on this landing so
crates/xmtp_protocan be regenerated againstmain.🤖 Generated with Claude Code
Note
Add
KpLivenesstask variant to MLS database protoAdds a new
KpLivenessmessage type and a correspondingkp_livenessoneof variant (field 8) to theTaskmessage in task.proto.KpLivenessis an empty message serving as a recurring singleton watchdog task.Macroscope summarized 913a1d2.