fix(snapshot): make payload integrity opt-in - #1346
Merged
Merged
Conversation
Honor the existing integrity opt-in and remove implicit payload verification from ordinary create, archive, boot, and descriptor migration paths. Add a sparse-aware fixed-leaf BLAKE3 Merkle representation for explicit verification while preserving released SHA descriptors and adjacent downgrade compatibility.
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Workflows to automatically generate PRs for you. |
Bind archive members to one-pass BLAKE3 digests so payload corruption is rejected even when persistent snapshot integrity is not recorded. Keep released inventories readable without another payload scan. Verify recorded Merkle roots before downgrade projection and pin explicit verification to a stable file identity. Preserve complete Merkle state in Node and Go while retaining digest compatibility aliases. Closes #1220
Resolve the documentation overlap ahead of the upstream merge. Keep main’s streamlined SDK reference structure while preserving optional-integrity behavior.
Temporarily align the remaining overlapping sentence with main so GitHub can perform a signed branch update. The corrected optional-integrity wording is restored after the merge.
Document that archive loading performs structural and entry checks while preserving recorded payload integrity for an explicit verification request.
Open the durability handle with write access before FlushFileBuffers and release it before atomic replacement. Heap-pin buffer-heavy archive entry and staged-cache futures so snapshot save/load fits Windows worker stacks.
Gate the Unix-only runtime endpoint helper import to Unix test builds and initialize the bind-mount symlink policy in the Windows path fixture. This restores compilation of the full Rust unit-test target on Windows.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
Makes snapshot payload integrity genuinely opt-in and removes the expensive hidden payload scans reported in superradcompany/microsandbox#1220. Ordinary create, open, boot, save, load, and descriptor-upgrade paths remain structural; explicit integrity recording uses a sparse-aware BLAKE3 Merkle tree, while released SHA descriptors keep their exact readers and verifiers.
Description
record_integrity/--integrityoption. New snapshots writeintegrity: nullby default and do not perform an independent content pass.msb-file-merkle-blake3-v1, with fixed 64 KiB leaves, domain-separated leaf/parent/root hashes, deterministic zero padding, and logical size bound into the root.msb-sparse-sha256-v1descriptors. Explicit verification continues to execute their original algorithms.not recordedverification result consistently through the CLI and Rust, TypeScript, Python, and Go bindings.Sparse Merkle BLAKE3 benchmark
A conservative pre-integration prototype was measured against 64 GiB APFS sparse files on an Apple M5 Max (18 cores, 36 GB RAM), using 64 KiB leaves, BLAKE3 1.8.6, Rayon with 18 workers, Microsandbox's
ExtentMap, and release-mode Rust. File creation is excluded; each image was hashed twice.Large allocated images sustained approximately 5.4–5.5 GiB/s. Extent discovery remained below 1 ms for 819 extents and was approximately 4.11 ms for 4,096 extents, so runtime scales with allocated content rather than virtual image size or ordinary fragmentation. The production implementation further batches reads and reuses buffers; these prototype numbers are intentionally not presented as a final performance ceiling.
Test Plan
cargo fmt --all -- --checkcargo check -p microsandbox-image -p microsandbox -p microsandbox-cli -p microsandbox-node -p microsandbox-py -p microsandbox-gocargo test -p microsandbox-image(203 passed, 3 ignored, including doc tests)cargo test -p microsandbox --no-default-features --lib snapshot::(18 passed)cargo test -p microsandbox --no-default-features --test snapshot_artifactwith isolatedMSB_HOME(41 passed)cargo clippy -p microsandbox-image --lib -- -D warningscargo clippy -p microsandbox --lib -- -D warningscargo test -p microsandbox --lib snapshot:: -- --test-threads=1(19 passed)cargo test -p microsandbox --test snapshot_artifact -- --test-threads=1(40 passed with the normal thread stack)cargo clippy -p microsandbox --lib -- -D warningsCloses #1220
Greptile Summary
The PR makes snapshot payload integrity opt-in while retaining mandatory structural and archive-member validation.
Confidence Score: 5/5
The PR appears safe to merge because no blocking failure remains within the eligible follow-up-review scope.
No blocking failure remains.
Important Files Changed
Sequence Diagram
sequenceDiagram participant User participant Snapshot participant Archive participant Payload User->>Snapshot: Create (integrity optional) alt Integrity requested Snapshot->>Payload: Compute sparse BLAKE3 Merkle root Snapshot-->>User: Descriptor with recorded integrity else Default Snapshot-->>User: Descriptor with integrity: null end User->>Archive: Save or load Archive->>Archive: Validate grammar, confinement, sizes, identities, and member transport Archive-->>User: Preserve recorded payload integrity opt Explicit verify User->>Snapshot: verify() Snapshot->>Payload: Execute recorded integrity algorithm Snapshot-->>User: Verified or mismatch endReviews (3): Last reviewed commit: "Merge branch 'main' into appcypher/optio..." | Re-trigger Greptile