distill_fs is a read-only FUSE filesystem for raw disk images and Nydus
RAFS images. The current implementation combines:
- on-demand reads from local or remote storage
- sparse local cache files with bitmap sidecars
- optional chunk-level deduplication backed by LMDB
- an optional local/peer chunk server for chunk sharing
- optional OTLP metrics export
mountis implemented on Linux only. On non-Linux targets, mounting returns an unsupported-operation error.--src localmounts a single raw file from the local filesystem.--src ossmounts a single raw file from a remote backend created from a NydusBackendConfigV2JSON file. Despite the flag name, this path is not OSS-only. With the current build features it can use OSS, S3, registry, and HTTP-proxy backends exposed bynydus-storage.--src nydusmounts a Nydus RAFS v5 filesystem from a bootstrap plus backend config.- Optional dedup uses:
ChunkDB: global content-addressed chunk storageIndexDB: per-image offset-to-checksum metadata
serve-chunkexposes a localChunkDBover both TCP and a Unix socket, and can cooperate with peers plus an optional Redis-backed chunk ownership index.gc-chunkandstats-chunkoperate onChunkDB.
- The filesystem is read-only.
- Raw-image mounts expose exactly one file at the mountpoint root. That file is
named by
--name. - For raw-image mounts,
--namemust be a single path component. The current implementation reuses it as both the mounted filename and the raw-image backend identifier. - For raw-image mounts,
--nameis also used as the dedup metadata identity. Reuse the same--nameonly for the same image namespace/content lineage when sharing animage_meta_dir. --chunk-db-dirand--image-meta-dirmust be configured together or both omitted.- The cache layer creates a sidecar bitmap file next to each cache file:
<cache path>.bitmap. --src localopens the source file read-write. When dedup is enabled, chunks that have been persisted toChunkDBmay be hole-punched from that file. Use a dedicated writable cache copy if you need to preserve the original file byte-for-byte.- In Nydus mode,
--nameis still required by the CLI parser, but it is not used by the mounted filesystem. - In Nydus mode,
--cache-diris optional. If set, blob payloads are cached on disk and wrapped in the same cache/dedup stack. If omitted, blob ranges are fetched directly from the backend, while decompressed chunks can still be served from and stored intoChunkDBwhen dedup is enabled. - The current Nydus read path supports RAFS v5. RAFS v6 images are rejected
with an unsupported-operation error until the read path is moved to Nydus'
BlobDeviceflow. - The current Nydus read path does not support encrypted chunks or batched chunks.
gc-chunk --dry-runonly skips deletion. It does not print a deletion plan.
Prerequisites:
- Rust 1.85.0;
rust-toolchain.tomlinstalls the matching formatter - Linux with
/dev/fuseaccess formount - writable directories for cache files, bitmap sidecars, LMDB data, and Unix sockets
- optional Redis if you want dynamic peer discovery or a Redis-backed chunk index
Build the main binary:
cargo build --bin distill_fsRun the CLI help:
cargo run --bin distill_fs -- --helpGitHub releases provide distill-fs-vX.Y.Z-linux-amd64.tar.gz, distill-fs-vX.Y.Z-linux-arm64.tar.gz, and one SHA256SUMS covering both archives. Each archive contains the distill_fs executable, manifest.json, LICENSE, NOTICE, and the exact Cargo.lock. The executable uses musl and has no dynamic loader or shared-library dependencies. Mounting still requires Linux FUSE support and suitable privileges; remote HTTPS backends still need a CA certificate store on the host.
Download a published release from https://github.com/inclusionAI/distill-fs/releases/download/vX.Y.Z/, verify the archive against a reviewed SHA-256 pin, extract it, and install distill_fs on PATH. Keep the license and provenance files with the binary. Production consumers must pin the release URL and checksum in source control; fetching a checksum alongside an unpinned binary is not an independent integrity check.
Build and test an artifact locally with Docker, selecting amd64 or arm64 to match the target host:
docker build --platform linux/arm64 --target test -f ci/release.Dockerfile .
docker build --platform linux/arm64 -f ci/release.Dockerfile \
--build-arg "SOURCE_REVISION=$(git rev-parse HEAD)" \
--output type=local,dest=dist .
(cd dist && sha256sum -c SHA256SUMS)Use a clean source checkout for release builds. manifest.json records the package version, source commit, musl target, and binary SHA-256. The Docker build checks the ELF for a dynamic interpreter and shared dependencies, then runs the executable in an empty chroot. The test target runs the default test suite against the musl release build; ignored FUSE tests need an explicitly privileged test environment. The pinned build image and Cargo.lock fix the toolchain and Rust dependency graph; Alpine build packages are resolved when the build runs, so independently rebuilding a tag is not a byte-for-byte reproducibility guarantee.
The CI workflow builds and tests pull requests, main, manual runs, and version tags, including real raw and RAFS v5 FUSE mount tests using the checksum-pinned Nydus v2.4.0 tool. Non-tag runs upload reviewable Actions artifacts without publishing a release. Pushing a feature branch alone does not trigger the workflow; opening or updating a pull request does. Manual dispatch becomes available once the workflow file exists on the default branch: use Actions → CI → Run workflow, or gh workflow run release.yml --repo inclusionAI/distill-fs --ref main. To publish, update the package version and lockfile, merge to main, and push a matching vX.Y.Z tag (or vX.Y.Z-rc.N for a prerelease). The workflow rejects version mismatches and tags outside main, then publishes both tested archives without rebuilding in the release job. Existing release assets are never overwritten. Review changes before pushing a release tag.
AKernel selects the architecture-specific archive through its own builder/distill-fs-versions.env and builder/scripts/install-distill-fs.sh. Publish the distill-fs release first, verify its checksums, and update the AKernel pins. No sandboxd pipeline or gitlink change is required. The distill-fs submodule in AKernel is optional source reference and is no longer a build input.
The main binary exposes four subcommands:
| Command | Purpose |
|---|---|
mount |
Mount a raw image or Nydus filesystem |
serve-chunk |
Serve local chunks over TCP and Unix socket, optionally with peer coordination |
gc-chunk |
Garbage-collect old/LRU chunks from ChunkDB |
stats-chunk |
Print lightweight JSON stats for ChunkDB |
Global flags:
-d,--verbose: enable debug logging
Common operational flags:
--daemon: daemonize the process--pid-file: PID file path to use with--daemon--log-file: log file path; if set, logs go through the rotating file writer--otel-endpoint: enable OTLP metrics export
Mount an existing local raw file as a single file in the mountpoint.
Required arguments:
--name: filename exposed inside the mountpoint and dedup metadata key--mountpoint: FUSE mountpoint--cache-file: path to the existing local raw file
Example:
mkdir -p /mnt/raw
cargo run --bin distill_fs -- mount \
--src local \
--name disk.raw \
--cache-file /data/disk.raw \
--mountpoint /mnt/rawMount a remote raw file as a single file in the mountpoint. The implementation
uses GeneralBackend, which loads a Nydus BackendConfigV2 JSON file. The
build enables local filesystem, OSS, S3, registry, and HTTP-proxy backends.
Required arguments:
--name: remote object/blob identifier and mounted filename; it must still be valid as a single filename inside the mountpoint--mountpoint: FUSE mountpoint--cfg: backend config JSON--cache-file: writable local cache file path
Example:
mkdir -p /mnt/raw /var/cache/distill
cargo run --bin distill_fs -- mount \
--src oss \
--name rootfs.raw \
--cfg /etc/distill/backend.json \
--cache-file /var/cache/distill/rootfs.raw \
--mountpoint /mnt/rawMount a Nydus RAFS filesystem from a bootstrap file plus backend config.
Required arguments:
--name: required by the CLI, but ignored by the Nydus mount logic--mountpoint: FUSE mountpoint--bootstrap: Nydus bootstrap path--cfg: backend config JSON
Optional but common:
--cache-dir: writable directory for blob cache files
Example:
mkdir -p /mnt/nydus /var/cache/distill/blobs
cargo run --bin distill_fs -- mount \
--src nydus \
--name nydus-image \
--bootstrap /images/bootstrap.rafs \
--cfg /etc/distill/backend.json \
--cache-dir /var/cache/distill/blobs \
--mountpoint /mnt/nydusDedup is optional for both raw and Nydus mounts. To enable it, configure both:
--chunk-db-dir--image-meta-dir
When enabled:
ChunkDBstores chunk payloads globally by checksumIndexDBstores per-image range metadata- raw-image reads can fall back from cache/backend to chunk reuse
- Nydus reads can serve decompressed chunks from
ChunkDBand asynchronously persist newly read chunks into it
Example with dedup enabled:
mkdir -p /mnt/raw /var/cache/distill /var/lib/distill/chunkdb /var/lib/distill/imagedb
cargo run --bin distill_fs -- mount \
--src oss \
--name rootfs.raw \
--cfg /etc/distill/backend.json \
--cache-file /var/cache/distill/rootfs.raw \
--chunk-db-dir /var/lib/distill/chunkdb \
--image-meta-dir /var/lib/distill/imagedb \
--mountpoint /mnt/rawserve-chunk serves a local ChunkDB in two ways at the same time:
- TCP on
0.0.0.0:<listen-port> - Unix socket for same-host clients
If --chunk-server-sock is omitted, the Unix socket defaults to:
<chunk_db_dir>/chunkserver.sock
Mount processes that use the same chunk_db_dir automatically default to that
socket path too.
There are two peer-discovery modes:
- static peers via
--peer-addrs host:port,host:port - Redis discovery via
--peer-discovery redis://...
If --peer-discovery is set, the current code expects it to be a Redis URL.
Other discovery schemes are rejected.
--chunk-index-url is separate from discovery. If configured, it enables a
Redis-backed chunk ownership index used by the peer client/chunk server.
Example:
mkdir -p /var/lib/distill/chunkdb
cargo run --bin distill_fs -- serve-chunk \
--chunk-db-dir /var/lib/distill/chunkdb \
--listen-port 9876 \
--advertise-addr 10.0.0.11:9876 \
--peer-discovery redis://10.0.0.20:6379/0 \
--chunk-index-url redis://10.0.0.20:6379/1Useful options:
--node-id: explicit node identity; otherwise defaults to$HOSTNAMEordistill-fs-<pid>--chunk-index-ttl: chunk ownership TTL in seconds, default43200--tokio-worker-threads: async runtime worker count, default8
gc-chunk works on ChunkDB only.
Current GC behavior:
- delete chunks whose access time is older than 24 hours
- if LMDB usage is still above 90%, delete least-recently-used chunks in batches until usage falls to 80% or no more candidates exist
Run GC:
cargo run --bin distill_fs -- gc-chunk \
--chunk-db-dir /var/lib/distill/chunkdbUse a non-default local chunk server socket when needed:
cargo run --bin distill_fs -- gc-chunk \
--chunk-db-dir /var/lib/distill/chunkdb \
--chunk-server-sock /run/distill/custom-chunkserver.sockSkip deletions:
cargo run --bin distill_fs -- gc-chunk \
--chunk-db-dir /var/lib/distill/chunkdb \
--dry-runstats-chunk prints lightweight JSON without a full chunk scan.
Example:
cargo run --bin distill_fs -- stats-chunk \
--chunk-db-dir /var/lib/distill/chunkdbRepresentative output:
{
"storage": {
"total_size_bytes": 107374182400,
"used_size_bytes": 52428800,
"free_size_bytes": 107321753600,
"usage_percent": "0.05"
},
"chunks": {
"total_count": 1234
},
"access_time": {
"oldest_epoch_secs": 1707500000,
"newest_epoch_secs": 1707600000
}
}If the database has no access records yet, the access-time fields may be
null.
- Read through the FUSE file exposed at the mountpoint root.
- If dedup metadata exists, try
ChunkDBfirst for covered ranges. - If data is not available in
ChunkDB, read from the cache/backend. - Queue background dedup work for touched 4 MiB chunks.
- Once a chunk has been persisted to
ChunkDB, the cache layer can invalidate that chunk and reclaim space by hole-punching the local cache file.
- Resolve inodes from the RAFS bootstrap.
- For each file chunk, compute the expected checksum from RAFS metadata.
- Try
ChunkDBfirst. - On a miss, fetch the compressed chunk range from the backend or blob cache, then decompress if necessary.
- Return the requested slice to FUSE and enqueue the full uncompressed chunk
for background storage in
ChunkDB.
The fixed chunk size used by the dedup/cache stack is 4 MiB.
This project does not define its own remote-backend JSON schema. The --cfg
file for remote raw and Nydus modes is deserialized directly as Nydus
BackendConfigV2.
In practice this means:
- reuse the backend JSON format you already use with Nydus tooling
- the accepted fields depend on the backend type selected in that JSON
- the available backend types depend on which
nydus-storagefeatures were compiled into this binary
Common commands:
cargo fmt --all -- --check
cargo build --locked --bin distill_fs
cargo run --bin distill_fs -- --help
cargo test --lockedRedis integration tests are behind a feature flag:
DISTILL_FS_TEST_REDIS_URL=redis://127.0.0.1:6379/15 \
cargo test --locked --features redis-integration-tests \
redis_ -- --test-threads=1The database number must be explicit and non-zero. The tests run FLUSHDB
before and after each case, so the URL must point at a disposable test
database.
The mount tests are ignored during the default test run. They require Linux,
/dev/fuse, permission to mount and unmount FUSE filesystems, and either
fusermount3, fusermount, or umount.
Run the raw-image mount test:
cargo test --locked --test mount raw_mount_reads_file -- --ignoredThe Nydus mount test also requires a nydus-image v2.4.0 binary. It builds a
minimal RAFS v5 image against the local filesystem backend before mounting it:
DISTILL_FS_NYDUS_IMAGE=/path/to/nydus-image \
cargo test --locked --test mount nydus_mount_reads_file -- --ignoredThese tests create all images, mountpoints, caches, and backend configuration under temporary directories and clean them up after completion.
Additional benchmark binaries:
cargo run --bin chunkdb_bench -- --help
cargo run --bin peer_bench -- --helpApache-2.0
mount, serve-chunk, gc-chunk, and stats-chunk accept --chunk-db-size, for example --chunk-db-size 64GiB. This sets the shared LMDB map capacity, including chunk and access-index pages; it is not a RAM reservation or a quota for all image caches. Use whole bytes or an integer followed by B, KiB, MiB, GiB, or TiB. Values must be at least 1 MiB, fit the addressable range, and align to the host page size.
Omitting the option preserves the default: 100 GiB for Linux production builds, 512 MiB for unit-test/non-Linux builds. Every process sharing a --chunk-db-dir, including statistics and GC, must receive the same capacity. The caller (such as sandboxd) owns this consistency; distill-fs does not coordinate or enforce capacity agreement. Resizing an existing database is unsupported. To change capacity, stop all users and select a new cache directory.
Pull requests, pushes to main, version-tag pushes, and manual workflow runs execute the CI workflow. The build job uses the pinned Rust musl container pipeline for formatting checks, an optimized Linux/amd64 static build, unit/integration tests, and privileged raw/Nydus FUSE tests. The release profile strips symbols at build time, so tests exercise the stripped executable that is packaged. ELF checks reject dynamic dependencies, symbol tables, and debug sections; empty-root smoke tests verify the executable works without a host runtime. The job uploads the verified archive and checksums.
Only a push of a version tag triggers the release job, after the build and tests pass. Tags must use vX.Y.Z or vX.Y.Z-rc.N, match Cargo.toml, and reference a commit on main. Release candidates are published as prereleases. The release job downloads and verifies the exact tested archive; it does not rebuild the executable or overwrite existing release assets. PRs, branch pushes, and manual runs (including manual runs selecting a tag) never publish a GitHub Release.