Repository navigation
Conversation
XciD
marked this pull request as ready for review
September 29, 2026 20:45
Contributor
POSIX Compliance (pjdfstest) |
Contributor
Benchmark Results |
XciD
added a commit
that referenced
this pull request
Sep 29, 2026
## Problem xet-core parses durations with humantime, which rejects a bare number. It then keeps its own default and logs a WARN under `xet_runtime`, which the default `hf_mount=info` filter hides. Two of the defaults set in `init_tracing` have therefore never been applied: - `HF_XET_CLIENT_READ_TIMEOUT=30`: reads ran with the 300 s default inactivity timeout, not 30 s. - `HF_XET_RECONSTRUCTION_TARGET_BLOCK_COMPLETION_TIME=30`: prefetch targeted 15 minutes of transfer, not 30 s. With `RUST_LOG=info`, v0.13.1 logs `Configuration value 30 for read_timeout cannot be parsed into correct type; reverting to default.` (same for `target_block_completion_time`). ## Fix Both values are now `30s`. The defaults move into `xet_env_defaults()`, and `xet_env_defaults_are_accepted_by_xet_core` passes each one through `XetConfig::with_config` for its `HF_XET_<GROUP>_<FIELD>` path. That rejects a value xet-core cannot parse and a setting it no longer knows (for example after an upstream rename). On main, the test fails and lists exactly these two defaults. The other nine pass. ## Effect With this branch, xet-core logs `read_timeout = 30s (user set)` and `target_block_completion_time = 30s (user set)`. The prefetch target is `rate x target_block_completion_time`, bounded by the 256 MiB download buffer limit, so fast streams see no change and only slow streams prefetch less. Cold sequential reads of a 3.09 GB xet file (`Qwen/Qwen2.5-1.5B-Instruct`, `--no-disk-cache`, m6i.2xlarge, 8 alternating runs each) gave a median of 556 MB/s on v0.13.1 and 558 MB/s with this change, within network noise. This PR and #244 both touch the xet env defaults list, so whichever lands second needs a rebase. The new test will then also check `HF_XET_TELEMETRY_FINAL_FLUSH_TIMEOUT`.
Move the pinned xet-core rev from 0e474005 to 388c93cc (40 commits). - Client::upload_shard takes an optional shard upload progress callback and returns () (v2 shard upload API, xet-core #884). CachedXetClient forwards it to the inner client. - Upload sessions now report transfer telemetry to CAS and await the terminal document in finalize(), up to 2 s by default (xet-core #932 to #937). A synchronous close() would pay that on every write, so HF_XET_TELEMETRY_FINAL_FLUSH_TIMEOUT=0s sends it detached. The usual opt-out variables (HF_HUB_DISABLE_TELEMETRY, DO_NOT_TRACK, HF_XET_TELEMETRY_ENABLED=false) still apply. - Download buffer defaults are now derived from usable memory (xet-core #943). hf-mount still sets the size and limit; only the per-file part, which it does not set, follows the new default. - Also brings sha2 0.11 and the h2, rustls and der security bumps.
XciD
force-pushed
the
chore/bump-xet-core
branch
from
September 30, 2026 06:50
4a57b10 to
64be18f
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Moves the pinned xet-core rev from
0e474005(2026-07-07) to388c93cc(current main, crates at 1.6.0).Changes needed in hf-mount
Client::upload_shardnow takes an optionalShardUploadProgressCallbackand returns()(v2 shard upload API, Stream shard finalization progress via POST /v2/shards xet-core#884).CachedXetClientforwards the callback to the inner client.POST /v1/telemetry, feat(telemetry): runtime config group and pre-shutdown drain hook xet-core#932 to #937), andfinalize()awaits the terminal document for up to 2 s by default. In the default write mode,close()waits for the commit, so every write would pay that round trip.HF_XET_TELEMETRY_FINAL_FLUSH_TIMEOUT=0ssends it detached instead: the daemon outlives the request, so it is still delivered. Measured with this branch on Linux (20 files of 1 KiB, default write mode, bucket in us-east-1): medianclose()259 ms with the upstream 2 s budget, 182 ms detached. Telemetry stays on (per xet-core, the payload has no file names, paths, hashes, repo ids or user ids), and the usual opt-outs still apply (HF_HUB_DISABLE_TELEMETRY,DO_NOT_TRACK,HF_XET_TELEMETRY_ENABLED=false). Reads send nothing: download sessions and streams go throughCachedXetClient, which keeps the defaulttransfer_telemetry()(None).Behavior notes
HF_XET_RECONSTRUCTION_DOWNLOAD_BUFFER_SIZEandHF_XET_RECONSTRUCTION_DOWNLOAD_BUFFER_LIMIT, so only the per-file increment changes: 512 MB before, usable memory / 64 now (clamped to [16 MB, 2 GB]), still under the 256 MiB limit.Found while bumping, not changed here
HF_XET_CLIENT_READ_TIMEOUT=30andHF_XET_RECONSTRUCTION_TARGET_BLOCK_COMPLETION_TIME=30have never been applied: xet-core parses durations with humantime, which rejects a number without a unit, so it falls back to its defaults (300 s and 15 min). The warning is hidden by thehf_mount=infolog filter (seen withRUST_LOG=infoon v0.13.1). Writing30swould activate two settings that never ran, so that belongs in a separate change.