Skip to content

fix(setup): give xet-core duration defaults a unit so they apply - #245

Merged
XciD merged 1 commit into
mainfrom
fix/xet-env-duration-units
Sep 29, 2026
Merged

XciD merged 1 commit into
mainfrom
fix/xet-env-duration-units

Conversation

@XciD

@XciD XciD commented Sep 29, 2026

Copy link
Copy Markdown
Member

Problem

xet-core parses durations with humantime, which rejects a bare number. It then keeps its own default and logs a WARN under xet_runtime, which the default hf_mount=info filter hides. Two of the defaults set in init_tracing have therefore never been applied:

  • HF_XET_CLIENT_READ_TIMEOUT=30: reads ran with the 300 s default inactivity timeout, not 30 s.
  • HF_XET_RECONSTRUCTION_TARGET_BLOCK_COMPLETION_TIME=30: prefetch targeted 15 minutes of transfer, not 30 s.

With RUST_LOG=info, v0.13.1 logs Configuration value 30 for read_timeout cannot be parsed into correct type; reverting to default. (same for target_block_completion_time).

Fix

Both values are now 30s. The defaults move into xet_env_defaults(), and xet_env_defaults_are_accepted_by_xet_core passes each one through XetConfig::with_config for its HF_XET_<GROUP>_<FIELD> path. That rejects a value xet-core cannot parse and a setting it no longer knows (for example after an upstream rename). On main, the test fails and lists exactly these two defaults. The other nine pass.

Effect

With this branch, xet-core logs read_timeout = 30s (user set) and target_block_completion_time = 30s (user set). The prefetch target is rate x target_block_completion_time, bounded by the 256 MiB download buffer limit, so fast streams see no change and only slow streams prefetch less. Cold sequential reads of a 3.09 GB xet file (Qwen/Qwen2.5-1.5B-Instruct, --no-disk-cache, m6i.2xlarge, 8 alternating runs each) gave a median of 556 MB/s on v0.13.1 and 558 MB/s with this change, within network noise.

This PR and #244 both touch the xet env defaults list, so whichever lands second needs a rebase. The new test will then also check HF_XET_TELEMETRY_FINAL_FLUSH_TIMEOUT.

xet-core parses durations with humantime, which rejects a bare number,
and then keeps its own default with a WARN that the hf_mount=info filter
hides. HF_XET_CLIENT_READ_TIMEOUT=30 and
HF_XET_RECONSTRUCTION_TARGET_BLOCK_COMPLETION_TIME=30 were therefore never
applied: reads ran with a 300 s inactivity timeout instead of 30 s, and
prefetch targeted 15 minutes of transfer instead of 30 s. Use 30s.

The defaults move into xet_env_defaults() so a unit test can pass each one
through XetConfig::with_config, which rejects a value xet-core cannot
parse and a setting it no longer knows.
@XciD
XciD marked this pull request as ready for review September 29, 2026 21:00
@github-actions

Copy link
Copy Markdown
Contributor

Benchmark Results

============================================================
  Benchmark — 50MB
------------------------------------------------------------
  Metric                                 FUSE          NFS
  ------------------------------ ------------ ------------
  Sequential read                    262.6 MB/s     147.3 MB/s
  Sequential re-read                2079.3 MB/s    2248.1 MB/s
  Range read (1MB@25MB)                0.2 ms         0.2 ms
  Random reads (100x4KB avg)           0.0 ms         0.0 ms
  Sequential write (FUSE)           1236.6 MB/s
  Close latency (CAS+Hub)            0.092 s
  Write end-to-end                   378.4 MB/s
  Dedup write                       1404.6 MB/s
  Dedup close latency                0.121 s
  Dedup end-to-end                   320.0 MB/s
============================================================
============================================================
  Benchmark — 200MB
------------------------------------------------------------
  Metric                                 FUSE          NFS
  ------------------------------ ------------ ------------
  Sequential read                    593.7 MB/s     514.0 MB/s
  Sequential re-read                2141.2 MB/s    2324.1 MB/s
  Range read (1MB@25MB)                0.2 ms         0.2 ms
  Random reads (100x4KB avg)           0.0 ms         0.0 ms
  Sequential write (FUSE)           1371.0 MB/s
  Close latency (CAS+Hub)            0.125 s
  Write end-to-end                   738.2 MB/s
  Dedup write                       1420.0 MB/s
  Dedup close latency                0.101 s
  Dedup end-to-end                   828.3 MB/s
============================================================
============================================================
  Benchmark — 500MB
------------------------------------------------------------
  Metric                                 FUSE          NFS
  ------------------------------ ------------ ------------
  Sequential read                   1020.1 MB/s     981.1 MB/s
  Sequential re-read                2089.7 MB/s    2257.6 MB/s
  Range read (1MB@25MB)                0.2 ms         0.2 ms
  Random reads (100x4KB avg)           0.0 ms         0.0 ms
  Sequential write (FUSE)           1249.5 MB/s
  Close latency (CAS+Hub)            0.116 s
  Write end-to-end                   969.3 MB/s
  Dedup write                       1403.8 MB/s
  Dedup close latency                0.106 s
  Dedup end-to-end                  1080.9 MB/s
============================================================
============================================================
  fio Benchmark Results
------------------------------------------------------------
  Job                        FUSE MB/s   NFS MB/s  FUSE IOPS   NFS IOPS
  ------------------------- ---------- ---------- ---------- ----------
  seq-read-100M                  359.7      395.3                      
  seq-reread-100M               2439.0       20.0                      
  rand-read-4k-100M                0.1        0.1         15         20
  seq-read-5x10M                 632.9      724.6                      
  rand-read-10x1M                  0.1        0.1         35         37
  Random Read Latency           FUSE avg      NFS avg
  ------------------------- ------------ ------------
  rand-read-4k-100M           68507.4 us   51004.3 us
  rand-read-10x1M             28528.4 us   27033.7 us
============================================================

@github-actions

Copy link
Copy Markdown
Contributor

POSIX Compliance (pjdfstest)

============================================================
  pjdfstest POSIX Compliance Results
------------------------------------------------------------
  Files: 130/130 passed    Tests: 832 total (0 subtests failed)
  Result: PASS
------------------------------------------------------------
  Category               Passed    Total   Status
  -------------------- -------- -------- --------
  chflags                     5        5       OK
  chmod                       8        8       OK
  chown                       6        6       OK
  ftruncate                  13       13       OK
  granular                    5        5       OK
  mkdir                       9        9       OK
  open                       19       19       OK
  posix_fallocate             1        1       OK
  rename                     10       10       OK
  rmdir                      11       11       OK
  symlink                    10       10       OK
  truncate                   13       13       OK
  unlink                     11       11       OK
  utimensat                   9        9       OK
============================================================

@XciD
XciD merged commit bf73fe6 into main Sep 29, 2026
6 checks passed
@XciD
XciD deleted the fix/xet-env-duration-units branch September 29, 2026 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant