fix: route shard saves through durable_write for crash safety

CHANGELOG.md

@@ -7,6 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- Shard rotation, `save()`, and `compact()` bypass the durable write path — all three called
+    `usearch.index.Index.save(path)` directly (no temp file, no fsync, no atomic rename). All
+    writable shard saves now go through buffer serialization + `durable_write`
+    ([#26](https://github.com/iscc/iscc-usearch/issues/26))
+
+### Changed
+
+- Persistence ordering during `save()` and shard rotation is now bloom → shard → tombstones.
+    Tombstone removals only become visible after the shard data they depend on is durable,
+    preventing previously deleted keys from reappearing after a crash
+- Missing or corrupt bloom filter files are automatically rebuilt from shard keys on load
+    instead of silently disabling bloom lookups
+
 ## [0.7.0] - 2026-05-06
 
 ### Added

docs/explanation/sharding-design.md

@@ -38,11 +38,14 @@ stateDiagram-v2
 
 When the active shard exceeds the configured `shard_size`, it is:
 
-1. Saved to disk as `shard_NNN.usearch`.
-1. Reopened in view mode (memory-mapped, read-only).
+1. Bloom filter and shard file saved durably (buffer → fdatasync → atomic rename).
+1. Tombstones persisted after the shard is durable.
+1. Shard reopened in view mode (memory-mapped, read-only).
 1. Replaced by a fresh, empty active shard.
 
-This rotation resets the HNSW insert curve and keeps throughput consistent.
+This rotation resets the HNSW insert curve and keeps throughput consistent. The
+bloom → shard → tombstones ordering ensures that tombstone removals only become visible after
+the shard data they depend on is safely on disk.
 
 ## Bloom filter integration
 
@@ -67,7 +70,8 @@ The bloom filter is:
 
 - Persisted alongside shard files as `bloom.isbf`.
 - Updated automatically when vectors are added.
-- Rebuilt via `rebuild_bloom()` if corrupted or missing.
+- Rebuilt automatically on load if the file is missing or corrupt.
+- Rebuilt manually via `rebuild_bloom()` if needed.
 
 ## Search fan-out
 

docs/howto/persistence.md

@@ -18,7 +18,7 @@ flushes to stable storage with `fdatasync`, then atomically renames the temp fil
 The result is both atomic (no partial files on crash) and durable (data survives power loss).
 
 Sharded indexes use the same durable file-save path for `.usearch` shard files. `ShardedIndex.save()`
-does not accept a path argument; it saves the active shard, bloom filter, and tombstones into the
+does not accept a path argument; it saves the bloom filter, active shard, and tombstones into the
 directory configured at construction time:
 
 ```python
@@ -28,6 +28,30 @@ index.save()
 Long-running sharded saves log start and completion messages at `INFO`, including the shard name
 and vector count.
 
+### Persistence ordering
+
+Sharded saves (both explicit `save()` and automatic shard rotation) persist files in this order:
+
+1. **Bloom filter** (`bloom.isbf`) — extra entries are harmless false positives.
+1. **Shard file** (`shard_NNN.usearch`) — the actual vector data.
+1. **Tombstones** (`tombstones.npy`) — tombstone removals only become visible after the shard
+    data they depend on is durable.
+
+This ordering prevents previously deleted keys from reappearing after a crash. If the process
+dies after writing the shard but before updating tombstones, the stale tombstone entries just
+hide the key from view shards — but the key is safely in the shard.
+
+### Crash recovery
+
+On load, `ShardedIndex` applies defensive recovery:
+
+- **Stale temp files** (`*.usearch.tmp`, `*.isbf.tmp`, `*.npy.tmp`) from interrupted durable
+    writes are deleted automatically.
+- **Missing or corrupt bloom filter** — rebuilt from all shard keys with a logged warning.
+    The bloom is a derived index, not a source of truth.
+- **Missing tombstone file** — assumed no tombstones. Previously tombstoned keys may reappear
+    from view shards.
+
 ## Load an index from disk
 
 `load()` reads the entire file into RAM:

docs/reference/for-coding-agents.md

@@ -158,6 +158,7 @@ semantics are required.
 - `contains()` / `get()` / `remove()` use bloom for fast rejection of non-existent keys.
 - False positives are expected (probabilistic). False negatives do not occur.
 - `rebuild_bloom()` creates a fresh filter from all existing keys.
+- Automatically rebuilt on load if the file is missing or corrupt (writable indexes only).
 - `compact()` calls `rebuild_bloom()` internally.
 - Bloom filter file: `bloom.isbf` in the shard directory.
 - Always loaded from disk if file exists, regardless of `_use_bloom` setting.
@@ -167,7 +168,7 @@ semantics are required.
 ```
 remove(key) → key added to _tombstones (in-memory set)
     ↓
-save() → _tombstones persisted as tombstones.npy
+save() → bloom persisted, then shard, then _tombstones as tombstones.npy
     ↓
 compact() → view shards rebuilt excluding tombstoned entries
     ↓
@@ -177,25 +178,27 @@ _tombstones.clear(), tombstones.npy deleted
 - `_needs_compact` flag: set when `add()` clears a tombstone (creating cross-shard duplicates).
 - `tombstones.npy` file existence signals `_needs_compact=True` on next load.
 - Tombstones are never written to bloom filter — bloom has no "remove" operation.
+- Tombstones are persisted **after** shard data so that tombstone removals only become
+    visible once the shard they depend on is durable.
 
 ---
 
 ## Side effects catalog
 
-| Method            | Disk writes                                                 | `_dirty`                | `_tombstones`              | Bloom update         | Shard rotation         |
-| ----------------- | ----------------------------------------------------------- | ----------------------- | -------------------------- | -------------------- | ---------------------- |
-| `add()`           | None (until rotation)                                       | `+= count_added`        | Clears matching tombstones | `add_batch()`        | Yes (if size exceeded) |
-| `remove()`        | None                                                        | `+= N` (existing only)  | Adds view-shard keys       | None                 | No                     |
-| `upsert()`        | None                                                        | Via remove + add        | Via remove + add           | Via add              | Via add                |
-| `add_once()`      | None                                                        | Via add (new keys only) | None                       | Via add              | Via add                |
-| `save()`          | `.usearch`, `bloom.isbf`, `tombstones.npy`                  | Reset to 0              | Persisted                  | Persisted            | No                     |
-| `load()`          | None                                                        | Reset to 0              | Loaded from `.npy`         | Loaded from `.isbf`  | No                     |
-| `view()`          | None                                                        | Reset to 0              | N/A (NphdIndex)            | N/A                  | No                     |
-| `reset()`         | None                                                        | Reset to 0              | Cleared                    | Cleared              | No                     |
-| `compact()`       | Rebuilds `.usearch`, `bloom.isbf`, deletes `tombstones.npy` | Reset to 0              | Cleared                    | Rebuilt              | No                     |
-| `search()`        | None                                                        | No change               | No change                  | None                 | No                     |
-| `get()`           | None                                                        | No change               | No change                  | None                 | No                     |
-| `rebuild_bloom()` | `bloom.isbf` (if `save=True`)                               | No change               | No change                  | Rebuilt from scratch | No                     |
+| Method            | Disk writes                                                             | `_dirty`                | `_tombstones`              | Bloom update         | Shard rotation         |
+| ----------------- | ----------------------------------------------------------------------- | ----------------------- | -------------------------- | -------------------- | ---------------------- |
+| `add()`           | None (until rotation; rotation durably writes bloom, shard, tombstones) | `+= count_added`        | Clears matching tombstones | `add_batch()`        | Yes (if size exceeded) |
+| `remove()`        | None                                                                    | `+= N` (existing only)  | Adds view-shard keys       | None                 | No                     |
+| `upsert()`        | None                                                                    | Via remove + add        | Via remove + add           | Via add              | Via add                |
+| `add_once()`      | None                                                                    | Via add (new keys only) | None                       | Via add              | Via add                |
+| `save()`          | `.usearch`, `bloom.isbf`, `tombstones.npy`                              | Reset to 0              | Persisted                  | Persisted            | No                     |
+| `load()`          | None                                                                    | Reset to 0              | Loaded from `.npy`         | Loaded from `.isbf`  | No                     |
+| `view()`          | None                                                                    | Reset to 0              | N/A (NphdIndex)            | N/A                  | No                     |
+| `reset()`         | None                                                                    | Reset to 0              | Cleared                    | Cleared              | No                     |
+| `compact()`       | Rebuilds `.usearch`, `bloom.isbf`, deletes `tombstones.npy`             | Reset to 0              | Cleared                    | Rebuilt              | No                     |
+| `search()`        | None                                                                    | No change               | No change                  | None                 | No                     |
+| `get()`           | None                                                                    | No change               | No change                  | None                 | No                     |
+| `rebuild_bloom()` | `bloom.isbf` (if `save=True`)                                           | No change               | No change                  | Rebuilt from scratch | No                     |
 
 ---
 

src/iscc_usearch/sharded.py

@@ -27,7 +27,7 @@
 )
 
 from iscc_usearch.bloom import ScalableBloomFilter
-from iscc_usearch.utils import atomic_write, timer
+from iscc_usearch.utils import atomic_write, durable_write, timer
 
 __all__ = ["ShardedIndex", "ShardedIndex128", "ShardedIndexedKeys", "ShardedIndexedVectors"]
 
@@ -1085,42 +1085,30 @@ def save(
                 "ShardedIndex.save() does not accept a path argument. "
                 "Files are saved to the directory specified at construction time."
             )
-        # Save bloom filter if it exists
-        if self._bloom is not None:
-            bloom_path = self._path / BLOOM_FILENAME
-            with timer("ShardedIndex save bloom filter", level="INFO"):
-                self._bloom.save(bloom_path)
+        # Persist bloom → shard → tombstones (safe ordering for crash recovery:
+        # shard must be durable before tombstone removals become visible)
+        self._persist_bloom()
 
-        # Save tombstones (also serves as the _needs_compact persistence flag:
-        # file exists → filtering needed on next load, even if tombstones are empty)
-        tombstone_path = self._path / TOMBSTONE_FILENAME
-        if self._tombstones or self._needs_compact:
-            with atomic_write(tombstone_path) as tmp:
-                with open(tmp, "wb") as f:
-                    arr = np.array(sorted(self._tombstones), dtype=self._key_dtype)
-                    np.save(f, arr)
-        elif tombstone_path.exists():
-            tombstone_path.unlink()
-
-        if self._active_shard is None or len(self._active_shard) == 0:
-            self._dirty = 0
-            return
+        if self._active_shard is not None and len(self._active_shard) > 0:
+            shard_path = self._get_active_shard_path()
+            active_count = len(self._active_shard)
+            with timer(
+                f"ShardedIndex save {shard_path.name} ({active_count:,} vectors)",
+                log_start=True,
+                level="INFO",
+            ):
+                data = self._active_shard.save(progress=progress)
+                durable_write(data, shard_path)
+            self._active_shard_path = shard_path
+            self._invalidate_shard_cache()
+
+        self._persist_tombstones()
 
-        shard_path = self._get_active_shard_path()
-        active_count = len(self._active_shard)
-        with timer(
-            f"ShardedIndex save {shard_path.name} ({active_count:,} vectors)",
-            log_start=True,
-            level="INFO",
-        ):
-            self._active_shard.save(str(shard_path), progress=progress)
-        self._active_shard_path = shard_path
-        # Invalidate cache since new shard file may have been created
-        self._invalidate_shard_cache()
         from loguru import logger
 
         num_shards = len(self._discover_shards())
-        logger.info(f"Saved {num_shards} shard(s) to {self._path}")
+        if num_shards:
+            logger.info(f"Saved {num_shards} shard(s) to {self._path}")
         self._dirty = 0
 
     def rebuild_bloom(self, save: bool = True, log_progress: bool = True) -> int:
@@ -1173,11 +1161,8 @@ def rebuild_bloom(self, save: bool = True, log_progress: bool = True) -> int:
         if log_progress:
             logger.info(f"Bloom filter rebuilt with {count:,} keys")
 
-        # Save if requested
         if save:
-            bloom_path = self._path / BLOOM_FILENAME
-            with timer("ShardedIndex save bloom filter"):
-                self._bloom.save(bloom_path)
+            self._persist_bloom()
 
         return count
 
@@ -1272,7 +1257,8 @@ def compact(self) -> int:
                 new_shard = self._create_shard()
                 new_shard.add(self._shard_batch_keys(live_keys), live_vectors)
 
-                new_shard.save(str(shard_path))
+                shard_data = new_shard.save()
+                durable_write(shard_data, shard_path)
 
                 viewed = self._restore_shard(shard_path, view=True)
                 if viewed is not None:  # pragma: no branch
@@ -1383,6 +1369,11 @@ def _load_existing(self) -> None:
         self._config["expansion_search"] = last_shard.expansion_search
         self._config["multi"] = last_shard.multi
 
+        # Rebuild bloom filter if missing or corrupt but shards exist
+        if self._use_bloom and self._bloom is None and not self._read_only:
+            logger.warning("Bloom filter missing — rebuilding from shard keys")
+            self.rebuild_bloom(save=True)
+
     @staticmethod
     def metadata(path: str | os.PathLike) -> dict | None:
         """Extract metadata from a sharded index directory.
@@ -1798,13 +1789,18 @@ def _load_bloom_if_exists(self) -> ScalableBloomFilter | None:
         in sync with index contents. The _use_bloom flag only controls whether
         to USE the bloom filter for fast rejection in lookups.
 
-        Returns None when no bloom file exists. Call rebuild_bloom() to create
-        a bloom filter for existing indexes that don't have one.
+        Returns None when no bloom file exists or if the file is corrupt.
         """
+        from loguru import logger
+
         bloom_path = self._path / BLOOM_FILENAME
         if bloom_path.exists():
-            with timer("ShardedIndex load bloom filter", level="INFO"):
-                return ScalableBloomFilter.load(bloom_path)
+            try:
+                with timer("ShardedIndex load bloom filter", level="INFO"):
+                    return ScalableBloomFilter.load(bloom_path)
+            except Exception:
+                logger.warning(f"Corrupt bloom filter at {bloom_path} — will rebuild")
+                return None
         return None
 
     def _restore_shard(self, path: Path, view: bool) -> Index | None:
@@ -1873,6 +1869,28 @@ def _schedule_next_size_check(self, current_size: int) -> None:
         else:
             self._adds_until_size_check = 1
 
+    def _persist_bloom(self) -> None:
+        """Save bloom filter to disk if it exists."""
+        if self._bloom is not None:
+            bloom_path = self._path / BLOOM_FILENAME
+            with timer("ShardedIndex save bloom filter", level="INFO"):
+                self._bloom.save(bloom_path)
+
+    def _persist_tombstones(self) -> None:
+        """Save tombstones to disk.
+
+        File presence also serves as the _needs_compact persistence flag:
+        file exists -> filtering needed on next load, even if tombstones are empty.
+        """
+        tombstone_path = self._path / TOMBSTONE_FILENAME
+        if self._tombstones or self._needs_compact:
+            with atomic_write(tombstone_path) as tmp:
+                with open(tmp, "wb") as f:
+                    arr = np.array(sorted(self._tombstones), dtype=self._key_dtype)
+                    np.save(f, arr)
+        elif tombstone_path.exists():
+            tombstone_path.unlink()
+
     def _rotate_shard(self) -> None:
         """Save current shard and create new one."""
         if self._active_shard is None:
@@ -1885,14 +1903,20 @@ def _rotate_shard(self) -> None:
         else:
             shard_path = self._get_shard_path(self._get_next_shard_number())
 
-        # Save current active shard
+        # Persist bloom → shard → tombstones (safe ordering for crash recovery:
+        # shard must be durable before tombstone removals become visible)
+        self._persist_bloom()
+
         active_count = len(self._active_shard)
         with timer(
             f"ShardedIndex rotate {shard_path.name} ({active_count:,} vectors)",
             log_start=True,
             level="INFO",
         ):
-            self._active_shard.save(str(shard_path))
+            data = self._active_shard.save()
+            durable_write(data, shard_path)
+
+        self._persist_tombstones()
         # Clear tracked path since we're creating a new unsaved shard
         self._active_shard_path = None
         # Invalidate cache since new shard file was created