Skip to content

refactor(shm): decompose and optimize label propagation - #156

Merged
DanielSeemaier merged 7 commits into
mainfrom
codex/lp-components-safe
Aug 17, 2026
Merged

refactor(shm): decompose and optimize label propagation#156
DanielSeemaier merged 7 commits into
mainfrom
codex/lp-components-safe

Conversation

@DanielSeemaier

@DanielSeemaier DanielSeemaier commented Jul 28, 2026

Copy link
Copy Markdown
Member

What changed

  • Replaced the shared-memory LP CRTP monolith with small production components for:
    • in-order and degree-bucket-aware chunk-shuffled iteration;
    • active-set and round state;
    • neighborhood rating aggregation and target selection;
    • clustering and balanced-move processing;
    • clustering postprocessing.
  • Consolidated label_propagation/ from 24 to 16 headers without creating an omnibus interface:
    • linear, fixed-capacity, and adaptive local maps share rating_map.h;
    • round statistics and traversal kernels share node_processing.h;
    • two-hop candidates and global/worker-local algorithms share two_hop_clustering.h;
    • the two single-caller workspaces now live in their owning .cc files.
  • Kept clusterer and refiner control flow explicit and short: they select an iteration order, construct a processor/kernel, and run rounds.
  • Restored all four two-hop algorithms as reusable, graph-only components:
    • global matching;
    • worker-local matching;
    • global clustering;
    • worker-local clustering.
  • Renamed the former single-/two-phase selector to the behavior it actually controls:
    • local: aggregate the complete neighborhood in worker-local storage;
    • deferred-parallel: defer neighborhoods that exceed local capacity and aggregate them in parallel.
  • Removed preset-unused research options, including growing rating maps, unused isolated-node variants, and the no-op two-hop disable mode.
  • Added cache-adaptive rating aggregation:
    • a tiny linear map for at most eight labels;
    • a fixed-capacity, insertion-order-preserving hash map for medium neighborhoods;
    • the existing direct map for small key universes and large neighborhoods.
  • Preserved the established hot-loop structure, tie order, RNG consumption, active-set bookkeeping, shared-memory synchronization, preset choices, and serialized enum values.

Why

The old CRTP design coupled iteration, state management, rating aggregation, selection, and moves in one large template hierarchy. The new components can be tested independently and reused without growing the template parameter surface of every LP consumer.

The performance changes retain only local, behavior-preserving work: fewer rating-map probes, smaller cache footprints, cached aggregation state, capacity-check elision when the bound proves safety, and cheaper iteration-order bookkeeping. Schedule-changing round-policy optimizations are deliberately excluded.

Performance

  • Local alternating smoke benchmark on an Apple M5 Pro, 12 paired repetitions per graph/preset:

    • grid: partitioning -10.82% default / -6.85% fast; total LP -18.62% / -13.95%;
    • 40.5M-directed-edge Recomp English graph: partitioning -15.30% / -9.28%; total LP -20.94% / -19.21%;
    • 46 wins, 2 ties, 0 losses for partitioning across 48 pairs.
  • UFM cluster validation (k=16, seed 1, epsilon 0.03, 64 threads, 73 graphs):

    • equal-graph partitioning time: -3.44% default (56/0/17 W/T/L) and
      -7.34% fast (52/0/21);
    • summed partitioning time in the one-seed matrix: +2.00% default and
      -4.73% fast; after repeating the two dominant outliers per preset over
      seeds 2--6 and replacing only those first-sample observations with their
      fresh-seed means, the sensitivity result is -3.31% default and -4.48%
      fast;
    • equal-graph total LP time: -14.80% default (73/0/0) and -10.57%
      fast (63/1/9); summed total LP time: -11.71% and -6.81%;
    • equal-graph coarsening LP: -16.12% default / -10.87% fast;
      refinement LP: -12.00% / -10.89%;
    • equal-graph cut: -0.47% default / -0.53% fast. The raw summed cut was
      -0.59% / +1.13%; five fresh seeds on the two dominant fast-preset
      social outliers improved summed cut by -1.28% (twitter-2010) and
      -1.14% (com-Friendster), changing the same outlier-replacement
      sensitivity to -0.75% default / -0.88% fast.
  • The restored strategies do not alter the default/fast preset selections, and their worker-local matching hot loop remains unchanged. A paired check against the previous PR head on com-lj (4.0M nodes / 69.4M adjacency entries, 12 performance cores) found no detectable regression:

    • dedicated 5-iteration LP benchmark, 15 pairs: paired median 0.00%, mean +0.35% (approximate 95% CI -0.25%..+0.96%);
    • dedicated 1-iteration LP benchmark, 15 pairs: paired median 0.00%, mean +0.10% (approximate 95% CI -0.64%..+0.84%);
    • nine paired full runs per preset remained within run-to-run noise.
  • The 24-to-16 header consolidation was checked against the preceding PR head on com-lj (4.0M nodes / 69.4M adjacency entries, 12 performance cores) with no stable regression signal:

    • dedicated LP medians: -0.74% for five iterations and -1.30% for one iteration;
    • corrected seven-pair alternating -P fast confirmation: paired medians -0.59% partitioning and -1.14% summed LP;
    • all 14 confirmation outputs were feasible, with identical imbalance and a -0.0053% median-cut change.

Validation

  • Current SHM Release build and full local test suite: 348/348 tests passed.
  • Focused LP/component suite: 44/44 tests passed in both 32-bit and 64-bit node-ID builds.
  • The two-hop tests cover candidate selection, global and worker-local matching/clustering, direct candidate repair, weight limits, cluster-weight consistency, and repeated four-worker contention; the eight strategy tests also passed 100 repetitions each (800/800).
  • ASan + UBSan: 44/44 focused LP tests and 10/10 SHM end-to-end tests passed with no diagnostics.
  • CLI smoke tests exercised all eight combinations of four two-hop algorithms and two canonical rating-aggregation modes; configuration tests also cover the legacy single-/two-phase value aliases.
  • GitHub Actions on commit 6f1f99e4: 20/20 substantive checks passed, including GCC/Clang, macOS, package, wheel, bindings, and example builds; seven release-only publication/signing jobs were correctly skipped.
  • Cluster matrix: 292/292 exact CSV/log pairs validated, all feasible, with exact graph pairing and CSV/log timer, cut, and imbalance reconciliation.
  • Targeted follow-up: 40/40 exact logs (20 baseline/candidate pairs), all feasible; exact commits, k=16, seeds 2--6, epsilon 0.03, and 64 threads validated.
  • Both cluster revisions were built on the iverson compute node with GCC 14.2 for Neoverse N1; no compilation ran on a login node.

@DanielSeemaier
DanielSeemaier marked this pull request as ready for review July 28, 2026 11:33
@DanielSeemaier
DanielSeemaier merged commit a2d1a7b into main Aug 17, 2026
27 checks passed
@DanielSeemaier
DanielSeemaier deleted the codex/lp-components-safe branch August 17, 2026 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant