Compute stable edge IDs on demand and stop storing them (DATA-6257)

[bnaul]

Compute stable edge IDs on demand; stop storing them per edge

Implements DATA-6257 (parent DATA-6256, MEMORY_REVIEW.md rec #1) in full. The ticket's staged plan (Phase 1 shadow-verify behind a flag, then Phase 2 format break) is collapsed into this one PR — we validate off-branch with real test runs rather than running a production shadow window.

The stable edge ID is a pure function of osmWayId + segmentIndex (+ direction) (StableIdEncodedValues.calculateStableEdgeId). The way ID is already stored split across osmid/osmid_high, and the segment index lives in gh_edge_id_to_segment_index — so the 16 stable_id_byte_* / reverse_stable_id_byte_* encoded values (16 B/edge, amplified through CH shortcut copies; the doc estimates 1.5–5 GB steady-state on a 60 GB router) are redundant. This PR derives the ID at every read site and stops storing the bytes.

This is a graph format change — affected graphs must be re-imported to reclaim the space.

What changed

Compute path

• StableIdEncodedValues.computeStableId(...) — from an EdgeIteratorState (serving/export) or from an edgeId + EdgeIntAccess (import); byte-identical to the old stored value.
• Read sites now always compute: the gRPC StreetPath.stable_edge_ids path-details builder and the BQ StreetEdgeExporter. The segment index reaches the serving hot path via OsmHelper, wired lazily through PathDetailsBuilderFactoryWithStableId (OsmHelperProvider).
• Virtual edges: routed paths include virtual edges at snapped endpoints (id ≥ base edge count); a virtual edge mirrors its original edge's encoded values, but its segment index must be looked up under the original edge id via VirtualEdgeIteratorState.getOriginalEdgeKey() (this fork's GH 8.0 has no EdgeIteratorState.isVirtual(), so detection is instanceof).

Stop storing

• EncodedValueFactoryWithStableId no longer registers the 16 byte EVs; GraphHopperManaged drops them from the encoded-values string.
• Import no longer stamps: CustomOsmReader drops the stamping call and EdgeCoalescer no longer re-stamps merged edges (it still mints their fresh segment index).
• StableIdEncodedValues loses the stored getStableId/setStableId byte paths; IdentityEncodedValues drops the byte names from the EdgeCoalescer routing-equivalence mask (now just the two osmid halves).

Import-time SEID custom speeds

• The custom-speeds tag parsers compute each edge's stable ID on demand during import. The segment index of the edge being parsed reaches them through a new ImportSegmentIndexLookup (pointed at the reader's in-progress map; the map write was moved before handleWayTags), threaded into StableEdgeIdManager via ReplicaVehicleTagParserFactory.

Format compatibility

Consistent with the existing convention (cf. the osmid_high split, which also changed the EV set without a version constant), there is no separate graph-version constant in the fork; the format change is realized by a coordinated re-import. GraphHopper's storage is self-describing (the encoding manager is persisted with the graph), so an old graph is not silently misread — its stored stable-id bytes are simply ignored by the new code until the graph is rebuilt. Coordinate the re-import with the road_network_version bump.

Tests

Runnable locally (mvn -o -pl replica-common,web-bundle test — green):

• StableIdComputeTest — computeStableId == calculateStableEdgeId across OSM ids (incl. > 2^31, exercising osmid_high), segments, both directions.
• ComputedStableIdImportTest — imports beatty.osm (coalescing off and on); asserts the byte EVs are gone, every parsed edge has a positive segment index and a derivable ID both directions, and every stable_edge_ids value the serving path returns is one of the graph's directly-computed IDs (this fails if the virtual-edge segment-index mapping regresses). 768 / 754 edge-pairs and 554 / 552 served IDs verified.
• SeidCustomSpeedImportTest — end-to-end proof of the trickiest path: imports beatty, writes a stable_edge_id-keyed closure for a real edge, re-imports under a custom-speeds car vehicle, and asserts the closure removed access on exactly that edge/direction (and left the reverse + a control edge alone). This exercises the full import-time seam — parse → custom-speeds access parser → getStableEdgeIdForEdge → computeStableId via ImportSegmentIndexLookup → match — so it fails if the segment-index threading regresses.
• EdgeCoalescerTest, IdentityEncodedValuesTest updated for the compute API.

Performance: computing on the fly costs ~100 ns/path-edge more than the old byte read (farmHash vs a memory read), and path details are built once over the returned path (not in the search loop), so even a ~200-edge route adds ~20 µs — negligible against CH routing, and the point is the GBs of memory it buys back.

✅ CI is green (run 27230547590 on c49597be0): the full web + grpc build and unit suite pass — including the web-module SEID custom-speeds integration tests (CustomSpeedsUtilsTest, StableEdgeIdManagerTest) — so the edits to those modules (which can't be compiled in the dev sandbox: no generated protobuf, web → grpc) are confirmed. An earlier red was a pre-existing test-fixture bug (1-int scratch IntsRef), fixed here.

📋 Remaining real-world step (not a blocker for review): the actual memory-savings measurement — rebuild a representative graph with the deployed CH-profile config and diff graphhopper_network/ size + loaded RSS, old vs new format. The in-repo work confirms the per-edge term (edge-flags row 8→4 ints = 16 B/edge on a car-only graph) but the headline 1.5–5 GB is CH-amplified and needs a production-scale rebuild. Hand-off brief: plans/data-6257_memory_validation.md.

Base branch

Stacked on brett/data-6259-... (edge coalescing), which carries the 64-bit osmid_high split the computation needs. Per DATA-6256, 6259 lands first; retarget if you'd rather it sit directly on tomtom-orbis-osmid-64bit.

Rollout

Merge → rebuild affected graphs (orbis 2026_Q2/USA + regional_transportation_network regions) → bump road_network_version. No production shadow window (phases combined).

🤖 Generated with Claude Code

[claude]

Claude finished @bnaul's task in 3m 31s —— View job

---

Review: Compute stable edge IDs on demand (DATA-6257 Phase 1)

• Read changed files and diff vs base branch
• Review correctness / logic
• Review concurrency & thread-safety (multi-threaded gRPC server)
• Review hot-path performance
• Review edge-case & error handling
• Post review feedback

Overall this is a clean, well-scoped Phase 1. The compute-on-demand path is genuinely byte-identical to the stored path (both funnel through calculateStableEdgeId → Long.toUnsignedString(Longs.fromByteArray(...))), the virtual-edge remap via getOriginalEdgeKey() is the right fix for the segment-index lookup, the lazy OsmHelper resolution is correctly double-checked-locked on a volatile, and the tests (unit equivalence + full beatty import with coalescing on/off + serving-path routing) cover the important cases. Default-off keeps it safe to land. The two edits in web/grpc look correct against the existing call patterns — agree they're worth a CI confirm since they weren't machine-compiled.

I left three inline comments. Two are worth resolving before the shadow rollout:

1. Unbounded WARN logging on mismatch (StableIdVerification.verifyComputed) — main concern

The doc says verification "must never disrupt serving," but the WARN is unthrottled: a systematic divergence logs once per (edge, direction) comparison. At production QPS × many edges per route, that's potentially thousands of lines/sec — which can pin a core, fill disk, and degrade the path it's shadowing. The high-mismatch case is exactly the one to guard. Suggested a cheap rate-limit keyed on the existing counter (keeps the count exact). Inline →

2. verifyComputed returns computed, not stored

This makes it a substituting verification, not a true shadow: during the Phase 1 window any divergence ships the (possibly wrong) computed ID to gRPC clients and into the BQ export — corrupting output, not just logging. Equality is fully captured by the counters regardless of which string is returned, so returning stored would prove computed == stored across real traffic while guaranteeing output is unchanged until Phase 2 deliberately flips it. At minimum, call out in the rollout plan that a non-zero routers.stable_id_computed_mismatches means you're serving wrong IDs that week, not just observing. Inline →

3. Misconfig fails lazily at request time, not boot (minor)

Flag-on + non-OSM/GTFS graph throws IllegalStateException on the first (and every subsequent) routing request rather than at startup. Both inputs are known at construction — consider fail-fast validation when the flag is on. Low priority since production graphs are always OSM/GTFS. Inline →

Non-blocking notes

• Per-edge cost while enabled: the compute path adds a String.format + farmhash + a DataAccess lookup per edge on the stable_edge_ids serving hot path. Fine given default-off and a one-router shadow window, but it ~doubles the per-edge stable-ID work while on.
• Thread-safety: getSegmentIndexForGhEdge does an absolute DataAccess.getInt with no mutation — safe under concurrent reads, same as the rest of the graph during routing. Path-detail builders are per-request, so prevEdgeId/currentValue aren't shared. No new concurrency issues.
• Metrics modeling: comparisons/mismatches are cumulative counters reported via client.gauge. Works for "flat-zero mismatches," but comparisons grows unbounded as a gauge — a counter/rate would read more naturally on the dashboard. Cosmetic.
  • branch brett/data-6257-compute-stable-edge-ids-on-demand-instead-of-storing-them

[claude]

Unbounded WARN logging on mismatch can itself disrupt serving. The doc comment promises verification "must never disrupt serving," but a systematic divergence (e.g. one edge class, or a virtual-edge handling gap) would log one WARN per (edge, direction) comparison — at production QPS with many edges per route, that's potentially thousands of log lines/sec, which can pin a core on logging, fill disk, and degrade the very serving path it's shadowing. The whole point of the flag is to catch mismatches in real traffic, so the failure mode where it fires a lot is the one most worth guarding.

Recommend rate-limiting/sampling the log (the AtomicLong total already gives you a cheap throttle key) while keeping the counter exact, e.g.:

Suggested change

	if (!computed.equals(stored)) {
	long total = mismatches.incrementAndGet();
	LOG.warn("Computed stable edge ID != stored value (total mismatches={}): computed={} stored={} [{}]",
	total, computed, stored, context.get());
	}
	if (!computed.equals(stored)) {
	long total = mismatches.incrementAndGet();
	// Rate-limit: a systematic divergence would otherwise log once per comparison on the
	// serving hot path. The counter stays exact; the metric is the real signal.
	if (total <= 100 || Long.bitCount(total) == 1) {
	LOG.warn("Computed stable edge ID != stored value (total mismatches={}): computed={} stored={} [{}]",
	total, computed, stored, context.get());
	}
	}

[claude]

Returning computed (not stored) means a mismatch ships the possibly-wrong value to clients/exports. This is a deliberate choice ("the path Phase 2 will keep"), but it makes this a substituting verification rather than a true shadow: during the Phase 1 window, any divergence on the serving path returns the computed stable_edge_ids to gRPC clients, and in StreetEdgeExporter writes the computed ID into the BQ export — so a non-zero mismatch corrupts output, not just a log line.

Since equality is fully captured by the counters regardless of which string you return, returning stored here would still prove computed == stored across real traffic while guaranteeing output is unchanged until Phase 2 deliberately flips it. Worth at least calling out the trade-off explicitly in the rollout plan (one Places router for a week) — if routers.stable_id_computed_mismatches goes non-zero, you're serving the wrong IDs for that window, not just observing.

[claude]

Minor: this IllegalStateException for a misconfigured graph type (flag on + non-OSM/GTFS graph) only fires lazily on the first routing request that asks for stable_edge_ids — and via resolveOsmHelper() it re-throws on every subsequent request (the volatile osmHelper stays null). So a misconfig surfaces as a stream of failed requests rather than a startup failure. Since useComputedStableId and graphHopper are both known here at construction time, consider fail-fast: validate graphHopper instanceof OsmHelperProvider now (when the flag is on) so a bad config dies at boot, not in the serving path. Low priority — production graphs are always OSM/GTFS.