Add regression tests for generate_insar_mask

[s-sasaki-earthsea-wizard]

Summary

Downshifts this PR to regression tests only, per the offer in the #358/#359 threads: the generate_insar_mask vectorization originally proposed here has been integrated into #379 (thanks @xhuang-jpl), so the implementation commit is dropped and what remains is the test suite that pins the mask semantics for any implementation.

What the tests pin

• SubSwaths.get_sample_sub_swath equivalence: out-of-bounds -> 0, first-match-wins sub-swath ordering, an empty valid-samples array claims every in-bounds sample, no-sub-swath-info assigns 1 in bounds;
• the two distinct rounding rules (sub-swath lookup truncates int(x + 0.5), exception-mask lookup rounds half to even), exercised at exact k + 0.5 landings where they diverge;
• whole-pixel 0 outside the reference swath; secondary indices outside the secondary swath drop the secondary exception bits but keep the sub-swath digits;
• exact (ref << 16) | (sec << 8) exception-byte packing, with MSB-set bytes chosen so that fixed-width uint8 scalar promotion (silently dropped bits, NEP 50 semantics included) would be caught;
• randomized adversarial cases cross-checked against a per-pixel scalar reference that reuses the module's _compute_subswath_mask_id helper with Python-int bit packing (so the oracle cannot inherit a fixed-width overflow). The helper is currently retained by Add polarization-specific validMask datasets to InSAR product #379; these tests rely on it staying importable — if it is ever removed, the oracle should be inlined here instead.

Scope and compatibility

The suite is a regression gate for the uint32 mask semantics (including the legacy uint8 / absent inputDataExceptionMask path, which is what the fixtures feed). It runs against the current scalar implementation on develop (7 passed, 4 skipped — the helper-level _subswath_numbers tests skip when no vectorized helper exists and activate automatically once one lands) and against #379's head as-is (11 passed). A small unpack_mask shim accepts both the single-array return and #379's two-element (mask, pol_valid_mask) tuple; when the tuple is present, the per-pol validity word is sanity-checked against the mask's sub-swath digits (validity bits set exactly where the corresponding sub-swath digit is nonzero) before the mask is compared. The finer per-pol bit semantics and the uint16 exception-mask path introduced by #370/#379 are intentionally out of scope here and are best covered by #379's own tests. Test runs above used numpy 1.26.4.

Follow-up once #379 lands

The transition affordances should then be removed: make the _subswath_numbers import unconditional (dropping the skipif) and unpack the tuple directly (dropping the single-array branch of the shim), so signature drift fails loudly instead of being tolerated.

[s-sasaki-earthsea-wizard]

Following up on the cross-check I posted on #358 with the real-data leg — posting it here since this PR is where the prepare_insar_hdf5 implementation decision lands.

I re-ran the three-way comparison on a real NISAR L-band frame (ascending 139/019 Boso pair, freq A: RIFG interferogram grid 6840 x 10581 = 72.4 Mpx, RUNW 2565 x 4069, plus the pixel-offset masks), as a standalone prepare_insar_hdf5 run against a pre-seeded geo2rdr scratch. Same overlay method as the synthetic benchmark: the only difference between runs is the utils.py content (resolved module path + md5 echoed in each log), one run per variant on an otherwise idle host.

utils.py variant	prepare_insar_hdf5 (journal)	vs scalar	peak RSS (/usr/bin/time -v)
develop (scalar loop)	406.3 s	1x	12.4 GiB
#358 (full-grid)	92.9 s	4.4x	17.9 GiB
#359 (row-wise)	78.9 s	5.1x	10.1 GiB

Bitwise equivalence holds on real data for both implementations. All 558 datasets across the GUNW skeleton and the RIFG/RUNW scratch skeletons were compared byte-for-byte against the scalar run: for both #358 and #359 the only differences are the run-varying metadata (processingDateTime, runConfigurationContents) — every science dataset, including all mask layers, is identical on real granules with real (NaN-bearing) offset rasters. So @lijun99's bit-for-bit claim from #358 holds on NISAR data as well; the two vectorizations and the scalar loop are output-equivalent three ways, which I think settles correctness for the file-driven path.

Where the two differ at production scale is memory, more than speed. The full-grid formulation holds several ~579 MB temporaries concurrently on the 72.4 Mpx RIFG grid, putting the step's peak at 17.9 GiB — +5.5 GiB over the scalar baseline. The row-wise formulation peaks at 10.1 GiB, below the scalar baseline (the scalar loop's 72.4M-element Python list of ints, ~2 GiB, is gone). The ~12.4 GiB floor common to all three is the workflow's own working set (RSLC-grid exception masks etc.). The wall-clock difference between the two vectorizations is real but secondary at step scale (~1.2x). Since NISAR PGE workers run under memory budgets, this seemed worth having on the record before any implementation decision, so I am not holding it back.

Full logs, provenance (variant SHAs/md5s, seed, environment), and the comparison script are public in my benchmark repo: artifacts/pr358-realdata-ab-20260827/ (real data) and artifacts/pr358-crosscheck-20260827/ (synthetic grid + oracle suite).

The offer from the #358 thread stands unchanged: I am happy with either implementation landing — if the maintainers prefer #358's for the prepare part, I can rebase this PR down to its regression tests.

[xhuang-jpl]

Part of codes in this PR to improve the prepare_insar_hdf5 efficiency has been integrated into the #379

[s-sasaki-earthsea-wizard]

@xhuang-jpl Thanks for the heads-up — glad the row-wise implementation is useful in #379.

Following the standing offer from this thread: since #379 now carries the prepare_insar_hdf5 implementation, I've downshifted this PR to the regression tests only. The suite pins the scalar mask semantics (sub-swath first-match ordering, the two rounding rules, exception-byte packing under NEP 50) against a self-contained per-pixel oracle, skips its helper-level tests when no vectorized _subswath_numbers exists, and tolerates both the current single-array return and #379's (mask, pol_valid_mask) tuple — I've verified it passes on current develop (7 passed, 4 skipped) and on #379's head (11 passed). A/B verification of #379 itself is posted on that thread.