Skip to content

Add polarization-specific validMask datasets to InSAR product - #379

Open
xhuang-jpl wants to merge 36 commits into
isce-framework:developfrom
xhuang-jpl:fix_insar_mask
Open

Add polarization-specific validMask datasets to InSAR product#379
xhuang-jpl wants to merge 36 commits into
isce-framework:developfrom
xhuang-jpl:fix_insar_mask

Conversation

@xhuang-jpl

Copy link
Copy Markdown
Contributor

Summary

Add polarization-specific validMask datasets to InSAR products for tracking reference and secondary image validity per polarization.

Changes

  • New Dataset: validMask (uint8) with bit encoding - bit 1: reference validity, bit 0: secondary validity
  • Geocoding: Added get_valid_mask_input_output() and updated CPU/GPU workflows to geocode validMask
  • Writers: Modified GUNW, GOFF, ROFF, and L1 writers to create validMask in pixelOffsets, unwrappedInterferogram, and wrappedInterferogram groups

Xiaodong Huang added 30 commits September 19, 2023 20:40
@s-sasaki-earthsea-wizard

Copy link
Copy Markdown

Thanks for integrating the row-wise generate_insar_mask vectorization from #359 into this PR — and the sliding-window _RSLCInputDataExceptionMask reader is a genuine improvement over #359's full-array loads, as the numbers below show.

I ran this PR's head (0c2ff0af5) against its merge-base (0d1600d8b) on a real NISAR L-band frame (ascending 139/019 pair, frequency A / HH, RIFG interferogram grid 6840 x 10581 = 72.4 Mpx), as a standalone prepare_insar_hdf5 run against a pre-seeded geo2rdr scratch — the same harness as the #358/#359 comparison posted here. Only the 7 files this PR changes differ between the runs.

variant prepare_insar_hdf5 (journal) vs scalar peak RSS
merge-base (scalar loop) 418.0 s 1x 12.5 GiB
this PR 84.8 s 4.9x 6.6 GiB

No regressions on this dataset. All 558 common datasets across the GUNW skeleton and the RIFG/RUNW scratch skeletons are byte-identical to the merge-base run except the usual run-varying metadata (processingDateTime, runConfigurationContents); that includes every existing mask layer. The only structural additions are the 7 new validMask datasets, and the radar-grid ones check out per pixel: HH validMask equals the sub-swath validity bits of the co-located mask (2*(ref digit > 0) + (sec digit > 0)) with zero mismatches on all four RIFG/RUNW layers (RIFG interferogram: 69,341,858 px with either image valid, 68,664,482 with both; pixelOffsets: 368,611 / 366,145). The GUNW validMask planes are creation-state skeletons at the prepare stage, so their geocoding is not covered by this check. Peak RSS lands below both the scalar baseline and #359 (10.1 GiB on the same harness), thanks to the windowed exception-mask reads. Full logs, provenance, and the comparison scripts: artifacts/pr379-ab-20260913/.

The performance and the tested-configuration results look good. Reading the integrated code turned up two validMask correctness notes in input configurations beyond the tested one, plus two minor cleanups — details and suggested fixes are folded below.

Two validMask correctness notes (multi-frequency products; non-HH polarizations with a uint8 or missing inputDataExceptionMask) — worth addressing before merge; neither manifests on the frame above
  1. The per-pol validMask update for pixelOffsets runs outside the frequency loop. In InSAR_L1_writer.add_pixel_offsets_to_swaths_group, the for pol in pol_list: block that writes pixelOffsets/{pol}/validMask (InSAR_L1_writer.py#L570-L583) sits at method level, while pol_valid_mask, pol_list, and offset_group_name are loop variables of the for freq, pol_list, _ in get_cfg_freq_pols(...) loop starting at L371. For a multi-frequency product only the last frequency's validMask is written; frequency A's stays at its creation bytes (all zeros = "both invalid", with _FillValue = 255). The interferogram counterpart in add_interferogram_to_swaths_group is correctly inside the loop. Note that indenting the update block alone is not quite enough: _add_datasets_to_pixel_offset_group(), which creates the per-pol validMask datasets, also runs after the loop (L568), so each frequency's dataset creation and validMask write need to happen inside the same frequency scope, in that order.

  2. The fallback for RSLCs without per-pol validity bits populates only the HH position. For RSLCs with a pre-RSLC missing data mask #370 uint8 inputDataExceptionMask — and equally when the dataset is absent — the fallback sets bit 8 (reference) / bit 0 (secondary) from sub-swath validity (utils.py#L893-L894, L914-L915). Those are the _PolBit.HH positions, so extract_pol_valid_mask returns all-invalid for every other polarization of such granules (e.g. HV of a DHDH pair). A minimal repro (in the artifact bundle linked above) confirms it at runtime: on a fully-valid grid with zero offsets, mask is 11 everywhere while the extracted validMask is HH=3 and HV/VH/VV=0, for both a uint8 and an absent exception mask. If the intent is "no per-pol information, fall back to sub-swath validity for all pols", broadcasting the fallback to all eight bit positions (* 0xFF00 / * 0x00FF) would preserve that.

Minor cleanups (fine as a follow-up)
  1. _subswath_numbers is defined twice in utils.py. The first definition (L497, the verbatim Add regression tests for generate_insar_mask #359 version) is shadowed by the second at L703 and can be deleted.

  2. pixel_offsets_ds_params in InSAR_L1_writer._add_datasets_to_pixel_offset_group lists slantRangeOffset twice (L328, L335) — harmless through require_dataset, but presumably unintended.

On #359: since this PR carries the implementation, I'll downshift #359 to its regression tests only (the offer from the #358/#359 threads). The suite passes against this PR's head as-is — 11/11, including the adversarial rounding and bit-packing cases — so it can serve as the regression gate for the uint32 mask semantics of the integrated implementation (it also sanity-checks the per-pol validity word against the mask's sub-swath digits; the finer per-pol bit semantics are this PR's own and are best covered by its tests).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants