fix(uc3-uhi): unblock the Guatemala UHI use case on develop by BachirNILU · Pull Request #148 · WUR-AI/aether

BachirNILU · 2026-08-26T09:04:53Z

Testing the new dataset/caption refactor on the Guatemala UHI use case (#125). Four things blocked it. Two are not use-case specific.

Up front: this branch was produced with AI assistance (Claude Code). I've reviewed and run everything, but please read it as a proposal rather than a finished patch — a few calls below are choices, not obvious fixes, and I'd rather you decide.

Bugs

base_datamodule.py — from_file split builds a MissingDataError instead of raising it, so exception objects land in the index array and surface much later as:

TypeError: list indices must be integers or slices, not MissingDataError

This fires for any name_loc in the split file that isn't in records. Affects every UC that loads a saved split, not just this one. Also: train silently dropped unmatched entries while val/test did not, and the test branch used the message "Validation split is missing data".

train.py — cfg.get("test") and wandb_logger is not None silently skips the test and validate stages when LOGGER=none (the .env.example default), even with test: True. Only the wandb_logger.log_metrics call needs the logger.

base_model.py — loss_fn.setup() is called unconditionally, but the predictive models are configured with torch losses directly. Every heat_guatemala_* prediction experiment died on 'MSELoss' object has no attribute 'setup'.

configs/model/heat_*_cnn_best.yaml — CNNEncoder no longer accepts freezing_strategy or output_dim, but both configs still passed them. These ship on develop and are incompatible with develop's own code.

New

return_aux_ids for HeatGuatemalaCaptionBuilder. Only ButterflyCaptionBuilder returns the tuple the base class unpacks, so the feature raised for us. YieldAfricaCaptionBuilder has the same gap — not touched here.

scripts/infer_aligned_embeddings.py — coordinates in, GeoTessera API fetch, aligned embeddings out, using a trained checkpoint. --captions prints a cosine-similarity matrix as a quick alignment check.

Please review — these are choices

B1 behaviour. I drop unmatched name_locs with a warning to match what train already did. The MissingDataError suggests you may have wanted a hard failure on any unmatched validation row. Both are defensible; if you prefer hard-fail it just needs to be an actual raise, and the split file becomes a strict contract. Your call.
Restored configs/experiment/heat_guatemala_*.yaml and heat_alignment.yaml, which were removed from develop. If that deletion was deliberate, drop this part of the diff and I'll relocate them.
Removing output_dim: 256 from the CNN configs changes the encoder output to the resnet default. Correct w.r.t. the new API, but it does change the tuned model.

Not fixed here — worth separate issues

setup_tessera() is unreachable: download_missing_tiles is a hardcoded local False. Inside it: an undefined avail_files in the full-download branch; get_tessera_embeds() called without its required version; except NoTileError or PartialTileError (only catches the first); and a mask recomputed against a stale avail_files, so freshly downloaded tiles get dropped anyway. Fixing it means deciding how downloads should be requested, which felt like your call.
Tessera version defaults to v1.1, which doesn't cover Guatemala City — 11 tiles for 2024 in the 1°×1° box, vs the 13,038 we hold from v1. A live fetch fails under v1.1 and succeeds under v1. Worth reconsidering the default.
Minor: logging.info instead of log.info in setup_tessera and store_concept_thresholds (lands on root); random() rebinds template_ids inside the loop iterating it; assert getattr(torch, dtype), KeyError(...) raises AttributeError first.

Verified

All 10 heat_guatemala_* experiments plus heat_alignment smoke-test clean. Full 5-epoch alignment run: val_loss 3.432 → 3.252, val_pos_sim 0.597 vs val_neg_sim 0.430, test_loss 3.602. n_captions_for_validation and return_aux_ids both confirmed working.

🤖 Generated with Claude Code

Four defects prevented `experiment=heat_alignment` and every `heat_guatemala_*` prediction experiment from running. Two of them are not use-case specific. base_datamodule: in the `from_file` split, a MissingDataError was built rather than raised, so exception objects ended up inside the index array and surfaced later as "list indices must be integers or slices, not MissingDataError". Unmatched name_locs are now dropped with a warning (matching what the train split already did) and an empty split raises. train.py: `cfg.get("test") and wandb_logger is not None` silently skipped the test and validate stages whenever LOGGER=none. Only the wandb log_metrics call needs the logger. base_model: loss_fn.setup() was called unconditionally, but the predictive models are configured with torch losses (nn.MSELoss), which have no setup hook. heat_guatemala_caption_builder: implements return_aux_ids. Templates use expert-legend label columns while the soft loss scores on the numeric ones, so an explicit label -> numeric map is needed. aux_lst is excluded so the target cannot leak into a caption. configs: CNNEncoder no longer takes freezing_strategy or output_dim, but both CNN model configs still passed them. Restores the heat_guatemala experiment configs, which were dropped from develop. scripts/infer_aligned_embeddings.py: fetches Tessera through the GeoTessera API and projects it into the aligned text space using a trained checkpoint. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnFiUYCJ445jKSqHKaNbXt

…ence script The csv output now carries name_loc, lat, lon, the raw pooled Tessera vector (tessera_000..127) and the aligned one (aligned_000..511), so the two can be compared directly downstream. Pooling moved into the per-tile load. Stacking full tiles first held ~1.7 GB across the 13k Guatemala locations; the pooled vectors are ~7 MB. Verified on all 13,038 locations, none skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnFiUYCJ445jKSqHKaNbXt

The csv now has five columns: name_loc, lat, lon, tessera_embedding, aligned_embedding, with the whole vector per cell as a bracketed list (json.loads reads it back). --wide-columns keeps the previous one-column-per-dimension layout. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnFiUYCJ445jKSqHKaNbXt

plot_concept_maps.py takes the aligned-embedding csv plus a json of concepts, encodes each concept's captions with the trained text branch, and writes a per-location score table and one map per concept. Scoring is a single matmul: the aligned embeddings are already L2-normalised. Maps are coloured by z-score. Raw cosines occupy a narrow band, so a map of them reads as flat; the ranking is unchanged. The run also prints the correlation between concepts, since near-duplicate concepts otherwise produce near-identical maps. demo_concepts.json holds the vocabulary, limited to the concepts with a positive skill index on the test split. Both scripts now ignore the legacy text_model.embeddings.position_ids buffer, which older transformers versions wrote into checkpoints and which was being rejected as an unexpected key. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LnFiUYCJ445jKSqHKaNbXt

gabrieletijunaityte · 2026-08-26T13:35:31Z

+        # set up loss if needed. Only BaseLossFn subclasses define setup(); the
+        # predictive models are configured with torch losses directly (nn.MSELoss,
+        # nn.BCEWithLogitsLoss, ...), which have no such hook.
+        if self.loss_fn is not None and hasattr(self.loss_fn, "setup"):


As discussed, I am not sure if we should keep it?

gabrieletijunaityte · 2026-08-26T13:35:49Z

    train_metrics = trainer.callback_metrics

-    if cfg.get("validate") and wandb_logger is not None:
+    # NB: only the wandb bookkeeping needs a wandb_logger; trainer.validate/test


I agree! Could we remove these comments?

gabrieletijunaityte · 2026-08-26T13:42:58Z

-                        records_name_to_idx[n]
-                        if n in records_name_to_idx
-                        else MissingDataError("Validation split is missing data")
+            def _to_record_indices(name_locs, split_name):


Before hand we had a MissingData error specifically for validation and test splits, which must remain identical. The split was already processed so that validation and test splits only contain locations that have all modalities available. Meanwhile train split can use as many locations as there are available for a specific location.

Looking at this code, I don't think it functions the intended way.

I agree it should be raise MissingDataError(f"{mode} split is missing data")

gabrieletijunaityte · 2026-08-26T13:49:25Z

The configurations were not removed but rather moved to experimetns/guatemala subfolder. Could you please merge the configurations to this subfolder?

base_datamodule: restore the original contract. Validation and test splits are already built to contain only locations with every modality available, so a name_loc missing from the records means the split and the data have diverged and the run must stop -- raise MissingDataError. The train split may legitimately be shorter, so unmatched entries there are dropped with a warning. The bug being fixed remains the same one: the error was constructed rather than raised, which put exception objects into the index array. Note this makes a saved split a hard contract: a split file predating the current model-ready csv now stops the run instead of warning. base_model: use isinstance(self.loss_fn, BaseLossFn) rather than a hasattr probe. Only BaseLossFn subclasses define setup(); the predictive models are configured with torch losses directly. train.py: drop the explanatory comment, keep the fix. configs: the guatemala experiment configs were moved to configs/experiment/guatemala/, not removed -- my earlier check missed the subfolder and restored duplicates at the top level. Those are removed. Only the two configs genuinely absent upstream are added to the subfolder: heat_guatemala_full_fusion_avg_reg and heat_guatemala_tessera_tabular_avg_best. All 11 guatemala experiments smoke-tested after the change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

BachirNILU · 2026-08-27T13:33:01Z

Hi @gabrieletijunaityte , all four comments addressed (with support from Claude code) please feel free to take another look.

Adds the Guatemala use case to src/inference/, following the layout the s2bms habitat-similarity script established, each with a config under configs/. inference_heat_guatemala_embeddings.py produces the aligned Tessera embeddings. inference_heat_guatemala_concept_similarity.py is the Guatemala counterpart of inference_s2bms_habitat_similarity.py: it collects the aligned embedding, the predictive model's LST estimate and the measured LST for every location in a split, then correlates each concept's text similarity against both. The prediction head consumes the pooled 128-d Tessera vector while the aligned embedding is that same vector projected into the 512-d text space, so the two branches share an input but not a representation and neither output can be fed to the other. Both checkpoints are therefore needed, loaded separately. Shared loaders live in src/utils/heat_guatemala_inference.py rather than beside the scripts: src/inference.py and src/inference/ collide as import targets, so src.inference cannot be made a package without shadowing the existing module. First result on the test split (3368 locations), which removed a concept from the demo vocabulary: predictive model r=0.853, RMSE=1.078 degC vegetation greenness r=-0.595 vs measured LST forest cover r=-0.646 established urban r=+0.431 cool area r=+0.217 <- wrong sign, dropped "Cool area" similarity correlates positively with measured LST, i.e. it points at hotter places. It tracks established urban fabric rather than coolness, so it is moved to the excluded list in demo_concepts.json. scripts/infer_aligned_embeddings.py is left untouched; it is the standalone argparse version already shared for the demo UI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…s normalisation epoch_093 is the tessera+average model the experiment config documents (R2=0.733, RMSE=1.011, MAE=0.814); this checkout reproduces R2=0.7330, MAE=0.8147 on the test split. It replaces epoch_059, a develop-side retrain that reaches only R2=0.7103. epoch_093 cannot simply be loaded by develop's PredictiveModel. Feature normalisation changed from a plain L2 F.normalize to a trainable nn.LayerNorm, so every checkpoint trained before that change lacks the LayerNorm weights. Loading one anyway raises nothing and silently scores R2 = -8230 on this split. load_prediction_branch now infers the era from the weights -- LayerNorm when normalizer.* is present, L2 otherwise -- and logs which it picked. Validation on the test split (3368 locations) is unchanged in sign and slightly stronger against predictions: predictive model r=0.862, RMSE=1.035 degC vegetation greenness r=-0.595 vs measured, -0.633 vs predicted forest cover r=-0.646 / -0.716 established urban r=+0.431 / +0.458 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

BachirNILU and others added 4 commits August 26, 2026 11:04

gabrieletijunaityte reviewed Aug 26, 2026

View reviewed changes

BachirNILU and others added 2 commits August 28, 2026 10:55

gabrieletijunaityte approved these changes Aug 31, 2026

View reviewed changes

gabrieletijunaityte added 2 commits August 31, 2026 13:30

Merge branch 'develop' into fix/uhi-develop-integration

1a96400

Merge branch 'develop' into fix/uhi-develop-integration

6e2463c

Navigation Menu

Uh oh!

fix(uc3-uhi): unblock the Guatemala UHI use case on develop - #148

fix(uc3-uhi): unblock the Guatemala UHI use case on develop#148
BachirNILU wants to merge 9 commits into
developfrom
fix/uhi-develop-integration

BachirNILU commented Aug 26, 2026 •

edited

Loading

Uh oh!

gabrieletijunaityte Aug 26, 2026

Uh oh!

gabrieletijunaityte Aug 26, 2026

Uh oh!

gabrieletijunaityte Aug 26, 2026

Uh oh!

gabrieletijunaityte Aug 26, 2026

Uh oh!

gabrieletijunaityte commented Aug 26, 2026

Uh oh!

BachirNILU commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

BachirNILU commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bugs

New

Please review — these are choices

Not fixed here — worth separate issues

Verified

Uh oh!

gabrieletijunaityte Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

gabrieletijunaityte Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

gabrieletijunaityte Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

gabrieletijunaityte Aug 26, 2026

Choose a reason for hiding this comment

Uh oh!

gabrieletijunaityte commented Aug 26, 2026

Uh oh!

BachirNILU commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

BachirNILU commented Aug 26, 2026 •

edited

Loading