Skip to content

ERA-5 model-level fetch stores virtual temperature in t: calcgeoh mutates its argument in place #819

Description

Context

Found while verifying the fix for #812 against archived CDS responses; the measurements below are from that verification (report: https://github.com/s-sasaki-earthsea-wizard/raider-arco-era5/blob/8576fbb0585e42ee05f9a206f16b6dd139d61562/reports/ml_offset_regression_812_fix.md, attribution script and JSON in the same tree).

The defect

utilFcns.calcgeoh converts temperature to virtual temperature in place: t_level *= 1 + 0.609133 * q_level iterates over views of the caller's array. Two call sites leak the mutation:

  1. Fetch (ECMWF._get_from_cds): t is the xarray-cached .values of the open t/q dataset; after calcgeoh the mutated array is written back out via ds_t_q.assign(...).to_netcdf(...). The stored t is exactly t_raw * (1 + 0.609133 * q) — bitwise, when the factor is evaluated in float32 as calcgeoh's in-place multiply does — verified against the raw archived CDS response (up to +1.36 K at the lowest model level in a humid Kanto scene, 2026-03-01 00:00 UTC).
  2. Loader (ECMWF._load_model_level): _calculategeoh mutates the in-memory t before it is assigned to self._t, so downstream saturation vapor pressure (_find_e_from_q) is computed from moisture-inflated temperatures even when the file's t is correct — and doubly inflated for CDS-fetched files, which already store Tv per (1).

Measured consequences (same scene)

  • The stored z cube matches calcgeoh(raw T) bitwise and is not corrupted by this side effect — t is corrupted only as a byproduct of computing it.
  • Re-integrating from the stored t (what the loader does) sits +0.04 to +0.35 m high at the lowest level and reaches +1.4 to +8.4 m around model level 101 (~3.6 km) as the double moisture correction accumulates — the same order as the ERA-5 model levels: loader passes the lowest full-level z where _calculategeoh expects the true surface geopotential (~+10 m column height bias) #812 offset, but humidity-dependent and growing with height rather than a rigid column shift.
  • The resulting wet and hydrostatic refractivity/delay errors (svp and the temperature-dependent terms see inflated t) are not yet quantified.

Proposed direction

Make calcgeoh non-destructive (compute the virtual temperature into a local array) as the primary fix; passing copies at the call sites is the fallback if the in-place behavior is intended for memory reasons. Either way the intended semantics should be pinned by regression tests — none exists today that would catch this:

  • calcgeoh leaves its t argument unchanged;
  • a fetched file's t equals the temperature in the raw CDS response;
  • self._t after _load_model_level is the file's temperature, not a re-inflated one.

Kept separate from #812 on purpose — that fix is orthogonal and already measured.

I'm happy to take this one as well once the #812 PR settles.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions