You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Running identical valid grids in parallel through the Rust bridge (run_suews_multi, the default path for any multi-grid SUEWSSimulation.run() because n_jobs=-1) produces output that differs from the serial run, and differs from one parallel run to the next. The cause is Fortran local variables with initialisers in their declaration, which the language makes implicitly SAVE (static, shared by every thread). At least one such variable sits on the default physics path and is mutated during the call.
Found while writing the serial/parallel equivalence test for #1736; it is independent of the fatal error store.
Reproduction
Sample config, four identical grids (only gridiv differs), two days of forcing, called directly through suews_bridge.run_suews_multi:
single vs single identical
serial grid 0 vs grid 1 identical
serial vs parallel grid 0 71499 cells differ, first at row 7, max abs diff 3553
serial vs parallel grid 3 75514 cells differ, first at row 220, max abs diff 4982
parallel vs parallel grid 0 79228 cells differ, first at row 39, max abs diff 8268
The first row to diverge differs between runs, so this is a race rather than a deterministic offset. The first columns to diverge are QH, QE, QHlumps, QElumps, Evap, State, SMD*; QF and QS follow once the water state has drifted.
LOGICAL:: switch1 =.FALSE., switch2 =.FALSE.INTEGER:: ii, from =2
switch1/switch2 control the step halving of the wet-bulb iteration and are set and cleared inside the loop. Because they are initialised in the declaration they are static: every grid on every thread reads and writes the same two flags, so a grid's iteration takes a step size decided by another grid. The loop also RETURNs with a switch possibly still .TRUE., so even in serial the next call starts from the previous call's state; the reference outputs currently bake that history in.
Other implicitly saved and mutated locals that are off the default path but break under the same conditions:
src/suews/src/suews_phys_estm.f95: converged = .FALSE. (line 232), ESTMStart = 0 (1152), Tair2Set = 0 (1664), the latter two also keyed on Gridiv == 1.
src/suews/src/suews_phys_waterdist.f95: flag_WuM = 1 (1262) is reset only inside a conditional.
src/suews/src/suews_phys_ohm.f95: id_prev = 0 (800), declared but unused.
Compile-time initialisers that are never mutated (atmmoiststab.f95:127,232, snow.f95:1254, stebbs.f95:501-502,750-751, ohm.f95:136-137) are harmless but hide the pattern; the fix should turn them into PARAMETERs or runtime assignments so a future edit cannot reintroduce shared state.
Impact
Every multi-grid run with the default n_jobs=-1 is non-reproducible and differs from the serial result, with per-cell differences of the order of thousands of W m^-2 in the fluxes once the water balance has drifted. The bridge compiles with -frecursive, which only covers variables without initialisers; it does not touch these.
Acceptance criteria
No mutated local in src/suews/src carries a declaration initialiser; each is assigned at the top of the procedure (or made PARAMETER when it is constant).
Serial and parallel output for identical grids are bit-identical, and parallel output is reproducible run to run; regression test added (the equivalence test from Fatal error state leaks between parallel grid workers #1736 can be promoted from expected-failure).
Serial reference outputs are refreshed if removing the cross-call carry-over in LUMPS_cal_AtmMoist changes them, with the change documented as scientific evidence.
Summary
Running identical valid grids in parallel through the Rust bridge (
run_suews_multi, the default path for any multi-gridSUEWSSimulation.run()becausen_jobs=-1) produces output that differs from the serial run, and differs from one parallel run to the next. The cause is Fortran local variables with initialisers in their declaration, which the language makes implicitlySAVE(static, shared by every thread). At least one such variable sits on the default physics path and is mutated during the call.Found while writing the serial/parallel equivalence test for #1736; it is independent of the fatal error store.
Reproduction
Sample config, four identical grids (only
gridivdiffers), two days of forcing, called directly throughsuews_bridge.run_suews_multi:The first row to diverge differs between runs, so this is a race rather than a deterministic offset. The first columns to diverge are
QH,QE,QHlumps,QElumps,Evap,State,SMD*;QFandQSfollow once the water state has drifted.Cause
src/suews/src/suews_util_meteo.f95, subroutineLUMPS_cal_AtmMoist:switch1/switch2control the step halving of the wet-bulb iteration and are set and cleared inside the loop. Because they are initialised in the declaration they are static: every grid on every thread reads and writes the same two flags, so a grid's iteration takes a step size decided by another grid. The loop alsoRETURNs with a switch possibly still.TRUE., so even in serial the next call starts from the previous call's state; the reference outputs currently bake that history in.Other implicitly saved and mutated locals that are off the default path but break under the same conditions:
src/suews/src/suews_phys_estm.f95:converged = .FALSE.(line 232),ESTMStart = 0(1152),Tair2Set = 0(1664), the latter two also keyed onGridiv == 1.src/suews/src/suews_phys_waterdist.f95:flag_WuM = 1(1262) is reset only inside a conditional.src/suews/src/suews_phys_ohm.f95:id_prev = 0(800), declared but unused.Compile-time initialisers that are never mutated (
atmmoiststab.f95:127,232,snow.f95:1254,stebbs.f95:501-502,750-751,ohm.f95:136-137) are harmless but hide the pattern; the fix should turn them intoPARAMETERs or runtime assignments so a future edit cannot reintroduce shared state.Impact
Every multi-grid run with the default
n_jobs=-1is non-reproducible and differs from the serial result, with per-cell differences of the order of thousands of W m^-2 in the fluxes once the water balance has drifted. The bridge compiles with-frecursive, which only covers variables without initialisers; it does not touch these.Acceptance criteria
src/suews/srccarries a declaration initialiser; each is assigned at the top of the procedure (or madePARAMETERwhen it is constant).LUMPS_cal_AtmMoistchanges them, with the change documented as scientific evidence.