Swarm.advection fails on a rank that holds no particles: estimate_dt reshapes an empty (0, 1, dim) velocity array with -1

[lmoresi]

Swarm.advection calls estimate_dt(V_fn) before stepping. On a rank with no local particles the evaluated velocity has shape (0, 1, dim) and

vel = vel.reshape(vel.shape[0], -1)

raises ValueError: cannot reshape array of size 0 into shape (0,newaxis) (numpy cannot infer -1 for a zero-size array). Any parallel run whose swarm is empty on some rank at some step dies there: measured with passive tracers released at the inlet of the DFG cylinder benchmark on four ranks (all ranks but the inlet's are empty at the first step), ~/+Simulations/navier_stokes_supg/cylinder/runs/dfg_supg_r40_C1_tracers.log.

Fix on feature/navier-stokes-supg: give reshape the explicit trailing size, vel.reshape(vel.shape[0], vel.shape[1] * vel.shape[2]). A second, separate point for the same routine: max_magvel is rank-local (the .max() of the local particles, and the empty-rank except returns None), so with step_limit=True the substep count can differ between ranks and the advection's collectives would mismatch; the default step_limit=False does not reach it.

Underworld development team with AI support from Claude Code

[lmoresi]

Correction to the second point: the routine already handles an empty rank after the reshape (the except sets max_magvel to zero and the maximum is allreduced), so the substep count is global. The reshape line is the only defect; it sits before that handling and raises for a zero-size array.

[lmoresi]

More of the same in Swarm.advection on four ranks once the first reshape is fixed: with 40 tracers released on the inlet rank and the other ranks empty at the first step, the ranks fail with ValueError: operands could not be broadcast together with shapes (0,2) (19,2) and ValueError: The truth value of an empty array is ambiguous, and because the exception is rank-local the run then hangs at the next collective (killed by the ten-minute cap). So the empty-rank case needs a pass over the whole advection path, not one line; every branch that touches the local particle arrays should tolerate zero rows, and the collective structure must not depend on the local count. Reproduce: ~/+Simulations/navier_stokes_supg/scripts/dfg_cylinder.py -uw_resolution 20 -uw_t_end 0.02 -uw_tracers 1 under mpirun -n 4.

[lmoresi]

Provenance (git log -S on the two lines): the reshape and the empty-rank except around it both arrived in 1761c6d, 2026-07-05, PR #323 ("fix(swarm): medium Track-0 ... BF-11/14/16/17"). Item BF-16 corrected the (n, 1, dim) shape of the evaluated velocity so that estimate_dt stopped returning None and the substep limit worked again, and it added the guard for a rank with zero particles; but the reshape it introduced sits before that guard and reshape(0, -1) raises for a zero-size array. So since 2026-07-05 any parallel swarm advection with an empty rank at any step has failed at the first line, before the guard could run. Before #323 the empty rank fell through to the swallowed IndexError and advection proceeded.

The two further failures reported above are on the same path after this line; their locations follow from a captured traceback.

[lmoresi]

Captured tracebacks from the four-rank reproduction (zz_probe_np4_attempt1.log, zz_probe_np4.log under ~/+Simulations/navier_stokes_supg/cylinder/runs/), with the reshape fix in place:

• Releasing the tracers with add_particles_with_coordinates(points) on every rank (the recommended rank-identical call) and then calling advection, all four ranks, including the one that holds the 40 particles, fail at the first line of the substep loop, X0.array[:, 0, :] = self._particle_coordinates.data[...] (swarm.py 4964) -> SwarmVariable.__setitem__ (852) -> unpack_uw_data_from_petsc (1346) -> DMSwarm.getField -> ValueError: negative dimensions are not allowed. A negative size from getField means the swarm's local size is negative on every rank after the add, so the defect is in the parallel add-then-advect path for a swarm that started empty, not only in the empty-rank guard.
• Releasing on rank 0 only with empty (0, dim) arrays elsewhere gives the same failure on the first attempt, and on other attempts operands could not be broadcast together with shapes (0,2) (19,2) and The truth value of an empty array is ambiguous further along the same path; the exception is rank-local and the run then hangs at the next collective.
• Serial is unaffected: the identical driver runs 1440 and 2880 steps with tracers released every step.

Reproduce: mpirun -n 4 python dfg_cylinder.py -uw_resolution 20 -uw_t_end 0.02 -uw_checkpoint_every 0 -uw_tracers 1 (the tracer block is about twenty lines: an empty Swarm(mesh), add_particles_with_coordinates each step, advection(v.sym, dt, order=2)).

[lmoresi]

Already fixed on development by #680 (merged 2026-09-04, 37308deb). Verified rather than assumed:

np=1  particles per rank: [6]           estimate_dt -> 0.0665205496514557
np=4  particles per rank: [0, 0, 6, 0]  estimate_dt -> 0.0665205496514557

Three of four ranks hold no particles and the answer is identical to serial.

Swarm.estimate_dt now guards the reshape explicitly:

if vel.size == 0:
    vel = np.zeros((0, vel.shape[2]) if vel.ndim >= 3 else (0,))
else:
    vel = vel.reshape(vel.shape[0], -1)

The second point in the report does not hold on development either. max_magvel is reduced — comm.allreduce(max_magvel, op=MPI.MAX) at swarm.py:5103 — and estimate_dt returns min_dx / max_magvel_glob, so the substeps count in advection is computed from a globally reduced value and cannot differ between ranks. It may well have been rank-local on feature/navier-stokes-supg, where this was found.

What the failure looked like before the fix, since it is worth recording that the symptom is a hang and not the reported ValueError. Reproduced against pre-#680 code under the #678 supervisor:

rank 2 is in estimate_dt()            <- swarm.py:5103, comm.allreduce(MPI.MAX)
ranks 1, 3: <no Python frame>         <- already gone

The empty ranks raised at the reshape, which sat outside the try/except that guards .max(), and left estimate_dt. The populated rank then waited in the allreduce forever. A crash on some ranks becomes a hang on the rest — the usual shape, and the reason an empty-rank guard has to be placed before the first collective rather than around the arithmetic.

Suggest closing this and #679 as fixed by #680, unless the reporter's branch shows something development does not.

Underworld development team with AI support from Claude Code

[lmoresi]

Verified on the reporter's branch with swarm.py taken from development (#680): the four-rank tracer case runs. The two further failures reported above were the driver's, not the swarm's:

• The "negative dimensions" from getField on every rank came from calling advection before the first release, on a Swarm that had never been populated. Such a swarm has a local size of -1 (the DMSwarm sentinel) on every rank, and this fails in serial exactly the same way: Swarm(mesh) then advection(...) raises ValueError: negative dimensions are not allowed on one process. A never-populated swarm cannot be accessed at all; a clear message at that point, or treating it as empty, is the only remaining suggestion from this report, and it is not a parallel matter.
• The "(0,2) against (19,2)" broadcast and the "truth value of an empty array" failures were the driver reusing its timing variable for the inlet-node array (rank-local, empty on ranks without inlet nodes) and then reducing it. Fixed in the driver.

With the first release before the first advection and the driver variable renamed, mpirun -n 4 runs the cylinder with tracers released every step (checkpoints written, forces reduced) on the FMG mesh. Nothing here is a defect the branch shows and development does not. Agree with closing this as fixed by #680, with the never-populated-swarm message as the one open point.

Underworld development team with AI support from Claude Code

https://claude.ai/code/session_018T2VHUGaZiQVJ95qQ4DiSL