Symptom
The semi-Lagrangian advection (AdvDiffusionSLCN) gives a different answer at different rank counts. On the LeVeque swirling flow carrying a conservative level set (uw.systems.LevelSetSolver(..., advection="slcn"), StructuredQuadBox 128 by 128, P2, Courant 0.5, 20 steps, reinitialisation every fifth step, no mass correction), the enclosed volume after 20 steps is
| ranks |
volume |
| 1 |
0.0706776688 |
| 2 |
0.0706771414 |
| 4 |
0.0706771438 |
| 8 |
0.0695739287 |
The same pipeline with the Eulerian transport (advection="supg") gives 0.0707207616 at every rank count, to ten digits, so the mesh, the level-set wrapper, the reinitialisation and the integral are partition-independent; the difference is in the trace-back. At 256 by 256 the shift at four and eight ranks is in the sixth digit (0.0706884035 / 0.0706891414 / 0.0706891222).
Reading
A 1e-5 change at two and four ranks is consistent with departure points near partition boundaries being evaluated on a slightly different path (ghost-cell interpolation against owner-cell interpolation); a 1.6% change at eight ranks on a mesh with about 2,000 cells per rank is not, and suggests departure points that leave the local partition being handled wrongly once the partitions are small relative to the step (0.5 cells per step here, so the feet should stay within the ghost layer). Related: #611 (empty-rank path) and the np=4 behaviour recorded around #596.
Reproduction
~/+Simulations/supg_vs_slcn_657/parallel/ls_parallel_timing.py on the feature/levelset-supg branch (#677):
mpirun -n 8 python ls_parallel_timing.py -uw_res 128 -uw_steps 20 -uw_advection slcn
prints the volume; compare with -n 1. The supg run is the control.
Underworld development team with AI support from Claude Code
Symptom
The semi-Lagrangian advection (
AdvDiffusionSLCN) gives a different answer at different rank counts. On the LeVeque swirling flow carrying a conservative level set (uw.systems.LevelSetSolver(..., advection="slcn"), StructuredQuadBox 128 by 128, P2, Courant 0.5, 20 steps, reinitialisation every fifth step, no mass correction), the enclosed volume after 20 steps isThe same pipeline with the Eulerian transport (
advection="supg") gives 0.0707207616 at every rank count, to ten digits, so the mesh, the level-set wrapper, the reinitialisation and the integral are partition-independent; the difference is in the trace-back. At 256 by 256 the shift at four and eight ranks is in the sixth digit (0.0706884035 / 0.0706891414 / 0.0706891222).Reading
A 1e-5 change at two and four ranks is consistent with departure points near partition boundaries being evaluated on a slightly different path (ghost-cell interpolation against owner-cell interpolation); a 1.6% change at eight ranks on a mesh with about 2,000 cells per rank is not, and suggests departure points that leave the local partition being handled wrongly once the partitions are small relative to the step (0.5 cells per step here, so the feet should stay within the ghost layer). Related: #611 (empty-rank path) and the np=4 behaviour recorded around #596.
Reproduction
~/+Simulations/supg_vs_slcn_657/parallel/ls_parallel_timing.pyon thefeature/levelset-supgbranch (#677):prints the volume; compare with
-n 1. Thesupgrun is the control.Underworld development team with AI support from Claude Code