Skip to content

Multi-backend (JAX / PyTorch / array-API) support [integration branch]#892

Draft
jobovy wants to merge 95 commits into
mainfrom
feat/backends
Draft

Multi-backend (JAX / PyTorch / array-API) support [integration branch]#892
jobovy wants to merge 95 commits into
mainfrom
feat/backends

Conversation

@jobovy

@jobovy jobovy commented Jun 4, 2026

Copy link
Copy Markdown
Owner

Long-lived integration branch for making all of galpy backend-agnostic and differentiable (numpy/scipy, JAX, PyTorch, array-API). This PR establishes the integration branch: the parallel potential-migration series and the autodiff / action-angle / df work all target feat/backends, are reviewed individually as PRs into it, and the whole branch is merged to main at the end (rebased on main after the variational feature branch #891 merges, since differentiable C orbit integration needs its state-transition matrix).

Draft: integration branch, not for merge until the full stack lands.

Foundation (in this PR)

  • PR0galpy.backend package: a data-first array-namespace resolver (get_namespace: explicit xp= > forced default > array namespace > context/global > numpy), use() context manager, set_default_backend(). numpy/Python inputs resolve to plain numpy (byte-identical path); jax/torch select jax.numpy / array-api-compat torch. Pilot-migrates PlummerPotential + IsochronePotential to private-layer dispatch. 70 new tests green on numpy/jax/torch (parity, grad-vs-FD, the d(Φ)/dR == −Rforce identity, jit/vmap, dtype, decorator pass-through, precedence).
  • P1 — productionized: config [backend] section, a force mode (use(..., force=True)) so a whole run can be pinned to a backend, a --backend {numpy|jax|torch} pytest option + autouse fixture (numpy = no-op), and an AST convention guard banning bare numpy.* in migrated compute methods. Verified --backend jax re-runs test_energy_jacobi_conservation (Plummer/Isochrone) green.

Series targeting this branch

  • P2.x — potential namespace-swap, parallel fan-out by family (spherical / disk / halo-bar / ellipsoidal / dissipative now in progress; then scipy.special-backed after Pspecial; wrappers last).
  • Pspecial — pure-backend special functions (native-preferring router + isolated fallbacks).
  • Tracks D/E/F — autodiff wrappers + in-backend integrators, action-angle, df/streams.

Design: galpy_backend_autodiff_discussion.md; project log in the galpy-jax repo.

🤖 Generated with Claude Code

@jobovy jobovy marked this pull request as ready for review June 4, 2026 17:08
@codecov

codecov Bot commented Jun 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.72925% with 62 lines in your changes missing coverage. Please review.
✅ Project coverage is 98.98%. Comparing base (7a31c19) to head (de36617).
⚠️ Report is 10 commits behind head on main.

Files with missing lines Patch % Lines
galpy/orbit/orbit_c_ext/integrateFullOrbit.c 74.01% 33 Missing ⚠️
galpy/df/sphericaldf.py 88.34% 19 Missing ⚠️
galpy/orbit/integrateFullOrbit.py 72.22% 10 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #892      +/-   ##
==========================================
- Coverage   99.93%   98.98%   -0.96%     
==========================================
  Files         225      258      +33     
  Lines       37975    42938    +4963     
  Branches      839      899      +60     
==========================================
+ Hits        37951    42501    +4550     
- Misses         24      437     +413     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jobovy added a commit that referenced this pull request Jun 4, 2026
…e test, config docs

- setup.py: remove the 'backend' extra (just array-api-compat — useless on its own);
  keep jax/torch extras (each pulls in array-api-compat).
- _optional_deps.py: correct the array-api-compat comment — there is no vendored
  fallback (removed during coverage cleanup); it is required for the non-numpy
  backends (pulled in by the jax/torch extras) and not needed for numpy-only use.
- Rename tests/test_backend_pilot.py -> tests/test_backend.py; fix the build.yml glob
  to tests/test_backend*.py so it still matches (old _*.py glob would have missed it).
- Document the [backend] config section (default=numpy) in installation.rst + galpyrc.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jobovy jobovy marked this pull request as draft June 5, 2026 01:43
@jobovy jobovy force-pushed the feat/backends branch 3 times, most recently from 30c89cd to 36c92e8 Compare June 13, 2026 03:11
jobovy added a commit that referenced this pull request Jun 14, 2026
…gration PR (#892) from push-to-feat/backends runs

On a push to feat/backends (no PR context) the report job now resolves the
open integration PR (head feat/backends -> base main) and posts/updates the
same sticky 'backend-status' comment on it. Does NOT add a pull_request
trigger for base main, so pushes to feat/backends still run the suite ONCE.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

All-backend test status (jax / torch)

Commit 03b7939cb862214af33a96c8ed1d8792b4842410

Green is achieved via the checked-in xfail-ledger (tests/backend_xfail.txt, applied xfail(strict=False)), so the metric to watch is the shrinking xfail count (burndown), not a raw pass count. A FAIL/ERR is an un-ledgered regression (reds the run). Because the ledger is non-strict, a now-passing ledgered test is a plain pass here (no per-push XPASS); burndown candidates -- in both directions -- are surfaced by the scheduled regen run, which rewrites the ledger from real outcomes. deferred is a separate burndown: tests skipped because they are unrunnable under the backend until the port is vectorized (see tests/backend_slow_skip.txt), e.g. the jax spherical-DF sampling/quadrature tests pending the Track F DF migration.

Overall: jax: 2947 passed · 233 xfail · 263 deferred · 10 exempt | torch: 2548 passed · 885 xfail · 14 deferred · 10 exempt

Ledger size: 1119 entries (jax=233, torch=886).

Test shard jax torch
actionAngle ✅ 137 pass · 68 xfail · 2 deferred ✅ 92 pass · 115 xfail
sphericaldf ✅ 164 pass · 26 xfail · 28 deferred ✅ 45 pass · 173 xfail
conversion + util + misc ✅ 82 pass · 5 xfail · 5 deferred ✅ 64 pass · 28 xfail
SpiralArms + scf + multipole ✅ 190 pass · 5 xfail ✅ 162 pass · 33 xfail
potential (5 split-groups) ✅ 1192 pass · 0 xfail · 16 deferred ✅ 1118 pass · 94 xfail
quantity + coords ✅ 301 pass · 35 xfail ✅ 244 pass · 92 xfail
orbit (energy/Jacobi + from_name) ✅ 98 pass · 0 xfail · 13 deferred · 4 exempt ✅ 94 pass · 17 xfail · 4 exempt
orbit + orbits (main) ✅ 441 pass · 14 xfail · 114 deferred · 6 exempt ✅ 436 pass · 120 xfail · 13 deferred · 6 exempt
evolveddiskdf ✅ 35 pass · 0 xfail ✅ 35 pass · 0 xfail
jeans + dynamfric ✅ 17 pass · 2 xfail · 2 deferred ✅ 12 pass · 8 xfail · 1 deferred
qdf + pv2qdf + streamgapdf_impulse + noninertial ✅ 57 pass · 75 xfail · 1 deferred ✅ 17 pass · 116 xfail
streamgapdf ✅ 30 pass · 0 xfail ✅ 28 pass · 2 xfail
diskdf ✅ 121 pass · 3 xfail · 5 deferred ✅ 129 pass · 0 xfail
streamdf + streamspraydf + streamTrack ✅ 82 pass · 0 xfail · 77 deferred ✅ 72 pass · 87 xfail
Per-shard counts
Test shard backend pass xfail deferred XPASS fail error
actionAngle jax 137 68 2 0 0 0
actionAngle torch 92 115 0 0 0 0
sphericaldf jax 164 26 28 0 0 0
sphericaldf torch 45 173 0 0 0 0
conversion + util + misc jax 82 5 5 0 0 0
conversion + util + misc torch 64 28 0 0 0 0
SpiralArms + scf + multipole jax 190 5 0 0 0 0
SpiralArms + scf + multipole torch 162 33 0 0 0 0
potential (5 split-groups) jax 1192 0 16 0 0 0
potential (5 split-groups) torch 1118 94 0 0 0 0
quantity + coords jax 301 35 0 0 0 0
quantity + coords torch 244 92 0 0 0 0
orbit (energy/Jacobi + from_name) jax 98 0 13 0 0 0
orbit (energy/Jacobi + from_name) torch 94 17 0 0 0 0
orbit + orbits (main) jax 441 14 114 0 0 0
orbit + orbits (main) torch 436 120 13 0 0 0
evolveddiskdf jax 35 0 0 0 0 0
evolveddiskdf torch 35 0 0 0 0 0
jeans + dynamfric jax 17 2 2 0 0 0
jeans + dynamfric torch 12 8 1 0 0 0
qdf + pv2qdf + streamgapdf_impulse + noninertial jax 57 75 1 0 0 0
qdf + pv2qdf + streamgapdf_impulse + noninertial torch 17 116 0 0 0 0
streamgapdf jax 30 0 0 0 0 0
streamgapdf torch 28 2 0 0 0 0
diskdf jax 121 3 5 0 0 0
diskdf torch 129 0 0 0 0 0
streamdf + streamspraydf + streamTrack jax 82 0 77 0 0 0
streamdf + streamspraydf + streamTrack torch 72 87 0 0 0 0

jobovy and others added 9 commits June 25, 2026 11:35
Add galpy.backend: a data-first array-namespace resolver (get_namespace; precedence
explicit xp= > forced default > namespace of the array args > context/global default
> numpy), a use()/set_default_backend() control with a force mode, and a [backend]
config section. numpy/Python inputs resolve to plain numpy so the numpy path is
byte-identical; jax/torch arrays select jax.numpy / array-api-compat torch.

Migrate PlummerPotential and IsochronePotential to the private-layer dispatch
convention (xp = get_namespace(R, z); numpy.<fn> -> xp.<fn>; numpy.pi -> math.pi) as
the reference for the per-family migrations.

Testing: tests/test_backend.py (resolver + reference potentials), tests/test_backend_
conventions.py (AST guard banning bare numpy.* in migrated compute methods), a
--backend pytest option + autouse fixture to re-run under a forced backend, and a
build.yml job running tests/test_backend*.py under jax+torch. jax/torch extras in
setup.py; config docs for [backend].

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…unit parser (#903)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ties (#913)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…#895)

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…-swap (#896)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
jobovy and others added 3 commits June 25, 2026 11:36
…ck E, PR-2 of 2) (#1009)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
… prereq (#1011)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
jobovy and others added 26 commits June 26, 2026 01:41
…f 2) (#1013)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…-2a) (#1014)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…Track E, PR-2b) (#1015)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…#1016)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
#1020)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…eval (AA-grid prereq) (#1021)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…uards (#1017)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ne (#1019)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…grid family) (#1022)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…e) (#1024)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…jax) (#1025)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
… (P0-1) (#1026)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…1027)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…e solve) (#1028)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
#1029)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…numpy_island) (#1038)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…er 5 torch) (#1039)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…fori_loop (under trace) (#1040)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…tion (#1043)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…raft (Track E deriv fix, PR A) (#1047)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…eans) (#1046)

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
…ck E deriv fix, PR D) (#1048)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ian + custom_vjp (Track E deriv fix, PR C) (#1051)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…d helpers (#1052)

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant