Skip to content

INTERNALS.md

Latest commit

 

History

History
225 lines (182 loc) · 10.4 KB

INTERNALS.md

File metadata and controls

225 lines (182 loc) · 10.4 KB
Raw

Internals

Notes on how floatium is wired internally. Not needed for ordinary use; see the README first.

Introspection

>>> import floatium
>>> floatium.info()
{'patched': True,
 'format_backend': 'zmij',
 'parse_backend': 'ffc',
 'available_format_backends': 'fmt,fmt_opt,ryu_opt,zmij',
 'available_parse_backends': 'fast_float,wuffs,ffc,stock',
 'default_format_backend': 'zmij',
 'default_parse_backend': 'ffc'}

Fields:

  • patched — whether install() is currently active.
  • format_backend / parse_backend — which backend is active now.
  • available_* — what was compiled into this wheel.
  • default_* — what install() picks if no backend is requested.

python -m floatium info prints the same as a quick CLI view, plus the package version.

Backend matrix

All format backends produce bit-identical output. The wheel ships the zmij + ffc pair (the CPython candidate; the defaults were fmt_opt

  • fast_float before 0.16.0). Pick one at build time, or build all of them and switch at install time:
Format backend Algorithm C++?
fmt_opt {fmt} Dragonbox (modes 0/2) + Ryu d2fixed (mode 3) yes
fmt {fmt} Dragonbox + fmt::detail::format_float yes
ryu_opt Ryu d2s (mode 0) + d2exp (mode 2) + d2fixed (mode 3) no
zmij (default) Żmij C port for every mode (shortest + fixed/scientific, bigint above 18 digits) no
Parse backend Algorithm C++?
fast_float Eisel–Lemire 64-bit + bignum fallback (C++ original) yes
wuffs Wuffs parse_number_f64 — Eisel–Lemire + HPD no
ffc (default) ffc.h, the C99 port of fast_float — same pipeline, no C++ no
stock libc strtod via floatium's wrapper no

For a true stock-CPython baseline on the format side, call floatium.uninstall() — there is no stock format backend, because backends speak the dtoa digit-string contract and stock CPython's formatter produces a packaged string directly. The parse side does have a stock backend (libc strtod) because libc happens to match the contract.

zmij or ryu_opt paired with ffc or wuffs gives fully pure-C operation — zero C++ on either the format or parse path:

import floatium
floatium.install(format_backend="zmij", parse_backend="ffc")

The CPython candidate: zmij + ffc

zmij is the configuration floatium proposes for CPython itself, and it is built to be lifted out as-is:

  • One unmodified upstream library, Żmij (MIT or BSL-1.0), vendored as its C port (zmij.c, zmij-c.h). It covers every _Py_dg_dtoa mode natively: shortest round-trip for repr, and correctly rounded fixed / scientific output at any precision for %f / %e / %g (a bigint path takes over above 18 digits). The shortest algorithm is formally verified upstream.
  • One adapter, src/cpython_adapter/zmij_dtoa.c: plain C11 with the exact _Py_dg_dtoa signature and output contract, using nothing from CPython but PyMem_Malloc. It re-reads Żmij's printf-style text into dtoa's digits + decpt shape and implements the one thing printf has no notion of — round(x, -k), i.e. mode 3 with negative ndigits — in ~90 lines of digit arithmetic. Copying it into Python/ next to pystrtod.c is the intended upgrade path.

Two things would make the adapter smaller and are worth proposing to Żmij upstream (see third_party/zmij/README.vendor): a to-decimal entry point in the C API (significand + exponent, the shape Żmij already computes internally) so mode 0 needs no text round-trip, and __asm__ instead of asm so zmij.c compiles under CPython's -std=c11.

The parse side is independent of this: _Py_dg_strtod and _Py_dg_dtoa only share dtoa.c's bignum helpers. A format-only change can land first, but dtoa.c can only be deleted once string-to-float is replaced too. ffc is the candidate for that half: one unmodified single header (ffc.h, the C99 port of fast_float, Apache-2.0 / MIT / BSL-1.0) plus the plain-C adapter src/cpython_adapter/ffc_strtod.c, which only maps ffc_from_chars_double_options onto the _Py_dg_strtod contract (endptr, errno = ERANGE, inf/nan left to _Py_parse_inf_or_nan). It was chosen over Wuffs because Wuffs' parser is an extract of a transpiler-generated file, rejects leading zeros (float("0001.5") would have to fall back), and is ~12× slower on 17-digit inputs; ffc.h matched fast_float bit for bit on 5.3M published corpus lines and 268k exact-midpoint / long-mantissa cases, and tools/fuzz_float_oracle.py passes with it installed. Neither zmij nor ffc.h performs any floating-point arithmetic, so the x87 precision control dtoa.c needs (and the legacy float repr mode it implies) can go too.

The ryu_opt adapter (mode dispatch, FP fast path for %e/%g, banker's rounding for round(x, k) with negative k) is ported from the rye_float companion CPython branch — the in-tree pure-C demonstrator that drops Python/dtoa.c. The Wuffs parser is ported from the dtoa_wuff companion branch.

Build options

Backends are CMake-selectable at build time. The defaults match what the wheel ships (fmt_opt + fast_float); to build other combinations:

# Build only the pure-C path (smaller wheel, no C++ in float I/O):
pip install -e . \
  -C cmake.define.FLOATIUM_FORMAT_BACKEND=zmij \
  -C cmake.define.FLOATIUM_PARSE_BACKEND=ffc

# Build every backend, switch at install time:
pip install -e . \
  -C cmake.define.FLOATIUM_FORMAT_BACKEND=all \
  -C cmake.define.FLOATIUM_PARSE_BACKEND=all

With all, every compiled-in backend can be selected at runtime via:

  • FLOATIUM_FORMAT_BACKEND / FLOATIUM_PARSE_BACKEND env vars (read by the autopatch hook).
  • floatium.install(format_backend=..., parse_backend=...) kwargs.

Format backend routing inside fmt_opt

For each _Py_dg_dtoa call fmt_opt picks per mode:

  • Mode 0 (shortest, drives repr) — fmt's Dragonbox via fmt::detail::dragonbox::to_decimal.
  • Mode 2 (significant digits, drives %e/%g) — fmt's format_float with presentation_type::exp.
  • Mode 3 (digits past decimal, drives %f) — hybrid:
    • If 3 × precision + max(exp2, 0) ≤ 60, stay on fmt. This keeps easy cases (small magnitudes / small precision) on fmt's fast subsegment path, which is ~10 ns faster per call than d2fixed for those inputs.
    • Otherwise route through Ryu d2fixed_buffered_n, which is block-based (9 digits per mulShift_mod1e9 step) and has no cliff.

The exp2 ≤ 60 boundary mirrors fmt's internal cutoff: fmt falls into Dragon4 + bigint once the value's decade plus requested precision exceeds the 19-digit Dragonbox first segment, at which point it is 2–6× slower than stock dtoa. log10(2) ≈ 0.30103 gives exp2 ≈ 3.32 × decade, hence the integer inequality above.

Parse hook (dual-slot patching)

float(s) is intercepted by swapping both PyFloat_Type.tp_new and PyFloat_Type.tp_vectorcall. The vectorcall slot is necessary because CPython 3.13+'s specializing interpreter quickens float(s) to the CALL_BUILTIN_CLASS opcode, which dispatches via tp_vectorcall — bypassing tp_new. Patching only tp_new would silently miss the common direct call shape; the wrapper handles both.

Both wrappers UTF-8-extract the input, strip ASCII whitespace, reject underscores / non-ASCII (those fall through to the saved original), null-terminate into a stack buffer when possible, and call the backend's strtod. Subclasses of str and inputs that the backend can't consume in full also fall through to the saved original tp_new, which preserves CPython's exact semantics on the edges (__float__ dispatch, inf/nan literals, etc.).

Vendored libraries

All four libraries are vendored directly from upstream — pinned versions live in each third_party/<lib>/README.vendor:

Library Upstream Resync
{fmt} fmtlib/fmt tools/sync_fmt.sh [/path/to/fmt]
fast_float fastfloat/fast_float tools/sync_fast_float.sh [/path/to/fast_float]
Ryu ulfjack/ryu tools/sync_ryu.sh [/path/to/ryu]
Wuffs google/wuffs (via the dtoa_wuff extract on eendebakpt/cpython) tools/sync_wuffs.sh [/path/to/cpython0]

Each script defaults to ~/<libname> (Wuffs defaults to ~/cpython0) and copies a fixed file set into third_party/<lib>/. None of the vendored files have algorithmic local modifications. Trivial mods:

  • Ryu: #include "ryu/X" → #include "X" (flattened directory).
  • Wuffs: vendored as a single-header float-parsing subset of the upstream Wuffs release (floatconv_wuffs.h); the extraction details are in third_party/wuffs/README.vendor.

License filenames follow each upstream's convention rather than a single house style (e.g. fast_float ships LICENSE-APACHE uppercase, Ryu ships LICENSE-Apache mixed case). Keeping the upstream casing makes the bytes copied here trivially comparable to what's on each project's GitHub.

The C++ adapter shims that bridge these libraries to CPython's formatting/parsing contracts (fmt_dtoa.cc, fmt_opt_dtoa.cc, fast_float_strtod.cc, ryu_opt_dtoa.cc, wuffs_strtod.cc) live in src/cpython_adapter/ as floatium-owned code.

Backend abstraction

The vendored libraries sit behind a narrow C interface (src/common/backend.h):

struct FloatiumFormatBackend { const char *name; floatium_dtoa_fn dtoa; ... };
struct FloatiumParseBackend  { const char *name; floatium_strtod_fn strtod; };

Swapping fmt for Schubfach or double-conversion is a new file in src/backends/ plus a corresponding adapter in src/cpython_adapter/. The backend registry resolves names at install-time, so a wheel built with all can switch backends at runtime without rebuilding.