Skip to content

Factorized aggregate sinks: COUNT/SUM/MIN/MAX/AND without enumerating the join - #130

Open
MesTTo wants to merge 1 commit into
trueagi-io:mainfrom
MesTTo:wco-ghd-upstream
Open

Factorized aggregate sinks: COUNT/SUM/MIN/MAX/AND without enumerating the join#130
MesTTo wants to merge 1 commit into
trueagi-io:mainfrom
MesTTo:wco-ghd-upstream

Conversation

@MesTTo

@MesTTo MesTTo commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Factorized aggregate sinks — COUNT / SUM / MIN / MAX / AND computed by variable elimination over a hypertree decomposition instead of enumerating the join: O(N^fhtw) rather than O(output), an exponent drop. COUNT runs FAQ InsideOut over the full projection; SUM/MIN/MAX/AND run over the Yannakakis semi-join-reduced single-column domain. The routing gate declines everything outside the proven fragment (grouping, partial projection for COUNT, nonground compounds, disconnected or single-factor bodies), and every routed shape is asserted byte-identical to the enumerate path.

Review shape: the routing is behind an experimental factorized_aggregate feature flag (default off), same as the leapfrog dispatch in #124 this stacks on. The default build runs the stock enumerate path unconditionally; within the feature the MORK_FACTORIZED_AGGREGATE runtime kill switch keeps working. cargo test -p mork --features factorized_aggregate runs the differential suite including a 600-program fuzz across six body shapes; mork bench aggregate (feature build) reports COUNT 473.4x / SUM 389.1x at k=800 with results asserted byte-identical both routes. mork bench all and mork test on the default build are unchanged.

The gate soundness is modeled in Alloy (github.com/MesTTo/alloy-mork): fac17–fac21 pin the aggregation fragment (full projection for COUNT, the semi-join domain for SUM, grouping declined, SUM≠weight-swap), and fac26 pins the feature-gate contract itself.

@MesTTo

MesTTo commented Jul 4, 2026

Copy link
Copy Markdown
Contributor Author

Current numbers on this tip (fe662ff, RUSTFLAGS="-C target-cpu=native", nightly, release):

mork bench aggregate
aggregate COUNT k=800 matches=640000  enumerate 592554us  factorized 1231us  speedup 481.4x
aggregate SUM   k=800 matches=640000  enumerate 547351us  factorized 1203us  speedup 455.0x

The bench runs both routes and asserts the results byte-identical before reporting. The full default suite (mork bench all) and mork test are unchanged, and the same 12-benchmark A/B against main shows byte-identical deterministic counters everywhere (kill switch MORK_FACTORIZED_AGGREGATE=0).

…ged leapfrog join

Rebuilt against `06cdcf3`. The ten commits underneath this were the leapfrog join, which landed
as trueagi-io#143 under `leapfrog.rs`, so they are gone from the branch and what remains is only the
aggregation half.

COUNT / SUM / MIN / MAX / AND computed by variable elimination over a generalized hypertree
decomposition instead of enumerating the join: O(N^fhtw) rather than O(output). COUNT runs FAQ
InsideOut over the full projection; SUM / MIN / MAX / AND run over the Yannakakis semi-join
reduced single-column domain. The routing gate declines everything outside the proven fragment
-- grouping, partial projection for COUNT, nonground compounds, disconnected or single-factor
bodies -- and every routed shape is asserted byte-identical to the enumerate path.

Behind the experimental `factorized_aggregate` feature, default off, the same shape as the
leapfrog dispatch it now sits on. Without it `factorized_aggregate_gate` is a stub returning
`None`, `ghd` is not compiled, and the enumerate sink runs unconditionally.

What the port had to change, since `trueagi-io#124` became `trueagi-io#143` under a different name and a rewritten
internals:

  - `query_multi_ghd` moved out of the join module into `ghd.rs`. Its callback took
    `BTreeMap<(u8, u8), ExprEnv>`; the merged join hands out `&Bindings`, so it takes that.
  - `collect_factor_vars` reads `EncodedTerm`'s variable mask, which the merged parse scan fills
    in, rather than re-walking the term the way the old helper did.
  - `first_subterm_len` is the encoding's own span walk; `is_nonground_compound` is
    `FactorColumn::Term` plus `!is_ground`. Both were helpers the merge dropped.
  - `leapfrog.rs` itself is touched four times only: `run_unify_join_stream`, its `on_tuple`
    field and `original_fact_bytes` widen from `#[cfg(test)]` to
    `#[cfg(any(test, feature = "factorized_aggregate"))]`, because materializing a bag streams a
    bag's own facts out of the join, and `EncodedTerm` gains the two `pub(crate)` readers `ghd`
    needs.

Verified on this tree: `cargo test -p mork` is 27 passed with the feature, 21 with `leapfrog`,
and the same 0 as main by default. `differential/run.py --slow --generated` is 110 ok / 0 failed
both with the feature and without it. `mork bench aggregate`, results asserted identical on both
routes:

  COUNT k=800  matches 640,000  enumerate 392,484us  factorized 1,146us   342.5x
  SUM   k=800  matches 640,000  enumerate 263,255us  factorized 1,009us   260.9x

Lower than the 473.4x / 389.1x posted against `54647135`, because the enumerate path it is
measured against has itself got faster since.
@MesTTo

MesTTo commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

Rebuilt on 06cdcf3; it had been conflicting since #143 merged.

The ten commits this used to stack on were the leapfrog join, so they are gone from the
branch — #143 landed them as leapfrog.rs — and what is left is only the aggregation half.

Porting notes, since the merge renamed the module and rewrote its internals:

  • query_multi_ghd moved out of the join module into ghd.rs, where it belongs. Its callback
    took BTreeMap<(u8, u8), ExprEnv>; the merged join hands out &Bindings, so it takes that.
  • collect_factor_vars now reads the variable mask EncodedTerm already carries rather than
    re-walking the term, and first_subterm_len / is_nonground_compound are expressed from
    Expr::span and EncodedTerm::is_ground. All three helpers the merge dropped.
  • leapfrog.rs is touched in four places only: run_unify_join_stream, its on_tuple field
    and original_fact_bytes widen from #[cfg(test)] to
    #[cfg(any(test, feature = "factorized_aggregate"))], because materializing a bag streams that
    bag's facts out of the join; and EncodedTerm gains two pub(crate) readers.

Still behind the experimental factorized_aggregate flag, default off. Without it
factorized_aggregate_gate is a stub returning None, ghd is not compiled at all, and the
enumerate sink runs unconditionally — cargo test collects the same zero tests as main.

Numbers on this tree, results asserted byte-identical on both routes:

matches enumerate factorized
COUNT k=800 640,000 392,484us 1,146us 342.5x
SUM k=800 640,000 263,255us 1,009us 260.9x

Lower than the 473.4x / 389.1x I posted against 54647135, and the reason is that the enumerate
path it is measured against has itself got faster since.

differential/run.py --slow --generated is 110 ok / 0 failed both with the feature and without;
cargo test -p mork is 27 with the feature, 21 with leapfrog, 0 by default.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant