Factorized aggregate sinks: COUNT/SUM/MIN/MAX/AND without enumerating the join - #130
Factorized aggregate sinks: COUNT/SUM/MIN/MAX/AND without enumerating the join#130MesTTo wants to merge 1 commit into
Conversation
|
Current numbers on this tip (fe662ff, RUSTFLAGS="-C target-cpu=native", nightly, release): The bench runs both routes and asserts the results byte-identical before reporting. The full default suite ( |
…ged leapfrog join Rebuilt against `06cdcf3`. The ten commits underneath this were the leapfrog join, which landed as trueagi-io#143 under `leapfrog.rs`, so they are gone from the branch and what remains is only the aggregation half. COUNT / SUM / MIN / MAX / AND computed by variable elimination over a generalized hypertree decomposition instead of enumerating the join: O(N^fhtw) rather than O(output). COUNT runs FAQ InsideOut over the full projection; SUM / MIN / MAX / AND run over the Yannakakis semi-join reduced single-column domain. The routing gate declines everything outside the proven fragment -- grouping, partial projection for COUNT, nonground compounds, disconnected or single-factor bodies -- and every routed shape is asserted byte-identical to the enumerate path. Behind the experimental `factorized_aggregate` feature, default off, the same shape as the leapfrog dispatch it now sits on. Without it `factorized_aggregate_gate` is a stub returning `None`, `ghd` is not compiled, and the enumerate sink runs unconditionally. What the port had to change, since `trueagi-io#124` became `trueagi-io#143` under a different name and a rewritten internals: - `query_multi_ghd` moved out of the join module into `ghd.rs`. Its callback took `BTreeMap<(u8, u8), ExprEnv>`; the merged join hands out `&Bindings`, so it takes that. - `collect_factor_vars` reads `EncodedTerm`'s variable mask, which the merged parse scan fills in, rather than re-walking the term the way the old helper did. - `first_subterm_len` is the encoding's own span walk; `is_nonground_compound` is `FactorColumn::Term` plus `!is_ground`. Both were helpers the merge dropped. - `leapfrog.rs` itself is touched four times only: `run_unify_join_stream`, its `on_tuple` field and `original_fact_bytes` widen from `#[cfg(test)]` to `#[cfg(any(test, feature = "factorized_aggregate"))]`, because materializing a bag streams a bag's own facts out of the join, and `EncodedTerm` gains the two `pub(crate)` readers `ghd` needs. Verified on this tree: `cargo test -p mork` is 27 passed with the feature, 21 with `leapfrog`, and the same 0 as main by default. `differential/run.py --slow --generated` is 110 ok / 0 failed both with the feature and without it. `mork bench aggregate`, results asserted identical on both routes: COUNT k=800 matches 640,000 enumerate 392,484us factorized 1,146us 342.5x SUM k=800 matches 640,000 enumerate 263,255us factorized 1,009us 260.9x Lower than the 473.4x / 389.1x posted against `54647135`, because the enumerate path it is measured against has itself got faster since.
d4ad3b4 to
396d233
Compare
|
Rebuilt on The ten commits this used to stack on were the leapfrog join, so they are gone from the Porting notes, since the merge renamed the module and rewrote its internals:
Still behind the experimental Numbers on this tree, results asserted byte-identical on both routes:
Lower than the 473.4x / 389.1x I posted against
|
Factorized aggregate sinks — COUNT / SUM / MIN / MAX / AND computed by variable elimination over a hypertree decomposition instead of enumerating the join: O(N^fhtw) rather than O(output), an exponent drop. COUNT runs FAQ InsideOut over the full projection; SUM/MIN/MAX/AND run over the Yannakakis semi-join-reduced single-column domain. The routing gate declines everything outside the proven fragment (grouping, partial projection for COUNT, nonground compounds, disconnected or single-factor bodies), and every routed shape is asserted byte-identical to the enumerate path.
Review shape: the routing is behind an experimental
factorized_aggregatefeature flag (default off), same as the leapfrog dispatch in #124 this stacks on. The default build runs the stock enumerate path unconditionally; within the feature theMORK_FACTORIZED_AGGREGATEruntime kill switch keeps working.cargo test -p mork --features factorized_aggregateruns the differential suite including a 600-program fuzz across six body shapes;mork bench aggregate(feature build) reports COUNT 473.4x / SUM 389.1x at k=800 with results asserted byte-identical both routes.mork bench allandmork teston the default build are unchanged.The gate soundness is modeled in Alloy (github.com/MesTTo/alloy-mork): fac17–fac21 pin the aggregation fragment (full projection for COUNT, the semi-join domain for SUM, grouping declined, SUM≠weight-swap), and fac26 pins the feature-gate contract itself.