Skip to content

Commit 8cfaf37

Browse files
committed
build(eql): validate the v4-only range files in docs:validate:documented-sql
The task ran psql over build/render/eql_v3/src only, so the 18 files confined to EQL v4 (the range domains and between) were never validated: they are not rendered into the v3 tree. It now also validates every file of build/render/eql_v4/src with no counterpart in the v3 tree, and fails if it finds none. The other v4 files are the v3 ones under another schema name and are not run twice. The AGENTS paragraph on confined files says so.
1 parent 3f78d8d commit 8cfaf37

2 files changed

Lines changed: 23 additions & 5 deletions

File tree

‎packages/eql/AGENTS.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,7 @@ The SQL source never names its schemas or its payload version literally. It writ
5858

5959
#### Files confined to one EQL version
6060

61-
A source file belongs to every version unless it carries the line `-- ONLY EQL VERSION: N` (the payload version, not the schema name, which the source may not write). Today only the generated range surface does: `src/v3/scalars/<T>/query_<T>_range_types.sql` and `query_<T>_range_functions.sql` hold the range query domains (`eql_v4.query_<T>_range_ore` / `_range_ope`), their `lo` / `hi` accessors and `eql_v4.between(column, range)` (ADR-0009 decision 7), and are `-- ONLY EQL VERSION: 4`. EQL v3 is what cipherstash-client writes, nothing there produces a range, and its SQL must not change. `tasks/build.sh` runs the install order through `eql-codegen select <3|4>` before rendering each version, `eql-codegen render` refuses to render a confined file as the other version, `verify_installer_complete.sh` filters its disk side the same way with its own `grep`, the symbol-order checker runs on the v4 order too (on a copy mapped back to `eql_v3` names), and the Doxygen filter reads a confined file from its own version's rendered tree. `eql-codegen order` still orders every file, so `src/deps-ordered-v3.txt` lists the v4-only files; the per-version lists are `build/render/eql_v<N>/deps-ordered.txt`.
61+
A source file belongs to every version unless it carries the line `-- ONLY EQL VERSION: N` (the payload version, not the schema name, which the source may not write). Today only the generated range surface does: `src/v3/scalars/<T>/query_<T>_range_types.sql` and `query_<T>_range_functions.sql` hold the range query domains (`eql_v4.query_<T>_range_ore` / `_range_ope`), their `lo` / `hi` accessors and `eql_v4.between(column, range)` (ADR-0009 decision 7), and are `-- ONLY EQL VERSION: 4`. EQL v3 is what cipherstash-client writes, nothing there produces a range, and its SQL must not change. `tasks/build.sh` runs the install order through `eql-codegen select <3|4>` before rendering each version, `eql-codegen render` refuses to render a confined file as the other version, `verify_installer_complete.sh` filters its disk side the same way with its own `grep`, the symbol-order checker runs on the v4 order too (on a copy mapped back to `eql_v3` names), the Doxygen filter reads a confined file from its own version's rendered tree, and `docs:validate:documented-sql` runs psql over the v4 render's confined files as well as the v3 render. `eql-codegen order` still orders every file, so `src/deps-ordered-v3.txt` lists the v4-only files; the per-version lists are `build/render/eql_v<N>/deps-ordered.txt`.
6262

6363
Anything that needs the source as real SQL reads the rendered tree, not `src/`: `docs:validate:documented-sql` runs psql over `build/render/eql_v3/src`, and the Doxygen input filter (`tasks/docs/doxygen-filter.sh`) reads each file's rendered copy from `build/render/eql_v3/`. A new tool that parses `src/v3` must do the same, and depend on `build`; do not substitute the placeholders a second way.
6464

@@ -92,7 +92,7 @@ This is the **Encrypt Query Language (EQL)** - a PostgreSQL extension for search
9292

9393
Adding a scalar encrypted-domain type is one row in the Rust catalog `eql-domains::CATALOG` (`crates/eql-domains/src/lib.rs`): a `DomainFamily` giving the type `name` (e.g. `bigint`), its `ScalarKind` (the `kind` field), the `Domain`s mapping each generated (bare) domain name to its fixed index `Term`s (`eq => [Hm]`, `ord`/`ord_ore => [Ore]`), and the `Fixture` value list. Term capabilities are fixed in the `Term` enum's `impl` methods (with unit tests): `Hm` provides equality, and `Ore` provides equality plus ordering. There is no TOML manifest and no Python — the catalog is the source of truth, validated by the compiler (an undefined term or unknown scalar is a compile error) plus catalog `#[test]`s. `mise run build` runs `cargo run -p eql-codegen`, which regenerates the scalar SQL surface into `src/v3/scalars/<T>/` from `CATALOG` at the start of every build; that surface includes supported comparison wrappers plus blockers for native `json` operators that would otherwise be reachable through domain fallback. `cargo run -p eql-codegen` regenerates every type at once (the same call `mise run build` uses; there is no per-type codegen task). The generated `*_types.sql` / `*_functions.sql` / `*_operators.sql` / `*_aggregates.sql` files are **committed in place** under `src/v3/scalars/<T>/` and drift-gated by `mise run codegen:parity` (regenerate in place + `git diff` + untracked check — the same regenerate-and-diff pattern `types:check` uses for the committed bindings). They are still machine-generated: change the catalog and rebuild, never hand-edit (CI fails on drift). The per-type plaintext fixture lists the SQLx matrix consumes are **not** a generated file — they are materialised from each `CATALOG` row at compile time as `eql_domains::INT4_VALUES` / `INT2_VALUES` (the `int_values!` macro) and read directly by `ScalarType::FIXTURE_VALUES`; a Rust source of truth no longer round-trips through a committed generated `.rs`. Generated SQL carries a `-- AUTOMATICALLY GENERATED FILE` header (the project-wide marker `docs:validate` greps on); change the catalog and rebuild, never hand-edit. Hand-written SQL beyond the fixed surface goes in `src/v3/scalars/<T>/<T>_extensions.sql` with no auto-generated header and explicit `-- REQUIRE:` edges — that file IS committed. `json`'s **SteVec** domains never enter this pipeline, but its **scalar storage domain does**: `generate_all` iterates `eql_domains::families_with_scalar_domains()` and renders each family's scalar domains (so `public.eql_v3_json` is generated into `src/v3/scalars/json/`), while `list-types` and the SQLx matrix still iterate `eql_domains::scalar_families()`, which excludes the whole non-scalar `json` family — so `json` stays out of the operator/coverage matrix by construction (a storage-only domain carries no operator matrix), even though its ciphertext-only storage domain now has a generated SQL surface.
9494

95-
**EQL v4 bindings:** the generator writes the scalar domains a second time into `crates/eql-bindings/src/v4/` (`eql_v4_*` names, `v: 4` through `SchemaVersion<4>`, an `inventory.rs`), plus the Stack Encrypt target table `v4/targets.rs`. **Only v4 structs derive Stack Encrypt's `EncryptFrom`/`DecryptInto`**: Stack Encrypt's output is not byte-for-byte compatible with cipherstash-client's, so `encryption_gap` refuses every v3 domain and a codegen test asserts no v3 struct derives either. v4 also has one range query struct per ordered family and order term (`IntegerRangeOreQuery`, `TextRangeOpeQuery`, …: `{v, i, lo, hi}`, each bound the single-bound query twin's term newtype), listed by `v4::all_range_query()`; v3 has none, matching the SQL. v4 has no SteVec bindings, payload enums, TypeScript export or published JSON Schema yet. The hand-written `v4/mod.rs` re-exports v3's term newtypes and `DomainType`, so a change to a term's validation in `v3/terms.rs` changes v4 too; the version boundary is the envelope's `SchemaVersion<V>` (one generic type in `lib.rs`; `v3::SchemaVersion` and `v4::SchemaVersion` are `SchemaVersion<3>` and `SchemaVersion<4>`).
95+
**EQL v4 bindings:** the generator writes the scalar domains a second time into `crates/eql-bindings/src/v4/` (`eql_v4_*` names, `v: 4` through `SchemaVersion<4>`, an `inventory.rs`), plus the Stack Encrypt target table `v4/targets.rs`. **Only v4 structs derive Stack Encrypt's `EncryptFrom`/`DecryptInto`**: Stack Encrypt's output is not byte-for-byte compatible with cipherstash-client's, so `encryption_gap` refuses every v3 domain and a codegen test asserts no v3 struct derives either. v4 also has one range query struct per ordered family and order term (`IntegerRangeOreQuery`, `TextRangeOpeQuery`, …: each bound the single-bound domain's `ob` or `op` value, typed as that twin's term newtype, in a `{v, i, lo, hi}` envelope that Stack Encrypt will need to build once the ordered types are producible), listed by `v4::all_range_query()`; v3 has none, matching the SQL. v4 has no SteVec bindings, payload enums, TypeScript export or published JSON Schema yet. The hand-written `v4/mod.rs` re-exports v3's term newtypes and `DomainType`, so a change to a term's validation in `v3/terms.rs` changes v4 too; the version boundary is the envelope's `SchemaVersion<V>` (one generic type in `lib.rs`; `v3::SchemaVersion` and `v4::SchemaVersion` are `SchemaVersion<3>` and `SchemaVersion<4>`).
9696

9797
The same generator also emits the **Rust payload bindings** under `crates/eql-bindings/src/v3/<family>.rs` (structs + `DomainType` impls) and the `inventory.rs` `all()` list, from the same `CATALOG` — committed with a `// @generated` header (the bindings must exist on a clean clone because `ts-rs`/`schemars` derive the committed TypeScript/JSON Schema off them; the SQL surface is now committed too, so both generated targets are handled consistently — committed in place, drift-gated by regenerate-and-`git diff`). The hand-written `DomainType` trait, the shared newtypes (`SchemaVersion`/`Identifier`/`Ciphertext`/`Hmac256`/`OreBlock256`/`BloomFilter`), the `PhantomData` plumbing, and the architectural module doc (including the non-derivable float-NaN and bool storage-only caveats) stay hand-written in `crates/eql-bindings/src/v3/{mod,domain_type,terms}.rs`. Generated structs carry a catalog-derived struct doc — a summary line (`` `eql_v3.<name>` — <capability>. ``) plus a detail line listing the supported operators and required payload keys, all derived from data the catalog already holds (the capability label, `Term::operators_for_terms`, and `ENVELOPE_KEYS` ++ `Term::term_json_keys` — see `struct_doc_lines` in `crates/eql-codegen/src/bindings.rs`). The required-key list makes structural distinctions visible — e.g. `text_ord` lists `` `v` `i` `c` `hm` `ob` `` (dual-term) versus an integer `integer_ord`'s `` `v` `i` `c` `ob` ``. There are **no per-field docs**: per-field/term semantics live on the shared term newtypes (`terms.rs`, flowing into the TS term files and JSON Schema `$defs`), and non-derivable per-family caveats (float-NaN, bool storage-only) in `mod.rs`. Free-form prose belongs at the **struct level** (a future optional catalog `doc` field emitted as extra `#[doc]` lines), never as per-field docs. JSON Schemas are emitted by **schemars 1.x** as JSON Schema 2020-12. `mise run types:generate` regenerates the Rust bindings (via `eql-codegen bindings`) then the TS/JSON; `mise run types:check` is the committed-reference drift gate — it regenerates and `git diff`s the Rust bindings (`crates/eql-bindings/src/v3` and `src/v4`), `bindings/` and `schema/`, the same regenerate-and-diff pattern `codegen:parity` now uses for the committed SQL surface. Both gates run in CI; `mise run install-hooks` wires a pre-commit hook that runs both locally whenever a commit touches the catalog, the generator, or a committed generated surface. The `json` family's **SteVec** structs (`SteVecDocument`, `SteVecEntry`, `SteVecQuery`, and `SteVecQueryEntry`) are hand-written for the same reason as their SQL — in `crates/eql-bindings/src/v3/json.rs`, symmetric with `src/v3/json/`. Their direct optional fields and `deny_unknown_fields` validation keep the Rust, TypeScript, JSON Schema, and PostgreSQL entry contracts aligned. The family's scalar storage struct **is** generated (a flat `{v,i,c}` struct, `Json`), via `render_scalar_only_bindings(&JSON)` into `crates/eql-bindings/src/v3/json_storage.rs`; the hand-written `json.rs` re-exports it (`pub use crate::v3::json_storage::Json;`) so `super::json::Json` resolves for the generated inventory/payload. Only inventory membership and ordering are catalog-driven: `inventory.rs::all()` iterates the full `CATALOG` (not `scalar_families()`), branching on `Shape` to include the three SteVec structs plus the generated `Json` alongside the other scalar structs.
9898

‎packages/eql/tasks/docs/validate/documented-sql.sh‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,11 @@ PGDATABASE=${PGDATABASE:-postgres}
1414
# version as {{prefix}} / {{eql_version}} placeholders (ADR-0002), which psql
1515
# cannot parse. `mise run build` renders it.
1616
source_directory="build/render/eql_v3/src"
17+
# Files confined to EQL v4 (`-- ONLY EQL VERSION: 4`: the range query domains
18+
# and eql_v4.between) are rendered only into the eql_v4 tree, so they are read
19+
# from there: every file of the v4 tree with no counterpart in the v3 tree. The
20+
# other v4 files are the v3 ones under another name, already validated above.
21+
v4_directory="build/render/eql_v4/src"
1722

1823
echo "Validating SQL syntax for all documented files..."
1924
echo ""
@@ -28,12 +33,25 @@ echo ""
2833
errors=0
2934
validated=0
3035

31-
if [ ! -d $source_directory ]; then
32-
echo "error: source directory does not exist: ${source_directory}"
36+
for dir in "$source_directory" "$v4_directory"; do
37+
if [ ! -d "$dir" ]; then
38+
echo "error: source directory does not exist: ${dir}"
39+
exit 2
40+
fi
41+
done
42+
43+
v4_only=()
44+
while IFS= read -r file; do
45+
if [ ! -f "$source_directory/${file#"$v4_directory"/}" ]; then
46+
v4_only+=("$file")
47+
fi
48+
done < <(find "$v4_directory" -name "*.sql" -not -name "*_test.sql" | sort)
49+
if [ ${#v4_only[@]} -eq 0 ]; then
50+
echo "error: found no file confined to EQL v4 under ${v4_directory}; expected the range surface"
3351
exit 2
3452
fi
3553

36-
for file in $(find $source_directory -name "*.sql" -not -name "*_test.sql" | sort); do
54+
for file in $(find $source_directory -name "*.sql" -not -name "*_test.sql" | sort) "${v4_only[@]}"; do
3755
echo -n "Validating $file... "
3856

3957
# Capture both stdout and stderr

0 commit comments

Comments
 (0)