Merge pull request #971 from cipherstash/codex/eql-texteq-transcoding

feat(eql): add Rust text encryption and queries

Author: coderdan

.changeset/eql-text-equality-transcoding.md

@@ -0,0 +1,5 @@
+---
+'@cipherstash/eql': minor
+---
+
+**Rust `TextEq` and `TextEqQuery` support Stack Encrypt's target-directed API behind the `stack-encrypt` feature.** The stored table and column identifier supplies the encryption context; native ciphertext and equality terms are transcoded into EQL payloads while Vitamin C owns plaintext encoding. This is a new producer profile with exact string equality, independent of existing cipherstash-client ciphertext and terms. Query operands carry no recoverable ciphertext.

.github/workflows/bench-eql.yml

@@ -64,6 +64,12 @@ on:
       - "packages/eql/docker/**"
       - "packages/eql/README.md"
       - "packages/eql/SUPABASE.md"
+      - "packages/stack-encrypt/**"
+      - "packages/stack-encrypt-derive/**"
+      - "packages/stack-kms/**"
+      - "packages/stack-auth/**"
+      - "packages/stack-profile/**"
+      - "Cargo.toml"
 
   schedule:
     # 02:00 UTC daily

.github/workflows/release-plz.yml

@@ -2,8 +2,9 @@ name: "Release crates (crates.io)"
 
 # Publishes two crates.io release lines via release-plz (crates.io Trusted
 # Publishing over OIDC — no CARGO_REGISTRY_TOKEN): `eql-bindings` from
-# packages/eql (`release`), and `stack-auth` + `stack-profile` from the root
-# workspace (`release-crates`, described above its own jobs at the end).
+# packages/eql (`release`), and the stack-* crates from the root workspace
+# (`release-crates`, described above its own jobs at the end). `release` runs
+# after `release-crates`: see the comment on its `needs:`.
 # Everything from here to `permissions:` is about the EQL line.
 #
 # THE FILENAME CANNOT CHANGE: crates.io binds the publisher to it.
@@ -85,8 +86,23 @@ jobs:
 
   release:
     name: "Release"
-    needs: [eql-armed]
-    if: needs.eql-armed.outputs.armed == 'true'
+    # AFTER `release-crates`, not beside it. eql-bindings' `stack-encrypt`
+    # feature names a stack-encrypt VERSION, and `cargo publish` resolves it
+    # from crates.io, so on a push that releases both, publishing in parallel
+    # races this job's resolution against the other job's upload. Ordered,
+    # the stack-* crates land first and this job's verify finds them.
+    # `!cancelled()`, not the implicit `success()`: `release-crates` is
+    # skipped on every push that moves no stack-* crate, and a skipped
+    # dependency would otherwise skip this job too. Not `always()` either:
+    # that would start this job after somebody cancelled the run, and a
+    # crates.io upload cannot be undone, only yanked (see the same rule on
+    # the publishing jobs in release.yml). A FAILURE of `release-crates`
+    # still stops this one, since the crates it needed did not publish.
+    needs: [eql-armed, release-crates]
+    if: >-
+      !cancelled()
+      && needs.eql-armed.outputs.armed == 'true'
+      && (needs.release-crates.result == 'success' || needs.release-crates.result == 'skipped')
     # GitHub-hosted, matching this repo's other publishing jobs.
     runs-on: ubuntu-latest
     timeout-minutes: 30

.github/workflows/test-eql.yml

@@ -100,6 +100,20 @@ on:
       - "packages/eql/docker/**"
       - "packages/eql/README.md"
       - "packages/eql/SUPABASE.md"
+      # The `rust-crates` job's encryption steps compile these by PATH
+      # (`eql-bindings`' `stack-encrypt` feature): stack-encrypt, its derive
+      # and stack-kms through `path =`, and stack-auth and stack-profile
+      # through `workspace = true` entries the root `Cargo.toml` resolves to
+      # paths. The mise-task derivation in eql-workflow-filters.test.mjs
+      # cannot see them (a task body names a cargo package, not a path); the
+      # manifest walk in eql-suite-ci.test.mjs does, and fails when one of them
+      # is missing from this list.
+      - "packages/stack-encrypt/**"
+      - "packages/stack-encrypt-derive/**"
+      - "packages/stack-kms/**"
+      - "packages/stack-auth/**"
+      - "packages/stack-profile/**"
+      - "Cargo.toml"
   schedule:
     - cron: "0 4 * * *"      # 04:00 UTC daily; full matrix, off the merge path
   merge_group: {}            # inert today (no queue); kept so enabling one works
@@ -193,6 +207,12 @@ jobs:
               - "packages/eql/docker/**"
               - "packages/eql/README.md"
               - "packages/eql/SUPABASE.md"
+              - "packages/stack-encrypt/**"
+              - "packages/stack-encrypt-derive/**"
+              - "packages/stack-kms/**"
+              - "packages/stack-auth/**"
+              - "packages/stack-profile/**"
+              - "Cargo.toml"
 
       # Explicit default (not `|| 'true'`, which trips GitHub's inconsistent
       # treatment of the string 'false'). push/schedule/merge_group/dispatch
@@ -520,6 +540,18 @@ jobs:
       github.event_name != 'pull_request'
       || needs.changes.outputs.relevant == 'true'
     runs-on: blacksmith-16vcpu-ubuntu-2204
+    services:
+      encryption-postgres:
+        image: postgres:17
+        env:
+          POSTGRES_PASSWORD: postgres
+        ports:
+          - 7433:5432
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
     steps:
       - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
         with:
@@ -549,6 +581,23 @@ jobs:
           rustup component add --toolchain "${active_rust_toolchain}" rustfmt clippy
           mise run test:crates
 
+      # The three encryption checks are mise tasks rather than raw cargo here,
+      # so `eql-suite-ci.test.mjs` counts them among the cargo tasks a root
+      # workflow must reach, and `eql-workflow-filters.test.mjs` derives the
+      # paths they read. The database task takes the connection from
+      # EQL_TEST_DATABASE_URL alone; the port is written once more, in the
+      # service's `ports:`, which no expression can reach from here.
+      - name: Test Rust text equality
+        run: mise run test:encryption
+
+      - name: Test Rust text equality with PostgreSQL
+        env:
+          EQL_TEST_DATABASE_URL: "host=localhost port=7433 user=postgres password=postgres dbname=postgres"
+        run: mise run test:encryption:postgres
+
+      - name: Compile Rust text equality for WASI without HTTP
+        run: mise run check:encryption:wasi
+
       # Freshness gate for the eql-types codegen output: regenerate the
       # TypeScript bindings and JSON Schemas and fail if the checked-in
       # copies differ. Reuses the toolchain from the step above.
@@ -562,9 +611,46 @@ jobs:
       # a real path dependency without a version — on the PR rather than at
       # release time. No token needed. `--allow-dirty` tolerates any files the
       # preceding regenerate-and-diff steps leave in the working tree.
+      #
+      # `--all-features`: the verify step compiles the DEFAULT feature set
+      # unless told otherwise, and the `stack-encrypt` feature is off by
+      # default — so this step passed while that feature did not build against
+      # the crates.io stack-encrypt the manifest names. Every other job here
+      # compiles stack-encrypt by PATH, from the tree, and cannot notice; the
+      # packaged manifest has no `path`, so this verify resolves stack-encrypt
+      # from the REGISTRY, which is the build a consumer and docs.rs get.
+      # release-plz does the same at publish time (`publish_all_features` in
+      # packages/eql/release-plz.toml).
+      #
+      # Two ways the verify can fail, and only one of them is this step's job.
+      # A stack-encrypt VERSION that is not on crates.io yet fails resolution
+      # ("failed to select a version") before anything compiles. That is the
+      # ordinary state of a PR that bumps the requirement: the stack-encrypt it
+      # names ships from main, this PR is not on main, and the two cannot be
+      # published in the other order. It is also already loud where it
+      # matters — `cargo publish` refuses it at release time, and
+      # release-plz.yml publishes the stack-* crates before eql-bindings — so
+      # here it is a warning, not a failure, and the registry build is left to
+      # the release. A version that RESOLVES and then does not compile is the
+      # silent case, the one a path build can never see, and that fails.
       - name: Verify eql-bindings packages cleanly for crates.io
         run: |
-          cargo publish -p eql-bindings --dry-run --allow-dirty
+          set -euo pipefail
+          # cargo's own status is kept in `status`: `|| status=$?` reads it
+          # before anything else runs. (`status=$?` on the line after an `if`
+          # reads the if's status, which is 0 when its condition failed, so a
+          # version that resolves and then fails to compile would exit 0.)
+          status=0
+          out="$(cargo publish -p eql-bindings --dry-run --allow-dirty --all-features 2>&1)" || status=$?
+          printf '%s\n' "$out"
+          if [ "$status" -eq 0 ]; then
+            exit 0
+          fi
+          if grep -qF 'failed to select a version for the requirement `stack-encrypt' <<<"$out"; then
+            echo "::warning title=eql-bindings not verified against the registry::The stack-encrypt version eql-bindings names is not on crates.io yet, so the packaged crate could not be built against the registry here. release-plz builds it at publish time (publish_all_features in packages/eql/release-plz.toml) and refuses to publish eql-bindings until that stack-encrypt has shipped."
+            exit 0
+          fi
+          exit "$status"
 
   codegen:
     name: "Encrypted-domain codegen"

AGENTS.md

@@ -85,7 +85,7 @@ Every npm package except EQL lives under `languages/typescript/`: packages in `l
 - `languages/typescript/packages/utils`: Shared config (`utils/config`) and logger (`utils/logger`)
 - `languages/typescript/packages/bench`: Performance / index-engagement benchmarks (private, not published)
 - `languages/typescript/packages/protect-ffi`: Native FFI bindings to the CipherStash Client SDK (`@cipherstash/protect-ffi`) — the Rust core that `languages/typescript/packages/stack` encrypts and decrypts through, absorbed from `cipherstash/protectjs-ffi`. Contains a **nested Cargo workspace** (`crates/`) and six per-platform binary packages under `platforms/*`, each published as `@cipherstash/protect-ffi-<platform>` and linked here via `workspace:*`. Also holds the repo's live FFI integration suite at `integration-tests/` — a private workspace member (`@cipherstash/ffi-integration-tests`) enrolled by its own literal entry in `pnpm-workspace.yaml`, needing Docker and credentials, and deliberately carrying **no `test` script** so `pnpm test` cannot reach it. See the "Working on protect-ffi" notes below before touching it — its default `test` and `build` are deliberately Rust-free.
-- `packages/eql`: The Encrypt Query Language subtree — the SQL bundle that stores and queries encrypted payloads — absorbed from `cipherstash/encrypt-query-language`. **The directory is the subtree root, not the package.** It was imported at a *verbatim prefix* so its repo-root-relative paths (mise tasks, `Doxyfile`, `sync-generated.mjs`) keep resolving, which puts the npm package `@cipherstash/eql` two levels down at `packages/eql/packages/eql` — the same shape as `languages/typescript/packages/protect-ffi/platforms/*`, and enrolled the same way, by an explicit `packages/eql/packages/*` glob in `pnpm-workspace.yaml`. The subtree root deliberately carries no `package.json`. Also contains a **nested Cargo workspace** at `packages/eql/crates/` (`eql-bindings`, published in lockstep with the npm package, plus `eql-domains` / `eql-codegen` / `eql-tests-macros`, which are not), a SQLx test crate at `packages/eql/tests/sqlx`, an ~900-line `mise.toml` task surface, its own `AGENTS.md`, and `docs/`. See the "Working on EQL" notes below before touching it.
+- `packages/eql`: The Encrypt Query Language subtree — the SQL bundle that stores and queries encrypted payloads — absorbed from `cipherstash/encrypt-query-language`. **The directory is the subtree root, not the package.** It was imported at a *verbatim prefix* so its repo-root-relative paths (mise tasks, `Doxyfile`, `sync-generated.mjs`) keep resolving, which puts the npm package `@cipherstash/eql` two levels down at `packages/eql/packages/eql` — the same shape as `languages/typescript/packages/protect-ffi/platforms/*`, and enrolled the same way, by an explicit `packages/eql/packages/*` glob in `pnpm-workspace.yaml`. The subtree root deliberately carries no `package.json`. Also contains a **nested Cargo workspace** at `packages/eql/crates/` (`eql-bindings`, published in lockstep with the npm package, plus `eql-domains` / `eql-codegen` / `eql-tests-macros`, which are not), a SQLx test crate at `packages/eql/tests/sqlx`, an unpublished encryption test crate at `packages/eql/tests/encryption` (real encryption with a fake key source, the executable Rustdoc example at `packages/eql/crates/eql-bindings/src/encryption/example.rs`, plus optional PostgreSQL coverage), an ~900-line `mise.toml` task surface, its own `AGENTS.md`, and `docs/`. See the "Working on EQL" notes below before touching it.
   **Repository ownership:** EQL now lives in `cipherstash/stack`. File and update
   EQL issues in this repository, never in the historical
   `cipherstash/encrypt-query-language` repository. Old upstream issue and PR
@@ -504,6 +504,30 @@ monorepo, which is where the silent failures are.
   changes what the whole EQL CI surface compiles against, and with the skip in
   place the release-stopper is closed without it. If wanted, the pin belongs
   upstream and arrives by subtree pull.
+- **`eql-bindings`' `stack-encrypt` feature names a stack-encrypt VERSION, and
+  the published crate is built against that version, not the tree.** The
+  dependency is `path` + `version`: the path is what every in-tree job
+  compiles, the version is what crates.io resolves once `cargo publish` strips
+  the path. The two agree only if the stack-encrypt on crates.io at that
+  version has the API `src/encryption.rs` uses — and nothing in a path build
+  can tell. stack-encrypt 0.1.0 shipped without `Describe`, and the feature
+  would have published against it while every test passed. Two guards compile
+  the feature FROM THE PACKAGED CRATE against the registry: `cargo publish
+  --dry-run --all-features` in `test-eql.yml`'s `rust-crates` job, and
+  `publish_all_features = true` in `packages/eql/release-plz.toml`. Without
+  `--all-features` both verify the default feature set, which is the feature
+  off. Consequences: the stack-encrypt a bump of that requirement names must
+  reach crates.io before eql-bindings does (the root release-plz line is
+  publish-only — a stack-* version moves by a hand-edited `Cargo.toml`, and
+  cargo refuses a requirement the in-tree path dependency does not satisfy, so
+  the bump lands in the stack-encrypt PR first); on a PR that names a
+  stack-encrypt not yet on crates.io the dry-run step cannot build against
+  the registry and says so as a WARNING rather than failing, because that PR
+  merges before the crate can ship and the case is already loud at publish —
+  only a version that resolves and then does not compile fails the step; and
+  `release-plz.yml`'s `release` job runs after `release-crates` so that on a
+  push releasing both, the crate eql-bindings resolves exists by the time it
+  looks.
 - **`eql-bindings` resolves by path from `languages/typescript/packages/protect-ffi`, never from
   crates.io**, and `scripts/lint-no-eql-registry-pins.mjs` (`pnpm run
   lint:eql-pins`) is what keeps it that way. The two halves of EQL are the Rust

CONTEXT-MAP.md

@@ -4,6 +4,9 @@
 
 - [EQL](./packages/eql/CONTEXT.md) — defines the PostgreSQL objects that store
   and query encrypted values.
+- [Stack Encrypt](./packages/stack-encrypt/CONTEXT.md) — encrypts values under
+  per-value ZeroKMS data keys and derives searchable terms from them, for Rust
+  and, through a WASI guest, Go.
 
 ## Relationships
 
@@ -12,3 +15,6 @@
 - **EQL → ORM adapters**: EQL defines durable encrypted column domains and
   disposable query machinery; adapters create application columns and derived
   search indexes against that surface.
+- **Stack Encrypt → EQL**: Stack Encrypt seals values and renders the ZeroKMS
+  descriptor; EQL's `eql-bindings` transcodes those into EQL payloads, and its
+  `Identifier` (a table and a column) is a two-segment Stack Encrypt `Label`.

Cargo.lock

@@ -44,6 +44,12 @@ includes the optional `stack-encrypt` feature of the MIT-licensed
 `eql-bindings` crate, which depends on `stack-encrypt` and so is usable only
 under the same terms.
 
+The `eql-encryption-tests` Rust crate (`packages/eql/tests/encryption`) is
+an unpublished test harness for these bindings. It uses real encryption with
+a fake key source, executes the Rustdoc text encryption example, and provides
+optional disposable PostgreSQL coverage. It adds no published package or
+production service.
+
 > **Note on publishing.** Every package in the table above, including all
 > seven `@cipherstash/protect-ffi*` packages, all seven `@cipherstash/auth*`
 > packages and `@cipherstash/eql`, is published from this repository by