Skip to content

Commit 3eaeeb8

Browse files
committed
feat(eql-bindings): query EQL v4 types through Stack Encrypt's typed query chain
Rust could store an EQL v4 type but query it only through a data plan: no v4 type implemented `op::Answers`, so `cipher.query(op::Eq, v).on(&plan)` did not compile on a plan for `TextEq`, while the Go SDK already had an `Eq` method per equality type and `Matches` with `Verify` on `TextMatch` (#1216). This gives Rust the same operators, with the same bytes (#1238). eql-codegen writes into `v4/answers.rs`, for each producible type with a query twin: `op::LabelledTarget` (it is sealed under the column a two-segment label names), the twin's assembly, and one `op::Answers` impl per operator. The operators come from the type's index keys, the input the data path's `TargetDescriptor::operators` reads: `"eq"` answers `Eq`, `"match"` answers `Matches`. It stops on any other index, or more than one, so making an ordered type producible has to extend the rule. The same rows give `ANSWERS`, which the `typed_queries` test holds to `TargetDescriptor::operators()` for every type. The file sits beside the target table rather than in it: `v4/targets.rs` is built on Stack Encrypt's `dynamic` layer and needs the `targets` feature, while a typed query needs neither, so `answers.rs` needs only `stack-encrypt`. The index each impl names is `TwinIndex<I, Q>`: Stack Encrypt's own index `I` (the same spec and derivation), whose query term is written into the twin `Q` with the column the query's label names, by the encodings the stored values use. It answers only for the twin's own plaintext (`QueryTwin::Plaintext`), so no plan can send one type's twin for another type's value, and its `check_label` refuses a label that names no column and any extension before any keyset is loaded: an extended query would be derived under more than the column the twin stores, and match nothing. Both refusals are a `QueryLabelError` in `Error::Other`, so a caller matches the variant rather than the message. `TextMatch` returns `MatchTwin<Q>`, which serializes as the twin and keeps the query's tokens for `verify` and `target_limit` under `EqlMatch`. `Hmac256` and `BloomFilter` now declare their indexes, so a type holding one declares the index a query is checked against, and `EqlMatch` implements `MatchCase`. Tests: every typed query is the data path's `query_term` bytes and domain (the float values the semantic baseline folds included), with the data path's inputs built as frames of Rust values; `TextMatch`'s `verify` and `target_limit` agree with the data path's; short text, a non-column label and an extended query are refused before any keyset is loaded; and compile-fail doctests for `op::Lt` on `TextEq`, `op::Matches` on `IntegerEq`, `.via(Equality)` and another type's `TwinIndex` on a `TextEq` plan, and `verify` on an exact term. The new module's example runs as a test in the encryption crate.
1 parent 2215991 commit 3eaeeb8

15 files changed

Lines changed: 1601 additions & 34 deletions

File tree

‎.changeset/eql-v4-bindings.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,6 @@
22
'@cipherstash/eql': minor
33
---
44

5-
**The `eql-bindings` crate has an `eql_bindings::v4` module, with the `stack-encrypt` feature: payload types for the EQL v4 domains.** EQL v4 is the v3 SQL installed under a second name (ADR-0002), for values written by Stack Encrypt, so the module exists only with that feature and its types derive Stack Encrypt's traits without a further gate. The module is generated from the same catalog as `v3`, with `eql_v4_*` domain names and `"v": 4`. `SchemaVersion` is now generic over the version it admits: `v3::SchemaVersion` is `SchemaVersion<3>` and `v4::SchemaVersion` is `SchemaVersion<4>`, each refusing the other's payloads, and plain `SchemaVersion` still means v3. Only v4 types derive Stack Encrypt's `EncryptFrom` and `DecryptInto`, and the Stack Encrypt target table (now `eql_bindings::v4::targets`) names v4 domains. Besides `TextEq`, Stack Encrypt now produces the storage-only types `Text`, `Integer`, `Bigint`, `Boolean`, `Real` and `Double`, and `TextMatch` (3-grams, `k = 6`, a 2048-bit filter, pinned as `eql_bindings::encryption::EqlMatch`; text shorter than three characters is refused, stored or queried, and only `stored @@ query` matches until #1171 is fixed), from the Rust primitives `i32`, `i64`, `bool`, `f32` and `f64`; the Go SDK's `encrypt/eql` table marks them producible. Stack Encrypt also produces the number equality types `IntegerEq`, `BigintEq`, `RealEq` and `DoubleEq`, whose `=` compares as PostgreSQL compares the plain types: an integer has one term whatever its width, and for floats `-0.0 = 0.0` and `NaN = NaN` (ADR-0002's semantic baseline). Every type it cannot produce yet names the issue that tracks its blocker. The by-name target entry points (`eql_bindings::encryption::targets::encrypt`, `decrypt` and `query`) exchange the plaintext as Stack Encrypt's `dynamic::ValueFrame`, one value as its transport frame, rather than a value tree. They, and the generated `v4::targets` table, need the new `targets` feature, which turns on `stack-encrypt` and Stack Encrypt's `dynamic` feature. `stack-encrypt` alone no longer builds Stack Encrypt's `dynamic` layer, so a Rust program using the typed v4 types through `encrypt_as` doesn't compile it.
5+
**The `eql-bindings` crate has an `eql_bindings::v4` module, with the `stack-encrypt` feature: payload types for the EQL v4 domains.** EQL v4 is the v3 SQL installed under a second name (ADR-0002), for values written by Stack Encrypt, so the module exists only with that feature and its types derive Stack Encrypt's traits without a further gate. The module is generated from the same catalog as `v3`, with `eql_v4_*` domain names and `"v": 4`. `SchemaVersion` is now generic over the version it admits: `v3::SchemaVersion` is `SchemaVersion<3>` and `v4::SchemaVersion` is `SchemaVersion<4>`, each refusing the other's payloads, and plain `SchemaVersion` still means v3. Only v4 types derive Stack Encrypt's `EncryptFrom` and `DecryptInto`, and the Stack Encrypt target table (now `eql_bindings::v4::targets`) names v4 domains. Besides `TextEq`, Stack Encrypt now produces the storage-only types `Text`, `Integer`, `Bigint`, `Boolean`, `Real` and `Double`, and `TextMatch` (3-grams, `k = 6`, a 2048-bit filter, pinned as `eql_bindings::encryption::EqlMatch`; text shorter than three characters is refused, stored or queried, and only `stored @@ query` matches until #1171 is fixed), from the Rust primitives `i32`, `i64`, `bool`, `f32` and `f64`; the Go SDK's `encrypt/eql` table marks them producible. Stack Encrypt also produces the number equality types `IntegerEq`, `BigintEq`, `RealEq` and `DoubleEq`, whose `=` compares as PostgreSQL compares the plain types: an integer has one term whatever its width, and for floats `-0.0 = 0.0` and `NaN = NaN` (ADR-0002's semantic baseline). Every type it cannot produce yet names the issue that tracks its blocker. A Rust caller queries those types through Stack Encrypt's typed query chain: `cipher.query(op::Eq, v).on(&plan)` on a plan for `TextEq` or a number equality type returns the type's query twin, and `cipher.query(op::Matches, text).on(&plan)` on a plan for `TextMatch` returns a `MatchTwin<TextMatchQuery>` with `verify` and `target_limit`, the same bytes and answers as the data path the Go SDK uses (`eql_bindings::encryption::query`; an operator a type does not answer does not compile). The by-name target entry points (`eql_bindings::encryption::targets::encrypt`, `decrypt` and `query`) exchange the plaintext as Stack Encrypt's `dynamic::ValueFrame`, one value as its transport frame, rather than a value tree. They, and the generated `v4::targets` table, need the new `targets` feature, which turns on `stack-encrypt` and Stack Encrypt's `dynamic` feature. `stack-encrypt` alone no longer builds Stack Encrypt's `dynamic` layer, so a Rust program using the typed v4 types through `encrypt_as` doesn't compile it.
66

77
Nothing a released version shipped has changed. The v3 types, the v3 SQL and the TypeScript package are as they were; the `stack-encrypt` derives on the v3 text types and the `v3::targets` module were added after the last release and are moved to v4 here, not removed from a release.

‎packages/eql/crates/eql-bindings/CHANGELOG.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
99

1010
### Added
1111

12+
- **Rust queries an EQL v4 field through Stack Encrypt's typed query chain**
13+
(`stack-encrypt` feature, #1238). `cipher.query(op::Eq, v).on(&plan)` on a
14+
plan for `TextEq`, `IntegerEq`, `BigintEq`, `RealEq` or `DoubleEq`
15+
(`Plan::value::<S>().context("table/column").encrypt_into::<T>()`)
16+
returns the type's query twin (`TextEqQuery`, …), and
17+
`cipher.query(op::Matches, text).on(&plan)` on a plan for `TextMatch`
18+
returns a `MatchTwin<TextMatchQuery>`: the twin, which it serializes as,
19+
with `verify` and `target_limit` under `EqlMatch`. The query is the bytes
20+
the data path (`stack_encrypt::dynamic::query_term`) derives for the same
21+
field, and `verify` and `target_limit` answer as the data path's do.
22+
eql-codegen writes the `op::Answers` impls into `v4::answers`, which needs
23+
no `dynamic` layer and so not the `targets` feature, from each type's
24+
indexes, by the rule the data path answers an EQL type's operators
25+
with, and stops on a producible type carrying an order index or more
26+
than one index; an operator a type does not answer, or another index
27+
named with `.via(..)`, does not compile. Each answering type is an
28+
`op::LabelledTarget`. The label must name the column, table then column,
29+
and the query must not be extended, or the awaited query is refused
30+
before any keyset is resolved. `eql_bindings::encryption::query` holds
31+
`TwinIndex` (Stack Encrypt's index, its query sent as the twin, for the
32+
twin's own plaintext only), `MatchTwin` and `QueryTwin`.
33+
`Hmac256` and `BloomFilter` now declare their indexes (`EncryptFrom::indexes`),
34+
so a type holding one declares the equality or match index, and `EqlMatch`
35+
implements `op::MatchCase` (case-insensitive).
1236
- **`eql_bindings::v4` needs the `stack-encrypt` feature.** Only Stack
1337
Encrypt produces an EQL v4 value (ADR-0002), so the module is compiled with
1438
that feature alone, and its producible types derive `EncryptFrom` and

‎packages/eql/crates/eql-bindings/README.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -185,6 +185,65 @@ descriptor is `<table>/<column>`, two parts, the same context and descriptor as
185185
a two-segment `stack_encrypt::Label`. A table or column containing `/` is
186186
escaped in the descriptor rather than read as two names.
187187

188+
### Querying an EQL v4 field from Rust
189+
190+
A query names an operator, and a plan for the column's EQL type finds the
191+
index that answers it (Stack Encrypt's ADR-0009). The term that comes back
192+
is the type's query twin, the value the `eql_v4.query_*` domain takes:
193+
194+
```rust,ignore
195+
use eql_bindings::v4::text::{TextEq, TextMatch};
196+
use stack_encrypt::op::{self, Probabilistic, Stats};
197+
use stack_encrypt::Plan;
198+
199+
// A plan for the column's type, under the label that names the column:
200+
// "users/email" is Identifier::for_column("users", "email").
201+
let email_plan = Plan::value::<String>().context("users/email").encrypt_into::<TextEq>().build()?;
202+
let query = cipher.query(op::Eq, "alice@example.com").on(&email_plan).await?; // a TextEqQuery
203+
// WHERE email = $1::eql_v4.query_text_eq, binding sqlx::types::Json(&query)
204+
205+
let name_plan = Plan::value::<String>().context("users/name").encrypt_into::<TextMatch>().build()?;
206+
let query = cipher.query(op::Matches, "ali").on(&name_plan).await?; // a MatchTwin<TextMatchQuery>
207+
let fetch = query.target_limit(10, &Stats::declared(1_000_000)).fetch_or(50);
208+
// WHERE name @@ $1::eql_v4.query_text_match LIMIT $2, then keep the
209+
// decrypted rows for which query.verify(&name) is true.
210+
```
211+
212+
| v4 type | Operator | Query term |
213+
| --- | --- | --- |
214+
| `TextEq` | `op::Eq` | `TextEqQuery` |
215+
| `IntegerEq`, `BigintEq`, `RealEq`, `DoubleEq` | `op::Eq` | `IntegerEqQuery`, `BigintEqQuery`, `RealEqQuery`, `DoubleEqQuery` |
216+
| `TextMatch` | `op::Matches` | `MatchTwin<TextMatchQuery>` |
217+
218+
`MatchTwin` serializes as the twin and adds `verify` and `target_limit`
219+
under `EqlMatch`: a filter can match a row that does not hold the query's
220+
text, so keep only the decrypted rows that verify. An exact query has no
221+
`verify`. An operator a type does not answer does not compile: `op::Lt` on
222+
`TextEq`, or `op::Matches` on `IntegerEq`. Nor does naming another index
223+
with `.via(..)`, such as the bare `Equality`, whose term the type's SQL does
224+
not take. The ordered types answer nothing until the engine produces them.
225+
226+
The label must name the column the value was written under, as two
227+
segments, table then column. A label of any other length, and a query
228+
extended with `.extend(..)` (whose terms would be derived under more than
229+
the column the twin stores), are refused when the query is awaited, before
230+
any keyset is resolved, with an `encryption::query::QueryLabelError` in
231+
`Error::Other`. A label's segments are plain, so a table or column name
232+
that starts with a digit or `-` (which `Identifier::for_column` accepts)
233+
cannot be named on a plan, and such a column cannot be queried this way.
234+
The plan is for queries: write with
235+
`keyset.encrypt_as(&value, column)` and open with `cipher.decrypt_as`, since
236+
an EQL type is sealed under its column rather than the plan's label. The
237+
query is the same bytes the data path derives for a field that names the
238+
type (`stack_encrypt::dynamic::query_term`), which is what the Go SDK sends,
239+
and `tests/encryption/tests/typed_queries.rs` checks that for every type and
240+
operator. The `op::Answers` impls are generated into `src/v4/answers.rs`
241+
from the catalog by the rule the data path uses, and need only the
242+
`stack-encrypt` feature, not `targets`; the supporting types are in
243+
`eql_bindings::encryption::query`, whose
244+
[executable example](src/encryption/query_example.rs) runs with
245+
`cargo test -p eql-encryption-tests --test query_example`.
246+
188247
### Interoperability matrix
189248

190249
The plan asked for the three producer/reader pairings to be proven rather than

‎packages/eql/crates/eql-bindings/src/encryption.rs‎

Lines changed: 53 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,8 @@
8484
//! require::<TextEq>();
8585
//! ```
8686
87+
pub mod query;
88+
8789
// The by-name dispatch over the EQL v4 types (`targets` feature). A plain
8890
// comment: a doc comment here would resolve the module's own links from
8991
// this scope.
@@ -95,7 +97,7 @@ use base64::{engine::general_purpose::STANDARD, Engine as _};
9597
use stack_encrypt::sem::MatchTerm;
9698
use stack_encrypt::sem::{EqualityTerm, MatchConfig, MatchOptions, TermError, Tokenizer};
9799
use stack_encrypt::target::transcode::{Reader, Transcode, Visitor};
98-
use stack_encrypt::target::{self, AeadContext, CallerContext};
100+
use stack_encrypt::target::{self, AeadContext, CallerContext, IndexSpec};
99101
use stack_encrypt::{
100102
CipherText, ContextPiece, Decrypt, DecryptField, DecryptInto, Decryptable, Decryption,
101103
Describe, Description, Encrypt, EncryptFrom, Encryption, Error, IntoContext, MaybeEmpty,
@@ -127,6 +129,20 @@ impl Identifier {
127129
c: column.into(),
128130
})
129131
}
132+
133+
/// The column a two-segment label names, table then column, or `None`
134+
/// for a label of any other length. One reading of a label for the
135+
/// by-name dispatch (`targets::Identifier::from_label`) and the typed
136+
/// query chain ([`query`]).
137+
pub(crate) fn column_of(label: &stack_encrypt::Label) -> Option<NonEmpty<Self>> {
138+
let mut segments = label.segments();
139+
let (Some(table), Some(column), None) = (segments.next(), segments.next(), segments.next())
140+
else {
141+
return None;
142+
};
143+
// A label segment is plain, so never empty.
144+
Self::for_column(table, column).ok()
145+
}
130146
}
131147

132148
impl MaybeEmpty for Identifier {
@@ -235,9 +251,15 @@ pub struct EqualityVisitor;
235251
impl Visitor for EqualityVisitor {
236252
type Value = Hmac256;
237253
fn equality(self, term: EqualityTerm) -> Result<Self::Value, Error> {
238-
Ok(Hmac256(hex::encode(term.as_bytes())))
254+
Ok(hmac256(&term))
239255
}
240256
}
257+
258+
/// EQL's `hm` for an equality term: the HMAC, hex encoded. One encoding for
259+
/// stored values and query twins alike.
260+
fn hmac256(term: &EqualityTerm) -> Hmac256 {
261+
Hmac256(hex::encode(term.as_bytes()))
262+
}
241263
impl Transcode for Hmac256 {
242264
type Visitor = EqualityVisitor;
243265
fn visitor() -> Self::Visitor {
@@ -252,6 +274,11 @@ impl<S: Encrypt + Clone> EncryptFrom<S> for Hmac256 {
252274
{
253275
target::equality().transcode()
254276
}
277+
/// The equality index: a plan for a type holding an `hm` answers
278+
/// `op::Eq` through it.
279+
fn indexes() -> Vec<IndexSpec> {
280+
vec![IndexSpec::Equality]
281+
}
255282
}
256283
impl Decryptable for Hmac256 {
257284
const DECRYPTABLE: bool = false;
@@ -314,22 +341,27 @@ impl Visitor for MatchVisitor {
314341
if terms.positions().is_empty() {
315342
return Err(Error::Term(TermError::EmptyTermText));
316343
}
317-
terms
318-
.positions()
319-
.iter()
320-
.map(|&position| {
321-
i16::try_from(position).map_err(|_| {
322-
Error::Other(
323-
"a match position does not fit a smallint; EQL's bloom filter stores \
324-
filters of at most 32768 bits"
325-
.into(),
326-
)
327-
})
328-
})
329-
.collect::<Result<_, _>>()
330-
.map(BloomFilter)
344+
bloom_filter(terms.positions())
331345
}
332346
}
347+
348+
/// EQL's `bf` for a filter's set positions, each as a `smallint`. One
349+
/// encoding for stored values and query twins alike.
350+
fn bloom_filter(positions: &[u16]) -> Result<BloomFilter, Error> {
351+
positions
352+
.iter()
353+
.map(|&position| {
354+
i16::try_from(position).map_err(|_| {
355+
Error::Other(
356+
"a match position does not fit a smallint; EQL's bloom filter stores \
357+
filters of at most 32768 bits"
358+
.into(),
359+
)
360+
})
361+
})
362+
.collect::<Result<_, _>>()
363+
.map(BloomFilter)
364+
}
333365
impl Transcode for BloomFilter {
334366
type Visitor = MatchVisitor;
335367
fn visitor() -> Self::Visitor {
@@ -350,6 +382,11 @@ impl<S: AsRef<str>> EncryptFrom<S> for BloomFilter {
350382
let terms: Encryption<'s, S, MatchTerm<EqlMatch>, K, CallerContext> = target::matching();
351383
terms.transcode()
352384
}
385+
/// The match index under [`EqlMatch`]: a plan for a type holding a
386+
/// `bf` answers `op::Matches` through it.
387+
fn indexes() -> Vec<IndexSpec> {
388+
vec![IndexSpec::Match(EqlMatch::options())]
389+
}
353390
}
354391
impl Decryptable for BloomFilter {
355392
const DECRYPTABLE: bool = false;

0 commit comments

Comments
 (0)