Skip to content

test(eql-types): SQL-backed property tests for int4 v3 domains#290

Closed
coderdan wants to merge 1272 commits into
eql_v3from
dan/eql-types-proptest
Closed

test(eql-types): SQL-backed property tests for int4 v3 domains#290
coderdan wants to merge 1272 commits into
eql_v3from
dan/eql-types-proptest

Conversation

@coderdan

@coderdan coderdan commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Adds SQL-backed property tests for the eql-types v3 scalar domains — first cut (int4), per issue #237.

Why

The pure-Rust tests prove the types match the catalog and round-trip through serde, but nothing checks them against the actual eql_v3.* SQL functions — the structs and the SQL domains are defined independently, so drift can pass silently.

Approach — a double + property per SQL function (the domain's "trait")

Each eql_v3 scalar domain has a set of generated SQL functions; the doubles mirror that surface:

  • term_compare exercises the extractor (eq_term/ord_term) directly, composed with the term type's own operator (the generated wrapper body).
  • call_bool covers each comparison operator across all three generated overloads(domain,domain), (domain,jsonb), (jsonb,domain).

Each domain's test module wires a property per function against the plaintext oracle: int4_eqeq_term/eq/neq; int4_ord & int4_ord_oreord_term + eq/neq/lt/lte/gt/gte (all overloads).

What's here

  • sqlx feature — jsonb Type/Encode/Decode<Postgres> for the domain types, so a payload binds as a function-double arg and casts to its domain in SQL.
  • proptest feature (depends on sqlx) — shared harness in src/v3/proptest_support.rs: one Tokio runtime + sqlx pool + cipherstash-client cipher (auto mode), a quickcheck Arbitrary (Sample<T>) that encrypts a random i32, and the call_bool/term_compare doubles.
  • int4 property tests co-located in src/v3/int4.rs.
  • mise: test:proptest:prep (build + install standalone v3 SQL into a scratch DB) and test:proptest. CI: a creds-bearing proptest job wired into ci-required.

cipherstash-client emits EQL v2 — so we convert

cipherstash-client only encrypts to the general EQL v2 payload (cipherstash_client::eql::EncryptedPayloadv/i/c + optional hm/bf/ob). Each domain's test module owns the conversion into its specific v3 struct (From<&EncryptedPayload>), reading the client type's typed fields. The v3 domain CHECK accepts extra keys (it only tests presence) — flag separately if we want it tightened.

Auth — no skipping

The cipher is built in auto mode (ZeroKMSBuilder::auto + EnvKeyProvider), exactly like the SQLx fixture generator: AutoStrategy resolves credentials from env (CI: CS_*) or a stash auth login profile. If auth is unavailable the build panics (the test fails) — the tests never silently skip.

Lean build is unaffected

All new deps (sqlx, tokio, quickcheck, cipherstash-client) are optional / feature-gated; the default build and mise run test:crates are unchanged.

Verification

  • Local: cargo build (default + --features sqlx/proptest), cargo test --features proptest --no-run (compiles all 7 property tests), clippy + fmt clean, mise run test:crates green.
  • Full run (the CI proptest job, or a dev box with Postgres + ZeroKMS auth): mise run test:proptest.

Scope / follow-ups

int4 first (issue #237's scope). The harness generalizes: other scalar families (int2/int8/date/timestamptz/text) and bloom-match are follow-ups, each adding its own From<&EncryptedPayload> conversions + per-function properties.

Addresses #237.

coderdan and others added 30 commits May 20, 2026 19:40
eql_v2_encrypted is a composite type and eql_v2.encrypted_operator_class
is its DEFAULT btree opclass, so PostgreSQL invokes the opclass FUNCTION 1
comparator during ANALYZE (and for sort-based GROUP BY / DISTINCT).

FUNCTION 1 was eql_v2.compare, which #211 made strict — it raises
without a Block-ORE `ob` term (U-005). So ANALYZE raised on every
hm-only encrypted column, and autovacuum runs ANALYZE routinely; this
broke silently across managed installs.

Give the opclass its own FUNCTION 1, eql_v2.encrypted_btree_compare: a
total, non-raising 3-way comparator — Block ORE when present, else a
total order on the hm term, else a deterministic payload tie-break.
eql_v2.compare stays strict for the < / > range-operator path.

This also fixes GROUP BY / DISTINCT on a bare encrypted column: with a
raising opclass comparator the planner fell through to PostgreSQL's
built-in record comparison (the composite type's implicit record_ops),
which compares raw ciphertext — so two encryptions of the same plaintext
(same hm, different c) failed to deduplicate.

- src/operators/operator_class.sql: add encrypted_btree_compare, repoint
  FUNCTION 1; the strict eql_v2.compare is untouched.
- operator_class_tests.rs: un-ignore index_behavior_with_different_data_types
  (broken by the strict comparator); add analyze_on_hmac_only_column_does_not_raise.
- CHANGELOG: Fixed entry.
…ted-columns

fix: non-raising btree comparator for eql_v2_encrypted (ANALYZE fix)
Promote the [Unreleased] section to [2.3.0] — 2026-05-20, add a fresh
empty [Unreleased], and update the link references.
The Upgrade notes paragraph said "Six numbered notes ... (U-006)", but
v2.3 has eight — U-007 (typed `->` selector lookup) and U-008 (typed
`stevec_query` containment), both added by #223, were already referenced
by the section's own Added entries. Also drop the mention of the removed
fused `eql_v2.hmac_256(col, '<selector>')` recipe from the U-004 summary.
docs(changelog): correct the 2.3.0 upgrade-notes summary
Surfaces query-only medians across four row-count tiers (10k–10M) in the
README, with a pointer to the cipherstash/benches repository for full
methodology, per-scenario SQL, and EXPLAIN plans.
Fixes #232. Under the v2.3 payload schema an `eql_v2_encrypted` column
admits two mutually exclusive top-level shapes: `EncryptedPayload`
(carries `c` at the root) and `SteVecPayload` (carries `sv` at the root
with the root document ciphertext at `sv[0].c`). `_encrypted_check_c`
previously required `val ? 'c'`, which rejected every insert of a
populated SteVec payload — blocking the entire JSON-encryption path for
callers on `@cipherstash/protect-ffi >= 0.22.0` (where `mode: standard`
is the new default).

The check now passes when either `c` or `sv` is present at the root;
per-element ciphertext validity on `sv` entries continues to be enforced
by the `eql_v2.ste_vec_entry` DOMAIN.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses review feedback from @coderdan on #233.

The pre-existing `ste_vec` and `ste_vec_vast` fixture migrations already
emit real v2.3 SteVec-shape payloads (root `sv`, no root `c`), but they
created their tables without applying `eql_v2.add_encrypted_constraint`
— so the migrations passed even under the pre-fix `_encrypted_check_c`
that rejected every populated SteVec payload. Apply the production
CHECK constraint at fixture-load time so the existing fixture data is
the regression guard: with the pre-fix function, all 10 inserts in
`003_install_ste_vec_data.sql` (and all 500 in `005_install_ste_vec_vast_data.sql`)
raise `Encrypted column missing ciphertext (c) field`; with this PR's
fix they load cleanly. Verified empirically — temporarily reverted the
`_encrypted_check_c` body, re-ran 003 against the resulting bundle, and
confirmed it raises on the first insert.

Also addresses two CodeRabbit nits:
- `CHANGELOG.md`: change the parenthetical link from #232 (the issue) to
  #233 (this PR), per the repo's changelog convention.
- `tests/sqlx/tests/constraint_tests.rs`: switch the
  `check_encrypted_accepts_stevec_payload` insert from `format!`-built
  SQL to a parameterized `$1` bind, matching the rest of the file's
  pattern around dynamic payload values.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses follow-up review feedback from @coderdan on #233 ("I recommend
adding the check for all fixture tables to prevent any regressions").

Extends the regression guard added in cf3d043 to the other fixture
tables whose seed data is the production EQL payload shape:

- `bench` (`tests/sqlx/migrations/007_install_bench_data.sql`) — loaded
  via `create_encrypted_json()` from `bench_data.sql`, every row carries
  the real envelope (`c`, `i`, `v=2`); constraint applied to all three
  encrypted columns. Verified by loading the 10K-row seed against the
  constraint locally.
- `encrypted` in `tests/sqlx/fixtures/like_data.sql` — also seeded from
  `create_encrypted_json()`.

The remaining fixture tables use intentionally-minimal synthetic
payloads that lack required EQL envelope fields and therefore can't
carry the production CHECK without rewriting the fixture data:

- `ore` / `ore_text` — `{"ob": [...]}` only (no `v`, `i`, `c`),
  testing ORE block operator logic in isolation.
- `agg_test` — v=1 payloads from the legacy aggregate-test SQL file.
- `encrypted` in `encrypted_json.sql` / `array_data.sql` /
  `order_by_null_data.sql` — used by tests that add/remove the
  constraint dynamically or copy minimal-shape rows from `ore`.
- `constrained` (`constraint_tables.sql`) — exists specifically to
  exercise Postgres-side UNIQUE/NOT-NULL/CHECK behaviour, including
  NULL inserts that would also collide with a production EQL CHECK.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ncrypted

fix(constraints): accept SteVec payloads in _encrypted_check_c
The eql-2.3.1 tag and GitHub release were cut, but CHANGELOG.md was
not promoted — the #233 SteVec CHECK fix was still under [Unreleased].

Rename [Unreleased] -> [2.3.1] — 2026-05-21, add a fresh empty
[Unreleased], and update the link references (new [Unreleased] compare
URL from eql-2.3.1, new [2.3.1] tag URL).
Add a verify-changelog job to the Release EQL workflow that fails the
release run when CHANGELOG.md (at the tagged commit) has no
`## [<version>]` section — i.e. the release was cut without first
promoting [Unreleased]. This is what happened with eql-2.3.1. Skips
pre-releases, which keep entries under [Unreleased] until the final cut.

It's an independent job: artifact build/publish still proceed (the
release is already published by then), but the workflow goes red so the
omission is visible.

Also fix the README Release EQL badge: it tracked `?branch=main`, but
the workflow effectively never runs on main (it runs on `release` tag
events). The last main run was a cancelled workflow_dispatch from
2025-11-26, so the badge showed "failing" despite every release
succeeding. Switch to `?event=release`.
chore(changelog): promote [Unreleased] to [2.3.1]
Adds the encrypted-int4 fixture generator and benchmark dataset
generation tooling: tasks/fixtures/ and tasks/bench.toml, the
tests/benchmarks/ harness, the CipherStash Proxy docker-compose used to
encrypt fixture data, the int4 fixture install migration, and bench-data
test coverage. Registers the new task files in mise.toml and ignores
local mise credential overrides.
Generator no longer shells out to a CipherStash Proxy container to encrypt
fixture values. `cipherstash-client` 0.35 is now a direct dependency of the
SQLx test crate and the new `fixtures::cipherstash` module owns ZeroKMS
bootstrap (cached in a OnceCell), the index-name -> ColumnConfig mapping,
and the per-value encrypt helper. The Rust fixture owns configuration and
encryption; the DB only owns table structure.

Removes: `tests/docker-compose.proxy.yml`, the `proxy:up`/`proxy:logs`/
`proxy:down` mise tasks, the `mise run proxy:up` step from `test:sqlx`,
and the `restart_proxy_and_wait` / `insert_through_proxy` / `PROXY_PORT`
plumbing in `driver.rs`. The working-table `payload` column is now plain
`jsonb` (no `eql_v2_encrypted` composite) and no `add_search_config` rows
are written to `eql_v2_configuration` during generation.

`mise run test:sqlx` now needs `CS_CLIENT_ACCESS_KEY` (or `CS_CLIENT_ID`
+ `CS_CLIENT_KEY`) and `CS_WORKSPACE_CRN` in the process env so
`AutoStrategy::detect()` / `EnvKeyProvider` can pick them up.
`cipherstash::encrypt_store` now takes a slice and returns a `Vec`, so a
fixture run issues one `encrypt_eql` call regardless of value count
instead of one per value. The 14-row `eql_v2_int4` fixture drops from 14
ZeroKMS round trips to one; the planned bench fixture (~10k rows) gets
the same treatment. Empty input short-circuits before `cipher()` so a
caller with nothing to encrypt does not pay the bootstrap cost.

`driver::insert_direct` now does one batched encrypt then a per-row
INSERT loop — the INSERT is invariant across rows so the SQL string is
lifted out of the loop. The working table is local Postgres and the
per-row execute cost is in microseconds, so batching the INSERTs is not
worth the dynamic-SQL complexity.

Test coverage closes the gap flagged in the PR review:

- Cheap unit tests (no live ZeroKMS): `index_type_for` mapping for
  unique/ore/match plus the unknown-name error path; an empty-batch
  short-circuit test that proves `cipher()` is not reached when there
  is nothing to encrypt (visible because the test runs without CS_*
  env vars).
- Live tests gated by `fixture-gen` + `#[ignore]`: single-value
  round trip, batch length + per-payload identifier check, and a
  distinct-plaintexts → distinct `hm` assertion that mirrors the
  fixture-tests' equality term check at the unit-test layer.
`test:sqlx` copies `release/cipherstash-encrypt.sql` into
`tests/sqlx/migrations/001_install_eql.sql` so the EQL extension is
applied to each per-test database. Without an explicit build dep a
stale release artifact silently ships an old EQL extension — visible
when regression-guard migrations (e.g. 003_install_ste_vec_data.sql)
fail on a `_encrypted_check_c` shape the current source has already
fixed. Declaring `depends = ["build"]` makes the release artifact
fresh on every direct invocation of `test:sqlx`; the top-level
`test.sh` already builds, so CI behaviour is unchanged.
The eql_v2_int4 fixture generator encrypts via cipherstash-client, whose
EnvKeyProvider requires CS_CLIENT_ID + CS_CLIENT_KEY to load the client key.
test:sqlx regenerates fixtures every run, so CI needs that pair — but
f8dfd92 only wired CS_CLIENT_ACCESS_KEY + CS_WORKSPACE_CRN (ZeroKMS auth),
leaving the generator failing with "CS_CLIENT_ID environment variable not
set". Add the client-key pair to the test job env.

Also correct comments in mise.toml, tasks/fixtures.toml, Cargo.toml, and
FIXTURE_SCHEMA.md that framed the two credential pairs as alternatives: auth
(access key + workspace CRN) and key material (client id + key) are distinct
roles and both are required.
feat(fixtures): plug-in framework + eql_v2_int4 reference fixture
Render eql_v2_<T> jsonb-backed domain families (types, functions,
operators, aggregates) from a minimal TOML manifest. Term capabilities
are fixed in terms.py (hm -> equality, ore -> equality+ordering);
output is byte-identical and headed AUTO-GENERATED — DO NOT EDIT.

Hardening: _sql_str() quote-doubler at every SQL interpolation
boundary, distinct-fixture-value validation in spec.py, SQL-identifier
validation of manifest domain names, and codegen:domain / codegen:domain:all
mise tasks. mise run build regenerates all domains every build.

Part of PR #239 (eql_v2_int4 variant family).
Four jsonb-backed domains for encrypted int4: storage-only eql_v2_int4
(every operator blocked), eql_v2_int4_eq (hm; =, <>), and the ordered
pair eql_v2_int4_ord / eql_v2_int4_ord_ore (ore; = <> < <= > >=).

Domain CHECK pins the envelope version (VALUE->>'v' = '2'). Unsupported
operators are error-throwing placeholders carrying an explanatory comment.
Uniform extractors eql_v2.eq_term/ord_term keep functional indexes engaging
without ::jsonb casts. Committed reference baselines guard byte parity.

Part of PR #239.
MIN/MAX aggregates on ord-capable int4 domains route comparison through
the domain's ORE operators — no decryption. min/max are associative, so
the state function doubles as combinefunc; with PARALLEL SAFE sfuncs and
parallel = safe, PostgreSQL can use partial/parallel aggregation on the
large GROUP BY workloads these aggregates exist to serve.

Part of PR #239.
Add blocker_language, blocker_strict, domain_over_domain, and domain_opclass
structural lints enforcing the encrypted-domain footguns (blockers must be
plpgsql and non-STRICT; no domain-over-domain; no opclass on a domain).
pin_search_path recognises the converged extractor/wrapper names intrinsically.

Part of PR #239.
feat: eql-types — JSON Schemas (stacked on #268)
@coderabbitai

coderabbitai Bot commented Jun 16, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: e8578c9c-e3e4-4cd4-b188-6a2646493f84

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch dan/eql-types-proptest

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderdan coderdan force-pushed the dan/eql-types-proptest branch from b1abeb6 to 4f6e131 Compare June 16, 2026 10:25
@coderdan coderdan marked this pull request as draft June 16, 2026 10:29
Comment thread crates/eql-types/src/v3/proptest_support.rs Outdated
Comment thread crates/eql-types/src/v3/proptest_support.rs Outdated
Comment thread crates/eql-types/src/v3/proptest_support.rs Outdated
Comment thread crates/eql-types/src/v3/int4.rs Outdated
@coderdan coderdan force-pushed the dan/eql-types-proptest branch from 4f6e131 to eb4a57d Compare June 17, 2026 00:04
@coderdan

Copy link
Copy Markdown
Contributor Author

Reworked the approach per the feedback (force-pushed eb4a57d).

Per-function doubles mirroring the domain "trait". Instead of one generic compare, the doubles now cover each generated SQL function of a domain:

  • term_compare(...) exercises the extractor (eq_term/ord_term) directly, composed with the term type's own operator (the generated wrapper body);
  • call_bool(func, overload, ...) covers every comparison operator across all three generated overloads — (domain,domain), (domain,jsonb), (jsonb,domain).

Each domain's test module then has a property per function: for int4_eqeq_term, eq, neq; for int4_ord/int4_ord_oreord_term plus eq/neq/lt/lte/gt/gte (all overloads).

Inline comments all addressed (replies on each thread): use cipherstash_client::eql::EncryptedPayload directly (no custom type, no serialize/deserialize round-trip); auth via auto mode + EnvKeyProvider (no stack-profile); panic instead of skipping.

Verified locally: builds (default + sqlx/proptest), --no-run compiles all 7 property tests, clippy + fmt clean, mise run test:crates green. The live run needs Postgres + ZeroKMS auth (the proptest CI job).

@coderdan coderdan force-pushed the dan/eql-types-proptest branch from eb4a57d to d5e7994 Compare June 17, 2026 00:22
Comment thread crates/eql-types/src/v3/int4.rs Outdated
@tobyhede

Copy link
Copy Markdown
Contributor

I am rolling this into the core property test work in PR 275 as additional property test coverage.

Proptest seems to now be the preferred option over quickcheck, but happy to revisit or use both if they provide orthogonal coverage.

Checks the eql-types Rust wire types against the real `eql_v3.*` SQL
functions (issue #237) via the "function double" pattern, modeled on the
cllw-ore crate: encrypt with cipherstash-client, run the generated SQL
function, assert it agrees with the plaintext oracle.

Shared harness in `proptest_support` (feature `proptest`): a `Runner` of 1:1
SQL-function doubles gated by per-capability traits (`EqTerm`/`OrdTerm`,
`Comparable`/`Ordered`); operand-typed comparison overloads (`&x` casts to the
domain, `Jsonb(&x)` stays raw jsonb); `ProptestScalar` (i32/i64) for the
plaintext→`Plaintext` mapping; `eq_comparisons_match`/`ordered_comparisons_match`
for the oracle; and `impl_eq_domain!`/`impl_ord_domain!` macros that stamp the
per-domain v2→v3 conversion + `EncryptableScalar` + `Arbitrary` + capability
markers. The jsonb `Type`/`Encode`/`Decode` macro moves to `sqlx_support`
(feature `sqlx`). With that, a scalar family is ~3 macro invocations + the
readable per-function properties.

- int4 (plaintext i32) and int8 (plaintext i64) families covered: `eq_term`/
  `ord_term` extractors + `eq`/`neq`/`lt`/`lte`/`gt`/`gte` across all three
  generated overloads, co-located per domain (`int4.rs`, `int8.rs`).
- cipherstash-client only emits the general EQL v2 `EncryptedPayload`; the
  conversions read its typed fields to build the narrow v3 structs.
- Auth: auto mode (`CS_*` env or `stash auth login` profile) + a
  FallbackKeyProvider for the client key; the tests panic (fail) without it.
- mise `test:proptest` (pins `QUICKCHECK_TESTS=32`); CI `proptest` job.

Default build stays lean (all new deps optional/feature-gated); `test:crates`
unchanged. Full DB run needs Postgres + ZeroKMS auth (CI); verified locally via
build / `--no-run` / clippy / fmt.

Follow-up filed: #292 (SQL-level mutation testing / fuzzing for these functions).
@coderdan coderdan force-pushed the dan/eql-types-proptest branch from a3ea6f2 to b2ba1e9 Compare June 17, 2026 04:45
@coderdan coderdan marked this pull request as ready for review June 17, 2026 04:47
@coderdan

Copy link
Copy Markdown
Contributor Author

I am rolling this into the core property test work in PR 275 as additional property test coverage.

Proptest seems to now be the preferred option over quickcheck, but happy to revisit or use both if they provide orthogonal coverage.

We've used quickcheck everywhere else (other repos) and its a lot simpler but we can shift to proptest if you think its better. TBH I like the simplicity of the Arbitrary crate.

Also, I feel pretty strongly about keeping these tests right next to the types themselves. Its super hard to reason about what unit tests are present if they're in a different crate.

@auxesis

auxesis commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Approach carried forward in #293 and #303.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants