feat: eql-types — canonical EQL v3 payload types (Rust)

[coderdan]

What this is

crates/eql-types — the canonical Rust types for EQL v3 payloads: one struct per eql_v3 SQL domain, intended as the single source of truth for every tool that produces or consumes EQL payloads (cipherstash-client, protect-ffi, CipherStash Proxy).

This PR is the Rust contract only (serde + serde_json, nothing else). The generated artifacts stack on top:

• feat: eql-types — TypeScript bindings (stacked on #236) #268 — TypeScript bindings (ts-rs) → bindings/v3/
• feat: eql-types — JSON Schemas (stacked on #268) #269 — JSON Schemas (schemars) → schema/v3/, stacked on feat: eql-types — TypeScript bindings (stacked on #236) #268

Review in that order; each is a small, mechanical diff.

The problem

Type information is lost at every hop of the chain:

EQL (SQL + JSON Schema)
  └─ cipherstash-client (Rust)
       └─ protect-ffi (hand-written TypeScript)
            └─ stack / protect / protect-dynamodb

protect-ffi hand-writes its TypeScript; it drifts from the Rust it describes; stack widens it further. By protect-dynamodb, a value's static type tells you almost nothing, so code hand-rolls runtime guards — e.g. term.k === 'ct' && term.hm, which checks a shape EQL never defined. A generated, single-source crate removes the hand-copying.

What's in the crate

Capability-encoded types. One type per SQL domain, mirroring eql-scalars::CATALOG — 23 across six scalars:

Token	Types
int4, int2, int8, date	<T>, <T>Eq, <T>OrdOre, <T>Ord
timestamptz (eq-only)	Timestamptz, TimestamptzEq
text	Text, TextEq, TextMatch, TextOrdOre, TextOrd

Each carries the envelope (v, i, c — exactly what the generated domain CHECKs require) plus its index terms as required fields. Option does not appear: hold an Int4Eq and hm is present, guaranteed by the Rust type. The runtime guessing that produced the protect-dynamodb bug becomes impossible to need. Each type also carries a SQL_DOMAIN const (e.g. "eql_v3.int4_eq").

Reusable term newtypes (src/v3/terms.rs): Ciphertext (c), Hmac256 (hm), OreBlockU64_8_256 (ob), BloomFilter (bf — signed i16, matching smallint[]). They serialize transparently but keep their names in the generated TS/schema outputs.

Drift protection (tests/catalog_parity.rs, dev-dep on eql-scalars):

1. The registry must list every CATALOG domain, in catalog order — append a scalar to the catalog without adding its types here and CI fails.
2. Behavioural required-keys check: a payload carrying exactly the catalog's keys (ENVELOPE_KEYS + Term::term_json_keys) must round-trip identically, and removing any one key must fail deserialization — so a term field can't silently become Option or grow a wrong wire key.

Versioning note. "v3" names the SQL schema generation (eql_v3.*). The JSON envelope stays v: 2 and the wire keys are unchanged from v2 (hm/ob/bf) — the purpose-named rename in docs/plans/eql-payload-scheme-discipline-rfc.md is deferred. The domain CHECKs assert v = '2'.

The crate joins the workspace and the lean mise run test:crates set (fmt, clippy, test — no database).

Design decisions (and reversals from the original prototype)

• No v2.3 tier. The original prototype froze the eql_v2_encrypted v2.3 wire contract alongside the new types. Dropped: EQL 2.3 won't consume these types; the crate targets the eql_v3 surface.
• Explicit structs, no macro. An eql_v3_domain! macro was tried and unrolled: in a protocol crate the struct definitions are the documentation, and the macro's only real guarantee (envelope uniformity) is already covered by the parity tests.
• No discriminated enums. Cross-token is impossible (an int4_eq and int8_eq payload are wire-identical); per-token key-sniffing unions recreate the exact untagged failure mode this crate exists to retire. Consumers read from typed columns and know the domain.
• _ord_ore is its own struct, not an alias of _ord — distinct SQL domain, and the parity test wants 1:1 CATALOG coverage.
• Int4Tagged (self-describing x tag) removed — not part of the v3 wire contract. The idea is preserved in the README as a future direction if a payload revision adds a discriminator.

Follow-ups (not in this stack)

• SQLx integration test: insert each serialized payload into its eql_v3.<domain> column and assert the CHECK accepts it (and rejects the missing-term form).
• New scalars (e.g. numeric) follow the established shape: one token file + registry entries; the parity test enforces the rest.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4dd1ff73-404c-4b75-a96e-bb3242edd195

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch dan/eql-types-crate

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderdan]

@tobyhede heads-up — one commit in this PR (aa46959) reaches outside crates/eql-types into your side of the v3 surface:

ENVELOPE_KEYS (["v", "i", "c"]) is now defined once, in eql-scalars, next to Term. Previously it existed in three unconnected copies: pub(crate) in eql-codegen/src/consts.rs (driving every generated domain CHECK), and inlined literals in eql-types' parity tests and the sqlx harness's Variant::payload_required_keys. A code review flagged that the parity gate was blind to exactly the drift it exists to catch — if you added an envelope key to the codegen const, every regenerated CHECK would change while the eql-types parity tests stayed green against their stale copy.

What changed on your side:

• crates/eql-scalars/src/lib.rs: new pub const ENVELOPE_KEYS, doc'd as cross-schema contract data (same rationale as Term::json_key)
• crates/eql-codegen/src/consts.rs: the pub(crate) const now aliases eql_scalars::ENVELOPE_KEYS — context::domain_block is untouched
• tests/sqlx/src/scalar_domains.rs: payload_required_keys chains from the shared const instead of a literal

Verified: the codegen golden-parity gate passes byte-for-byte (generated SQL is unchanged), cargo test -p eql-scalars -p eql-codegen is green, and cargo check -p eql_tests compiles clean. Net effect is zero behaviour change today; the win is that a future envelope change now propagates to the CHECK generation, the canonical types, and the test harness from one definition.

Shout if you'd rather this hoist land separately from the types crate — it's a self-contained commit and easy to peel off.

[coderdan]

@tobyhede should this be 3 now?!

[coderdan]

@tobyhede — a second, more substantive question out of the type-modelling discussion happening around this PR. This one is about the v3 surface itself, not the crate.

1. = / <> via ore_block_u64_8_256 is not exact for text

Term::Ore claims =,<> for every scalar uniformly. For the full-domain integers that's justified — ORE is lossless there, so ob legitimately doubles as an exact equality term. For text, we need an hm field as well.

2. Are combination domains planned, or out?

DomainSpec.terms is a slice, and the term.rs helpers (operators_for_terms dedupe, extractor_for_operator first-supporting-term-wins) are clearly built for multi-term domains — but every populated CATALOG row carries zero or exactly one term. There's no eq+match text domain (the naming RFC's enc_text_eq_search), no ord+match. The TEXT_DOMAINS comment frames the current shape as test-matrix convenience ("so text still runs the standard ordered matrix"), so: is single-term-per-domain staging, or a decision?

IMHO there should be 3 text types:

• eql_v3_text (no search at all)
• eql_v3_text_eq (adds an hm)
• eql_v3_text_search (hm, ob and bf terms)

[tobyhede]

Could use SQL_SCHEMA

[auxesis]

Test-coverage review — PR #236 (eql-types crate)

The new eql-types crate adds 23 capability-encoded payload structs across 6 tokens. Behavioural serde coverage in v3_conformance.rs is solid for the reference token (int4: roundtrip for every domain, missing-term rejection, wrong-version rejection, unknown-key rejection) plus text_match. The drift gate in catalog_parity.rs pins the domain-name inventory and order against eql-scalars::CATALOG.

The gap: every other token (int2, int8, date, timestamptz, and text's eq/ord/storage domains) has zero behavioural serde tests. catalog_parity.rs only checks sql_domain_static() strings — it never deserializes a payload, so the wire field names (v/i/c/hm/ob/bf) on those 18 structs are exercised by nothing. These are hand-written, copy-pasted structs; a transposed or misspelled field name in int8.rs (hmm for hm) would pass catalog_parity and the build, and only surface as a wire-incompatibility in a downstream consumer. The two inline comments below target the highest-value slices of this.

Note: the module doc in catalog_parity.rs claims the "exhaustive catalog-driven sweep (every domain, every required key)" lives there — but that test does not check keys; per-key strictness is deferred to the stacked schemars change. The comment oversells current coverage.

Additional coverage gaps not posted inline

• Missing-envelope-key negatives untested. rejects_* tests cover term keys (hm/ob/bf) and v, but no test asserts a payload missing c or i is rejected. Same serde mechanism as the term-key tests, but v/i/c are the shared envelope contract and the one thing every domain CHECK asserts — a missing-c / missing-i case mirroring int4_eq_rejects_missing_hmac would close it.
• Instance sql_domain(&self) on real values never called. Every test uses the sql_domain_static() associated fn; catalog_parity calls .domain() only on zero-sized PhantomData handles. The instance method on a deserialized payload value is a one-line wrapper, so this is low priority.
• Changed files consts.rs / scalar_domains.rs need no new tests — ENVELOPE_KEYS is re-sourced from eql-scalars with byte-identical values (["v","i","c"]); existing codegen/matrix tests cover the behaviour.

(Out of scope per review rules: no crypto/security issues noted.)

[auxesis]

Gap: The timestamptz family is the one structurally-distinct token (equality-only — no _ord/_ord_ore, per the deliberate 8-block-ORE limitation in the module doc) yet has no behavioural serde test at all. A roundtrip and a missing-hm rejection would protect both the wire shape and the intentional absence of an ordered domain (the int4 template was copy-pasted to produce this; an accidental extra ob field or a dropped hm would not be caught by catalog_parity, which only checks domain names).

use eql_types::v3::timestamptz::{Timestamptz, TimestamptzEq};

#[test]
fn timestamptz_eq_round_trips() {
    let wire = json!({
        "v": 2,
        "i": { "t": "events", "c": "occurred_at" },
        "c": "mp_base85_ciphertext",
        "hm": "deadbeef"
    });
    let parsed: TimestamptzEq = serde_json::from_value(wire.clone()).unwrap();
    assert_eq!(serde_json::to_value(&parsed).unwrap(), wire);
    assert_eq!(TimestamptzEq::sql_domain_static(), "eql_v3.timestamptz_eq");
}

#[test]
fn timestamptz_eq_rejects_missing_hmac() {
    let no_hm = json!({
        "v": 2,
        "i": { "t": "events", "c": "occurred_at" },
        "c": "mp_base85_ciphertext"
    });
    let r: Result<TimestamptzEq, _> = serde_json::from_value(no_hm);
    assert!(r.is_err());
}

Expected: both pass on current code; timestamptz_eq_round_trips fails if a field is renamed/added, and _rejects_missing_hmac fails if hm is ever made optional.

[coderdan]

Added timestamptz_round_trips_and_enforces_equality_term in tests/v3_conformance.rs — roundtrips both Timestamptz (storage) and TimestamptzEq byte-for-byte, pins the domain names, and asserts a missing-hm payload is rejected (so the equality term cannot silently become optional and a stray ob would fail the strict roundtrip). Thanks — good catch on the one structurally-distinct token. (b3c0cef)

[auxesis]

Gap (lopsided coverage): Only int4 (and text_match) is roundtripped; int2, int8, date, text_eq, text_ord/text_ord_ore, and the storage-only Text carry the same wire field names but are serialized by no test. catalog_parity.rs checks domain names only, so a copy-paste field typo (e.g. hm -> hmm in int8.rs) ships green. A single token-parametrised roundtrip sweep closes the whole class cheaply:

#[test]
fn ordered_tokens_round_trip_each_domain() {
    let eq = |t: &str| json!({"v":2,"i":{"t":t,"c":"x"},"c":"ct","hm":"deadbeef"});
    let ord = |t: &str| json!({"v":2,"i":{"t":t,"c":"x"},"c":"ct","ob":["b0","b1"]});
    use eql_types::v3::{int2::*, int8::*, date::*, text::*};
    macro_rules! rt { ($ty:ty, $w:expr) => {{
        let w = $w; let p: $ty = serde_json::from_value(w.clone()).unwrap();
        assert_eq!(serde_json::to_value(&p).unwrap(), w);
    }}; }
    rt!(Int2Eq, eq("a"));  rt!(Int2Ord, ord("a"));  rt!(Int2OrdOre, ord("a"));
    rt!(Int8Eq, eq("a"));  rt!(Int8Ord, ord("a"));  rt!(Int8OrdOre, ord("a"));
    rt!(DateEq, eq("a"));  rt!(DateOrd, ord("a"));  rt!(DateOrdOre, ord("a"));
    rt!(TextEq, eq("a"));  rt!(TextOrd, ord("a"));  rt!(TextOrdOre, ord("a"));
}

Expected: passes now; fails the instant any non-int4 token's wire field name drifts from the shared envelope/term contract.

[coderdan]

Added non_int4_tokens_round_trip_every_domain — roundtrips storage/_eq/_ord/_ord_ore for int2/int8/date/text (and pins each catalog domain name), so a copy-paste field typo like hm -> hmm now fails the build instead of shipping green. Went slightly beyond the snippet to also cover the storage-only structs you flagged. Also added rejects_missing_envelope_keys from your non-inline note (missing v/i/c rejection). (b3c0cef)