diff --git a/diagrams/mermaid/01-architecture-deployment.md b/diagrams/mermaid/01-architecture-deployment.md index 3123e00..ea4e160 100644 --- a/diagrams/mermaid/01-architecture-deployment.md +++ b/diagrams/mermaid/01-architecture-deployment.md @@ -2,9 +2,21 @@ > All diagrams below reflect the **real** deployed system (verified against the parent > `CLAUDE.md`, `identity-core-api/CLAUDE.md`, `infra/traefik/config/*.yml`, the prod -> compose files, and the actual Java/Python package trees on `HEAD`, 2026-06-02). +> compose files, and the actual Java/Python package trees on `HEAD`, 2026-06-02; the +> client-side-embedding + puzzle-layer path refreshed 2026-06-12 against `origin/main`). > > **Honesty notes that shaped these diagrams:** +> - **Client-side face embedding (flag-gated, default OFF).** When +> `app.auth.client-side-embedding` is ON, the **browser** computes the Facenet512 +> embedding (onnxruntime-web) and uploads **only the 512-float vector** — the raw face +> image never leaves the device. The server then owns liveness + the pgvector match + +> the accept/reject decision. With the flag OFF (the default) the legacy image-upload +> path runs unchanged: the browser pre-filters and uploads a JPEG; the server extracts +> the embedding. Both paths are drawn where they matter (see §1, the biometric pipelines +> doc, and the puzzle-layer flow). Privacy framing is precise: the raw image is not +> transmitted; the only biometric data sent is a derived, non-invertible 512-d embedding, +> over TLS, stored encrypted at rest (Fernet) — this is **data minimization**, NOT +> "biometric data never leaves the device". > - **Tenant isolation is application-layer**, not Postgres RLS — it is a Hibernate > `@Filter(tenantFilter)` (`@FilterDef` on `User` + 8 tenant-scoped entities) plus a > controller-level `TenantScopeResolver`. There is **no Postgres Row-Level Security**. @@ -40,7 +52,7 @@ C4Container System_Boundary(clients, "Clients") { Container(web, "Web Dashboard", "React 18 + TS + Vite", "app.fivucsas.com — admin & self-service (Hostinger static)") - Container(verify, "Hosted Login + Auth Widget", "React build + nginx", "verify.fivucsas.com — OIDC universal login + step-up MFA iframe (Docker)") + Container(verify, "Hosted Login + Auth Widget", "React build + nginx", "verify.fivucsas.com — OIDC universal login + step-up MFA iframe (Docker). Flag-gated: computes the Facenet512 embedding in-browser (onnxruntime-web) and uploads only the 512-vector") Container(mobile, "Mobile App", "Kotlin Multiplatform / Compose", "Android · iOS · Desktop — AppAuth OIDC") System_Ext(thirdparty, "Third-Party App", "Tenant relying party — redirective OIDC via FivucsasAuth SDK") } @@ -49,7 +61,7 @@ C4Container System_Boundary(backend, "Backend (Docker, Hetzner CX43)") { Container(api, "Identity Core API", "Spring Boot 3.4.7 / Java 21 :8080", "Auth, OAuth2/OIDC, MFA, RBAC, multi-tenancy (Hibernate @Filter), 29 controllers, hexagonal") - Container(bio, "Biometric Processor", "FastAPI / Python 3.12 :8001 — INTERNAL ONLY, X-API-Key", "Face/voice embeddings, liveness, anti-spoof, NFC eMRTD passive-auth (CPU-only, ALLOW_HEAVY_ML=false)") + Container(bio, "Biometric Processor", "FastAPI / Python 3.12 :8001 — INTERNAL ONLY, X-API-Key", "Liveness + pgvector match + decision. Accepts a client-computed 512-vector (/verify-embedding, /enroll-embedding) OR extracts Facenet512 from an image (legacy path). Voice, anti-spoof, puzzle re-score, NFC eMRTD passive-auth (CPU-only, ALLOW_HEAVY_ML=false)") } System_Boundary(data, "Data Stores (Docker volumes)") { @@ -67,7 +79,7 @@ C4Container Rel(tenantdev, thirdparty, "Builds") Rel(web, traefik, "REST /api/v1", "HTTPS") - Rel(verify, traefik, "REST + OAuth2", "HTTPS") + Rel(verify, traefik, "REST + OAuth2; FACE step uploads a 512-d embedding (flag ON) or a JPEG (legacy)", "HTTPS / TLS") Rel(mobile, traefik, "OIDC + REST", "HTTPS") Rel(thirdparty, traefik, "OIDC redirect + /oauth2/token", "HTTPS") diff --git a/diagrams/mermaid/03-biometric-pipelines.md b/diagrams/mermaid/03-biometric-pipelines.md index 0734a12..1b2b318 100644 --- a/diagrams/mermaid/03-biometric-pipelines.md +++ b/diagrams/mermaid/03-biometric-pipelines.md @@ -2,13 +2,27 @@ Accurate, code-grounded Mermaid diagrams of the FIVUCSAS biometric stack. All facts below were read directly from `biometric-processor`, `spoof-detector`, -and `identity-core-api` source on 2026-06-02 — not invented. +and `identity-core-api` source on 2026-06-02 (client-side-embedding + puzzle-layer +paths refreshed 2026-06-12 against `origin/main`) — not invented. **Honesty notes (read these before trusting any marketing slide):** -- **Auth decision is server-side.** The browser pre-filters with MediaPipe and - may send a 128-dim client embedding, but that is **log-only** (D2). The - authoritative 1:1 cosine match runs in `VerifyFaceUseCase` on the server. +- **Auth decision is server-side, embedding can be client-side (flag-gated).** + Two FACE paths exist and both end with the server owning the verdict: + - **Legacy / default (`app.auth.client-side-embedding=false`):** the browser + pre-filters with MediaPipe (and may send a 128-dim *log-only* landmark-geometry + embedding, D2), uploads the **image**, and the server extracts Facenet512 and + runs the 1:1 cosine match in `VerifyFaceUseCase`. + - **Client-side embedding (`app.auth.client-side-embedding=true`):** the browser + computes the **authoritative** Facenet512 embedding in onnxruntime-web and uploads + **only the 512-float vector** (raw image never leaves the device). The server skips + detection/quality/server-Facenet512 and runs the SAME pgvector match + threshold + via `POST /api/v1/verify-embedding` (enrollment: `POST /api/v1/enroll-embedding`). + A precomputed embedding carries **no liveness** — this path MUST be paired with a + liveness factor (passive or the puzzle layer) before it is trusted as a login factor. + Privacy framing: the raw image is not transmitted; the only biometric data sent is a + derived, non-invertible 512-d embedding, over TLS, stored encrypted at rest (Fernet) — + **data minimization**, NOT "biometric data never leaves the device". - **Server-side fingerprint was REMOVED** (it was a SHA-256 placeholder, never a real biometric). `FINGERPRINT` is delivered *only* via WebAuthn / FIDO2 platform authenticators in `identity-core-api` (`FingerprintAuthHandler`). @@ -29,15 +43,24 @@ and `identity-core-api` source on 2026-06-02 — not invented. ## 1a. Face ENROLLMENT pipeline (8 stages in 4 cards) -`POST /enroll` and `/enroll/multi`. Server-authoritative passive liveness + -anti-spoof now run **before** the embedding is persisted +`POST /enroll` and `/enroll/multi` (image path). Server-authoritative passive +liveness + anti-spoof run **before** the embedding is persisted (`ENROLL_LIVENESS_ENABLED=true`, default; multi-image is **fail-CLOSED** per frame). Embedding is written twice: encrypted ciphertext (store-of-record) + plaintext pgvector column (search index). +**Client-side-embedding branch (flag ON):** when +`app.auth.client-side-embedding=true`, the browser computes the Facenet512 +vector and enrollment goes through `POST /api/v1/enroll-embedding` — Stages 1–6 +(detect, quality, landmark, align, server-Facenet512, fusion) already happened in +the browser, so the server jumps straight to **Card 3 → Card 4** (encrypt at rest ++ dual-column persist). Liveness for the enrolling frame is enforced upstream +(Identity Core, e.g. the puzzle layer), since no image reaches bio on this branch. + ```mermaid flowchart TB - IN[/"Image(s) upload
multipart/form-data"/] + IN[/"Image(s) upload
multipart/form-data
POST /enroll · legacy/default"/] + INV[/"512-d vector (JSON)
POST /enroll-embedding
client-side embedding · flag ON
raw image NOT sent"/] subgraph C1["Card 1 — Detect + Quality"] S1["Stage 1 · Detect face
DeepFace + OpenCV (default backend)
AsyncFaceDetector → thread pool
+ circuit breaker, 30s timeout"] @@ -58,7 +81,7 @@ flowchart TB end subgraph C3["Card 3 — Embed + Encrypt"] - S5["Stage 5 · Extract embedding
Facenet-512 (512-dim, L2-normalized)
AsyncEmbeddingExtractor → thread pool"] + S5["Stage 5 · Extract embedding
Facenet-512 (512-dim, L2-normalized)
AsyncEmbeddingExtractor → thread pool
skipped on /enroll-embedding"] S6["Stage 6 · Multi-image fusion (optional)
EmbeddingFusionService
quality-weighted centroid + L2"] S7["Stage 7 · Encrypt at rest
EmbeddingCipher (Fernet / AES-128-CBC
+ HMAC-SHA256), u32-len header"] S5 --> S6 --> S7 @@ -73,52 +96,74 @@ flowchart TB LIVE -->|"live + not spoof"| C2 LIVE -.->|"non-live / spoof
fail-CLOSED → HTTP 400"| REJ[["Reject enrollment"]] C2 --> C3 --> C4 + INV -->|"client already did
Stages 1–6 in-browser;
liveness enforced upstream"| S7 C4 --> OUT[/"BiometricResponse
quality_score, dimension"/] ``` --- -## 1b. Face VERIFICATION (1:1) — sequence +## 1b. Face VERIFICATION (1:1) — sequence (two paths, one verdict owner) + +`FaceVerifyMfaStepHandler` routes on `ClientSideEmbeddingPolicy` (flag +`app.auth.client-side-embedding`, default OFF): + +- **Legacy / default:** browser MediaPipe pre-filter (log-only) → upload **image** → + server detect → embed → cosine match (aged-threshold adaptation) → flag-gated + anti-spoof veto. `/verify` runs a passive-liveness **floor** (score ≥ 0.4) first. +- **Client-side embedding (flag ON):** the browser computes the authoritative + Facenet512 vector and uploads **only the 512-d embedding** to + `POST /api/v1/verify-embedding`; the server runs ONLY the pgvector match + + threshold/decision. This path has **no image and therefore no liveness of its own** — + it MUST be paired with a liveness factor (passive or the puzzle layer §7) upstream. -Client MediaPipe pre-filter (log-only) → server detect → embed → cosine match -with aged-threshold adaptation → flag-gated anti-spoof veto. The `/verify` route -runs a passive-liveness **floor** (score ≥ 0.4) before the match. +In both branches the **server** owns the accept/reject decision (and lockout / rate-limit). ```mermaid sequenceDiagram autonumber - participant B as Browser (MediaPipe FaceLandmarker) - participant API as identity-core-api (FaceVerifyMfaStepHandler) - participant V as bio /verify (verification.py) + participant B as Browser (MediaPipe + onnxruntime-web) + participant API as identity-core-api (FaceVerifyMfaStepHandler + ClientSideEmbeddingPolicy) + participant V as bio verification.py participant LU as CheckLivenessUseCase (UniFace) participant UC as VerifyFaceUseCase participant DB as PostgreSQL + pgvector - B->>B: detect face, quality pre-filter,
passive-liveness gate (0.45) - B->>API: face image (+ optional 128-dim
client_embedding = LOG-ONLY) - API->>V: POST /verify (X-API-Key, user_id, file) - V->>V: validate magic bytes + format - V->>LU: passive liveness (UniFace MiniFASNet) - LU-->>V: is_live, score - alt not is_live OR score < 0.4 - V-->>API: 400 LIVENESS_FAILED - end - V->>UC: execute(user_id, image_path, tenant_id) - UC->>UC: detect → face region → quality (≥50) - UC->>UC: extract Facenet-512 embedding - UC->>DB: find_by_user_id (tenant-scoped) - DB-->>UC: stored embedding (decrypted from ciphertext) - UC->>UC: cosine distance vs threshold 0.45 - UC->>DB: find_created_at - DB-->>UC: enrollment age - UC->>UC: if age > 2 yr → use _AGED (more permissive) - UC-->>V: verified, distance, confidence, threshold - V->>V: anti-spoof helpers (flag-gated):
device-risk, assembler, EAR (single frame) - alt block_reason AND ANTISPOOF_BLOCK_ENFORCE - V-->>API: 403 ANTISPOOF_BLOCKED {reason} - else allowed - V->>DB: BackgroundTask: log client-embedding
observation (offline divergence, no auth use) - V-->>API: 200 VerificationResponse + alt client-side embedding (app.auth.client-side-embedding = ON) + B->>B: detect + align (MediaPipe eye-aligner)
compute Facenet512 in onnxruntime-web
(BGR, [0,1], 160×160, L2-norm) + Note over B: raw image NEVER leaves the device
only the 512-float vector is sent + B->>API: 512-d embedding (JSON, TLS) — NO image + API->>API: liveness already proven by a paired
factor (passive / puzzle §7); else FAIL + API->>V: POST /api/v1/verify-embedding
{tenant_id, user_id, embedding[512]} + V->>UC: match_embedding (skips detect/quality/
liveness/server-Facenet512) + UC->>DB: find_by_user_id (tenant-scoped) + DB-->>UC: stored template (decrypted from ciphertext) + UC->>UC: cosine distance vs threshold 0.45
(aged > 2 yr → _AGED) + UC-->>V: verified, distance, confidence + V-->>API: 200 VerificationResponse (anti-spoof fields None — no image) + else legacy image upload (flag OFF — default) + B->>B: detect face, quality pre-filter,
passive-liveness gate (0.45) + B->>API: face image (+ optional 128-dim
client_embedding = LOG-ONLY) + API->>V: POST /verify (X-API-Key, user_id, file) + V->>V: validate magic bytes + format + V->>LU: passive liveness (UniFace MiniFASNet) + LU-->>V: is_live, score + alt not is_live OR score < 0.4 + V-->>API: 400 LIVENESS_FAILED + end + V->>UC: execute(user_id, image_path, tenant_id) + UC->>UC: detect → face region → quality (≥50) + UC->>UC: extract Facenet-512 embedding + UC->>DB: find_by_user_id (tenant-scoped) + DB-->>UC: stored embedding (decrypted from ciphertext) + UC->>UC: cosine distance vs threshold 0.45
(aged > 2 yr → _AGED) + UC-->>V: verified, distance, confidence, threshold + V->>V: anti-spoof helpers (flag-gated):
device-risk, assembler, EAR (single frame) + alt block_reason AND ANTISPOOF_BLOCK_ENFORCE + V-->>API: 403 ANTISPOOF_BLOCKED {reason} + else allowed + V->>DB: BackgroundTask: log client-embedding
observation (offline divergence, no auth use) + V-->>API: 200 VerificationResponse + end end API-->>B: MFA step result (server decides) ``` @@ -270,6 +315,13 @@ EAR/MAR/yaw, hand-gesture challenges scored from landmarks), tracks the session, and on success issues a short-lived **HS256 JWT verification token** (TTL 300s, UUID `jti`). 23 micro-challenges exist in the web registry: **14 face + 9 hand**. +> The two flows below are the standalone liveness mechanisms. The **PUZZLE +> auth-flow layer** (a server-issued, single-use, anti-replay session that an admin +> can compose into a login flow, optionally identity-bound to a face embedding) is +> the newer converged design — see **§7** below. It is flag-gated +> (`app.auth.puzzle-layer`, default OFF) and supersedes the legacy stateless +> GeneratePuzzle/VerifyPuzzle variant shown in the third subgraph here. + ```mermaid flowchart TB subgraph PASSIVE["PASSIVE — single frame"] @@ -398,3 +450,125 @@ flowchart LR NOTE["NO embedding · NO pgvector · NO bio-processor call.
Server-side SHA-256 'fingerprint' path was REMOVED (P1.4).
AuthMethodType.FINGERPRINT is retained ONLY for WebAuthn."] SERVER -.-> NOTE ``` + +--- + +## 7. PUZZLE auth-flow layer — server-authoritative, anti-replay session + +**Flag-gated** (`app.auth.puzzle-layer`, default OFF). The Biometric Puzzle is a +**first-class auth-flow layer**, not a FACE sub-component: a tenant admin composes it +into a login flow (allowed challenge types, count, difficulty) the same way they pick +any other factor. Liveness is proven by a **server-issued, server-randomized, single-use +session**: the browser runs MediaPipe detection locally and uploads only landmark/gesture +**traces** (no frames), and the server (`biometric-processor`) re-scores those traces +against the challenges it issued. The browser never sends a "passed" boolean the server +trusts; bio is the sole authority and **consumes** the session at verdict. + +Because the biometric-processor has **no public route**, the browser talks to Identity +Core, which proxies to bio over the internal Docker network with `X-API-Key`. Contract: +`docs/superpowers/plans/2026-06-12-puzzle-session-convergence.md` (canonical, snake_case). + +**Trust properties (all hold):** challenges are randomized per attempt; `session_id` is +server-generated, single-use (consumed on verdict), short-TTL (300s), and bound to +`user_id` + `tenant_id`; a captured valid trace cannot be replayed (replay needs a fresh +server-issued session with its own random challenges). + +**Optional identity-binding (default-secure):** when the layer has +`alsoMatchFaceIdentity` AND `app.auth.client-side-embedding` is ON, the browser grabs a +best frontal frame from the **same live puzzle session**, computes the Facenet512 vector +(SP-A `embedCapturedFace`, raw image never transmitted), and submits it with the verdict +request. The handler passes ONLY if **both** liveness AND the server-side pgvector identity +match succeed — double-gated, fail-closed, single-capture (defeats split-capture, where an +attacker presents a photo for the match and a live face for the liveness separately). + +### 7a. Session lifecycle — CREATE → SUBMIT(s) → VERDICT (sequence) + +```mermaid +sequenceDiagram + autonumber + participant B as Browser (PuzzleStep · MediaPipe) + participant API as identity-core-api
(/auth/mfa/puzzle/session* + PuzzleVerifyMfaStepHandler) + participant BIO as bio (PuzzleSessionManager)
/api/v1/liveness/puzzle-session* + participant DB as PostgreSQL + pgvector + + Note over B,BIO: bio has NO public route — Identity Core proxies with X-API-Key + + rect rgb(235,245,255) + Note over B,BIO: 1 · CREATE (authorized by the in-progress MFA session token) + B->>API: POST /auth/mfa/puzzle/session
(allowed_challenge_types, count, difficulty) + API->>BIO: POST /api/v1/liveness/puzzle-session
(tenant_id, user_id, allowed_challenge_types, count, difficulty) + BIO->>BIO: randomly select N challenges —
store session (session_id, issued challenges,
status pending, TTL 300s, single-use,
owner = user_id + tenant_id) + BIO-->>API: session_id + challenges (action, params) + API-->>B: session_id + challenges (owner stamped from MFA session, not client) + end + + rect rgb(245,245,235) + Note over B,BIO: 2 · SUBMIT per challenge (UX feedback only — NOT the gate) + loop each issued challenge + B->>B: perform action —
MediaPipe FaceLandmarker 478 / HandLandmarker 21 IN-BROWSER + B->>API: POST /auth/mfa/puzzle/session/SID/challenge
(action, metrics, start_ts_ms, end_ts_ms, confidence) + API->>BIO: POST /api/v1/liveness/puzzle-session/SID/challenge + BIO->>BIO: score traces vs issued challenge
(metric REQUIRED — absent/empty ⇒ fail) —
mark challenge complete + BIO-->>API: verified, action, reason_code? + API-->>B: per-challenge UX result (advisory) + end + end + + rect rgb(235,245,235) + Note over B,BIO: 3 · VERDICT (the auth gate) — optional identity-binding + B->>B: if alsoMatchFaceIdentity AND client-side-embedding ON:
grab best frontal frame → Facenet512 in-browser
(raw image NOT sent) + B->>API: verifyStep PUZZLE (puzzle_session_id, optional embedding)
— NO metrics, NO client verdicts + API->>BIO: POST /api/v1/liveness/puzzle-session/SID/verdict
(user_id, tenant_id) + BIO->>BIO: verified = all issued challenges validated
AND owner == user_id+tenant_id
AND not expired AND not already consumed + BIO->>BIO: CONSUME session (single-use) + BIO-->>API: verified (bool) + opt identity-binding (embedding present) + API->>BIO: POST /api/v1/verify-embedding
(tenant_id, user_id, embedding 512) + BIO->>DB: pgvector cosine match vs stored template + DB-->>BIO: distance + BIO-->>API: verified (identity) + end + API->>API: PASS only if liveness == true
(AND identity == true when bound) —
HARD-FAIL on missing field / error / 404 / false + API-->>B: MFA step result (server decides) + end +``` + +### 7b. Why the session is anti-replay + the layer composition (flowchart) + +```mermaid +flowchart TB + subgraph ADMIN["Tenant admin — composes the flow (web AuthFlowBuilder)"] + AB["PUZZLE layer config (auth_flow_steps.config JSONB):
allowed challenge types (checkbox) · count · difficulty
· alsoMatchFaceIdentity (default ON)"] + end + + subgraph FLOW["Example composed login flow"] + L1["L1 identifier
{email · QR · passkey}"] + L2["L2 PUZZLE layer
(liveness ± identity-binding)"] + L3["L3 other factor
(fingerprint / OTP / …)"] + L1 --> L2 --> L3 + end + AB -.configures.-> L2 + + subgraph SESSION["Server-authoritative session (bio PuzzleSessionManager)"] + R["server-RANDOMIZED challenges
per attempt"] + SID["session_id: server-generated,
unguessable, single-use, TTL 300s"] + OWN["bound to user_id + tenant_id
(owner stamped from MFA session)"] + STATE["all scoring state in bio;
client sends traces, never a trusted 'passed'"] + CONS["CONSUMED on verdict"] + end + L2 --> SESSION + + subgraph REPLAY["Replay / forgery attempts → REJECTED"] + X1["captured valid traces replayed →
need a FRESH session with NEW random challenges"] + X2["session reused →
already consumed (single-use)"] + X3["session issued for A used by B →
owner mismatch"] + X4["expired (> TTL) → rejected"] + X5["photo for match + live face for liveness
(split-capture) → embedding taken from the SAME
live session frames; mismatch → fail-closed"] + end + SESSION --> REPLAY + + classDef ok fill:#bbf7d0,stroke:#15803d,color:#1f2937; + classDef bad fill:#fecaca,stroke:#b91c1c,color:#1f2937; + class R,SID,OWN,STATE,CONS ok; + class X1,X2,X3,X4,X5 bad; +``` diff --git a/plans/CLIENT_SIDE_ML_PLAN.md b/plans/CLIENT_SIDE_ML_PLAN.md index e91a3e4..895388d 100644 --- a/plans/CLIENT_SIDE_ML_PLAN.md +++ b/plans/CLIENT_SIDE_ML_PLAN.md @@ -1,188 +1,204 @@ -# Client-Side ML Strategy (Pre-Filter Only) +# Client-Side ML Strategy (Client-Authoritative Embedding + Puzzle-as-Layer) -**Version:** 2.0 -**Last Updated:** 2026-04-14 -**Status:** Active (rewritten from 1.0 aspirational design) +**Version:** 3.0 +**Last Updated:** 2026-06-12 +**Status:** Active — landed dark, feature-flagged (default OFF). Supersedes v2.0 (pre-filter-only) and its D1/D2 locks. **Project:** FIVUCSAS — Face and Identity Verification Using Cloud-Based SaaS **Server:** Hetzner CX43 — 8 vCPU / 16 GB RAM — **NO GPU** --- -## 1. Strategic Position - -The server has no GPU. All heavy ML **inference for auth decisions** stays server-side (CPU-lean, pgvector-backed). The client role is **pre-filtering**: detection, quality, liveness pre-screen, crop, and voice activity detection. The goal is to reduce server load and upload bandwidth — not to move verdicts to the client. +## 0. What changed since v2.0 (read this first) -**What changed vs v1.0:** the 2026-04-05 design document claimed client-primary face verification, a 128↔512 projection matrix, and CDN-hosted model delivery. Audit on 2026-04-14 confirmed none of that was built. This version retires those ambitions explicitly. - ---- +v2.0 (2026-04-14) locked **pre-filter only**: the client detected/quality-gated/cropped a +face and uploaded a **JPEG**; the **server** ran MTCNN → quality → passive liveness → +Facenet512 → pgvector match. Any client embedding was **log-only** (D2). -## 2. Locked Decisions (2026-04-14) +v3.0 moves the **face embedding into the browser** and makes it **authoritative**, while +keeping the **server** the sole owner of liveness, the pgvector match, and the +accept/reject decision. It also promotes the **Biometric Puzzle to a first-class auth-flow +layer** backed by a server-issued, single-use, anti-replay session. Everything is +flag-gated and landed dark; the v2.0 legacy image path remains as the default and the +fallback. -| ID | Decision | Rationale | +| | v2.0 (pre-filter only) | v3.0 (this doc) | |---|---|---| -| D1 | **Pre-filter only** | Client pre-screens; server is sole source of truth for embeddings and verdicts. Delivers latency/bandwidth wins without rebuilding auth. | -| D2 | **`client_embedding` log-only** | Client embedding is now landmark-geometry (512-dim, MediaPipe-based). MobileFaceNet ONNX deprecated 2026-04-18 — server-side DeepFace Facenet512 remains the sole trusted embedding for auth. Server accepts and persists the field for offline divergence analysis only. Never trusted for auth decisions. | -| D3 | **Build-time model fetch (Hostinger static + SHA256 manifest)** | Deterministic deploys, no git-lfs, matches the no-dockerize-static rule. | -| D4 | **Voice V1 now, V2 later** | V1: Silero VAD client-side, skip upload on silence. V2 (ECAPA-TDNN client embedding, remove librosa pin): deferred until V33 stable. | - ---- +| Face embedding | **server** (DeepFace Facenet512) | **browser** (Facenet512 via onnxruntime-web), authoritative; server path kept as fallback | +| What the client uploads for FACE | a JPEG image | only a 512-float vector (flag ON); JPEG (flag OFF / fallback) | +| Liveness | server passive (`/verify`) | server: passive **or** the active Puzzle layer; the vector path carries no liveness and MUST be paired | +| Client embedding role | log-only (D2) | authoritative (when the flag is ON); the old 128-dim landmark-geometry log-only field is retained, unchanged, for observability | +| Puzzle | a FACE liveness sub-component | a first-class auth-flow **layer** an admin composes, server-authoritative session, optional identity-binding | -## 3. Inference Distribution — Actual State (2026-04-14) +**The reason** is that this is exactly the **client-loaded, GPU-less** design already shown +in the system-context / container diagrams and the poster: privacy-preserving (the raw +image stays on the device), bandwidth-light (~2 KB vector vs a JPEG), and it offloads the +server's heaviest CPU step. -| Task | Client | Server | Status | -|---|---|---|---| -| Face detection | **Primary** (MediaPipe FaceMesh / BlazeFace fallback) | — | Live | -| Face quality gate | **Primary** | — | Live (`QualityAssessor.ts`) | -| Face crop 224×224 before upload | **Primary** | — | Live (`faceCropper.ts`) | -| Face tracking (IoU) | **Primary** | — | Live (`FaceTracker.ts`) | -| Passive liveness pre-screen | **Primary** (advisory today, gating in Phase 5) | Authoritative | Live (`PassiveLivenessDetector.ts`) | -| Active liveness puzzle | **Primary** | **Authoritative** | Live (`BiometricPuzzle.ts`) | -| Face embedding | — | **Only** (ArcFace 512-dim, DeepFace) | Server only | -| Face 1:1 verify | — | **Only** (cosine in pgvector) | Server only | -| Face 1:N search | — | **Only** | Server only | -| Voice VAD | **Primary** (Phase 4, this plan) | — | In progress | -| Voice embedding | — | **Only** (Resemblyzer 256-dim) | Server only, V2 may revisit | -| Voice 1:1 verify | — | **Only** | Server only | -| Card detection | **Only** (YOLOv8n ONNX / WASM, in-browser) | — | Live — client-only; server fallback removed (web-app #111). True 12.3 MB YOLOv8n delivered + integrated (web-app #109, biometric-processor #116) | -| Card OCR + MRZ | — | **Only** | Server only | -| Proctoring gaze / deepfake / object | — | **Only** | Server only | +--- -**No client-side verdict path exists.** The client never compares embeddings, never applies thresholds, never decides accept/reject. UI state reflects server responses. +## 1. Strategic Position ---- +The server has no GPU. Under v3.0 the **client computes the face embedding** (the heavy +CPU step), and the **server keeps the security-critical, untrusted-client-resistant work**: +liveness, the 1:1 / 1:N pgvector match, the threshold/decision, lockout and rate-limit. The +client is untrusted, so it is never allowed to assert "liveness passed" or "match = true" — +it can only produce a vector, which the server matches against the enrolled template and +decides on. -## 4. Model Inventory - -### 4.1 Face Detection — MediaPipe FaceMesh / BlazeFace -Live in browser. No action needed. Model bundled with MediaPipe runtime. - -### 4.2 Face Embedding — MobileFaceNet (client pre-screen, **not** for verdicts) -| Property | Value | -|---|---| -| Format | ONNX (WASM) | -| Size | ~4.9 MB INT8 | -| Input | 112×112 aligned face | -| Output | 128-dim L2-normalized embedding | -| Role | **Sent to server as `client_embedding` for log-only observation (D2).** Never used for client-side verify. | - -**Out of scope:** 128→512 projection matrix, client-side cosine verification, embedding cache on device. - -### 4.3 Voice VAD — Silero -| Property | Value | -|---|---| -| Format | ONNX (WASM) | -| Size | ~1.8 MB | -| Input | 16 kHz mono PCM, 512-sample frames | -| Output | Per-frame speech probability | -| Role | Skip upload when `speechRatio < 0.2`. Graceful fallback when model missing. | - -### 4.4 Card Detection — YOLOv8n -| Property | Value | -|---|---| -| Format | ONNX (WASM) | -| Size | ~12.3 MB (true YOLOv8n, opset 12) | -| Input | 640×640 RGB (letterboxed) | -| Output | Bounding boxes + class | -| Role | Real-time client overlay + detection (client-only); crop sent to server for OCR + MRZ. No server fallback for detection (removed, web-app #111). | - -### 4.5 Passive Liveness — MobileNet-v3 Anti-Spoof -Currently heuristic (texture/moire/color) in `PassiveLivenessDetector.ts`. No neural model. Phase 5 will wire it into a gating threshold or demote it — no new model purchase required. - -### 4.6 Explicitly Deferred -- ECAPA-TDNN voice embedding (D4 V2) -- Neural DNN liveness (no incident justifies XL effort) -- Iris / gaze tracking (no browser API) -- Demographics (server-only, DeepFace Python) +This keeps the v2.0 win (latency/bandwidth) and adds a genuine **data-minimization** +privacy win, without moving any verdict to the client. --- -## 5. Model Delivery (Phase 3 of Rollout Plan) +## 2. Locked Decisions (2026-06-12, v3.0) -**Hostinger static bucket** at `app.fivucsas.com/models/` serves the three `.onnx` files. Repo contains a committed `manifest.json` with SHA256 hashes; `scripts/fetch-models.mjs` runs as `prebuild` to download and verify. Fatal on hash mismatch. - -```json -{ - "base_url": "https://app.fivucsas.com/models", - "files": [ - {"name": "mobilefacenet.onnx", "sha256": "", "bytes": 4915200}, - {"name": "yolo-card-nano.onnx", "sha256": "", "bytes": 12300000}, - {"name": "silero-vad.onnx", "sha256": "", "bytes": 1850000} - ] -} -``` +| ID | Decision | Rationale | +|---|---|---| +| **D1′** | **Client computes the authoritative face embedding** (Facenet512, onnxruntime-web). Flag-gated `app.auth.client-side-embedding` (default OFF). | Matches the presented GPU-less design; offloads the server's heaviest step; raw image never leaves the device. **Supersedes v2.0 D1 (pre-filter only).** | +| **D2′** | **Only the 512-d vector leaves the device for FACE** (flag ON). The legacy 128-dim landmark-geometry `client_embedding` field stays log-only for observability. | The transmitted biometric is a derived, non-invertible template, not the image. **Supersedes v2.0 D2 (client embedding log-only as the only client embedding).** | +| **D3** | **Download-once, SHA256-verified model delivery.** FP16 Facenet512 ONNX (~47 MB) served from `app.fivucsas.com/models/`, content-hashed filename, cached Service-Worker (CacheFirst) + IndexedDB, integrity-checked, re-verify-on-read. INT8 **rejected** (onnxruntime-web WASM lacks `ConvInteger`/`MatMulInteger`/`DynamicQuantizeLinear`). | Deterministic, no git-lfs, honors no-dockerize-static; extends the YOLO-card `fetch-models` pattern. | +| **D4** | **Server owns liveness + match + decision.** The vector path has no liveness; it MUST be paired with a liveness factor (passive or the Puzzle layer) before it is a trusted login factor. | The client is untrusted; a stolen/forged vector replayed without a fresh server session must not pass. | +| **D5** | **Puzzle is a first-class auth-flow layer**, backed by a server-issued, single-use, randomized, owner-bound, short-TTL session (anti-replay). Optional **identity-binding** extracts the embedding from the same live puzzle frames. Flag-gated `app.auth.puzzle-layer` (default OFF). | Closes split-capture; makes the active puzzle composable like any other factor; the server is the sole authority. | +| **D6** | **Re-enroll developers** with the client pipeline so probe and template share the client preprocessing (self-consistency, no MTCNN-parity gymnastics). No real users exist, so this is acceptable. | Avoids mixed old/new templates and parity risk. | -`.onnx` files are git-ignored. CI runs `npm run fetch-models` before `npm run build`. +The voice strategy (Silero VAD client-side, server Resemblyzer embedding) is unchanged from +v2.0 D4 (see §8 legacy appendix). Card detection (client-side YOLO) is unchanged. --- -## 6. Server Contract for Log-Only Embedding - -Web-app already posts `client_embedding` (single JSON array) and `client_embeddings` (JSON array of arrays). Server persists them into `client_embedding_observations` via FastAPI `BackgroundTasks`: +## 3. Trust model (the crux) + +Browser computes; the **server verifies only what it can re-check without trusting the +client's word.** + +- **Identity:** the client uploads the 512-d embedding; the **server** runs the pgvector + cosine match against enrolled templates and decides. A forged vector cannot match a real + template; the server owns the threshold + verdict. +- **Liveness:** proven by the **active Biometric Puzzle** — the server issues a *randomized* + challenge, the client performs it and uploads landmark/gesture **traces** (not an image), + and the **server re-scores** the traces against the challenge it issued. Random challenge + + server re-score + single-use session defeats replay; no image needed. +- **Binding (anti split-capture):** when identity-binding is on, the embedding is extracted + from the **same live puzzle frames** as the liveness challenge, so identity and liveness + come from one capture. + +**Platform offers methods; tenant configures policy.** The *assurance level* is a tenant +config, not a hard mandate, with two tiers: + +- **Baseline correctness (always on):** the embedding is accepted ONLY inside a + server-issued, single-use, short-TTL session, with anti-replay. Without this floor the + upload-the-vector method is simply broken, so it is table stakes — not a security policy. +- **Assurance level (tenant config + warning):** above the floor, the admin chooses how + strong the liveness binding is (passive only / passive + active-puzzle-bound-to-the-same- + capture). The bound option is the recommended default and closes split-capture; weaker + options are offered with a clear "lower assurance" warning. + +### 3a. Privacy statement (honest framing) + +The correct claim is precise: **the raw face image never leaves the device; the only +biometric data transmitted is a derived, non-invertible 512-d embedding, sent over TLS and +stored encrypted at rest (Fernet).** We do **not** claim "biometric data never leaves the +device" — the embedding is personal biometric data under KVKK/GDPR. The genuine win is +**data minimization**: the image is not transmitted, only a template-grade vector. -```sql -client_embedding_observations ( - observation_id UUID PK, user_id, tenant_id, session_id, - modality ('face'|'card'), flow ('enroll'|'verify'), - client_embedding vector(128), client_model_version, - server_embedding_ref UUID NULL, cosine_similarity FLOAT8 NULL, - device_platform, user_agent, created_at -) -``` +--- -Failure to record must **not** break the primary flow. Cosine similarity is computed offline, not at request time. +## 4. Components (landed dark) + +1. **Client embedding module** — Facenet512 ONNX via onnxruntime-web; one module used by + BOTH the FACE (passive) path and the PUZZLE (active) path. Replicates DeepFace's exact + preprocessing: aligned face crop → aspect-preserving resize + centre black-pad → + `(1,160,160,3)` float32, **BGR**, **[0,1]** (`normalization="base"`, NOT prewhiten) → + L2-normalize. In-browser **eye aligner** (similarity transform to canonical eye coords) + addresses the alignment risk. (`web-app` `embedCapturedFace.ts`.) +2. **Model cache** — FP16 `facenet512-.onnx`, downloaded once, SW CacheFirst + + IndexedDB, SHA256-manifest integrity-checked, re-verify-on-read. (D3.) +3. **Upload contract (FACE)** — browser → Identity Core → bio: + - verify: `POST /api/v1/verify-embedding {tenant_id, user_id, embedding[512]}` + - enroll: `POST /api/v1/enroll-embedding {…, embedding[512]}` + - bio runs ONLY the pgvector match (verify) / encrypt+persist (enroll); no image, no + server-side detect/quality/Facenet512. +4. **Identity Core routing** — `ClientSideEmbeddingPolicy` (`app.auth.client-side-embedding`, + global or per-canary-tenant, default OFF, fail-closed). `FaceVerifyMfaStepHandler` and + `EnrollBiometricService` route to the embedding path when enabled. +5. **Puzzle layer (D5)** — `AuthMethodType.PUZZLE` + `PuzzleLayerPolicy` + (`app.auth.puzzle-layer`). bio `PuzzleSessionManager` + `/api/v1/liveness/puzzle-session*` + (create / submit / verdict; single-use, TTL 300s, owner-bound, server-randomized). + Identity Core MFA-flow proxy (`/auth/mfa/puzzle/session*`) + server-authoritative + `PuzzleVerifyMfaStepHandler` (`{puzzle_session_id[, embedding]}` → bio verdict, + HARD-FAIL, no client-trust). Optional identity-binding double-gated on + `alsoMatchFaceIdentity` AND `app.auth.client-side-embedding`. +6. **Server fallback (kept)** — the v2.0 image → server-Facenet512 path stays in code as + the default and the fallback; it runs when the flags are OFF. + +The canonical puzzle-session contract is +`docs/superpowers/plans/2026-06-12-puzzle-session-convergence.md`. --- -## 7. Rollout Plan (Active) +## 5. Inference Distribution — v3.0 state -Tracked in `/home/deploy/.claude/plans/resilient-finding-thunder.md`. +| Task | Client | Server | Status | +|---|---|---|---| +| Face detection | **Primary** (MediaPipe FaceLandmarker / BlazeFace) | — | Live | +| Face quality gate | **Primary** | — | Live | +| Face align (eye-aligner) | **Primary** (client-side embedding path) | — (server aligns on the legacy path) | Live (flag) | +| **Face embedding** | **Authoritative** (Facenet512, onnxruntime-web) when flag ON | Fallback (DeepFace Facenet512) when flag OFF | Landed dark (flag) | +| Face 1:1 verify (match + decision) | — | **Only** (pgvector cosine + threshold) | Server only | +| Face 1:N search | — | **Only** | Server only | +| Passive liveness pre-screen | **Primary** (advisory) | Authoritative (legacy path) | Live | +| Active liveness puzzle (detection) | **Primary** (MediaPipe in-browser) | — | Live | +| Active liveness puzzle (scoring + verdict) | — | **Authoritative** (server-issued session, re-score) | Landed dark (flag) | +| Voice VAD | **Primary** (Silero) | — | Live | +| Voice embedding / verify / search | — | **Only** (Resemblyzer 256-dim) | Server only | +| Card detection | **Only** (YOLO ONNX, in-browser) | — | Live | +| Card OCR + MRZ | — | **Only** | Server only | -| Phase | Work | Status | -|---|---|---| -| 1 | Rewrite this doc + memory files | Done (this commit) | -| 2 | Alembic V4 `client_embedding_observations` + route wiring | In progress | -| 3 | Hostinger bucket + `fetch-models.mjs` + manifest + CI prebuild | Awaiting model hashes (user action) | -| 4 | Client-side Silero VAD (`VoiceVAD.ts`) + upload gate | In progress | -| 5 | Wire `PassiveLivenessDetector` into capture gate **or** demote | Pending | +**No client-side verdict path exists.** The client never compares embeddings, never applies +thresholds, never decides accept/reject; it produces a vector and traces, the server +decides. UI state reflects server responses. --- -## 8. Explicitly Out of Scope +## 6. Reversibility / rollout (flag-gated, landed dark) -- 128↔512 projection matrix (retired) -- Client-primary face verify (retired) -- Client-side 1:N search -- Encrypted on-device embedding cache -- ECAPA-TDNN client voice embedding (deferred per D4) -- Neural DNN liveness -- Voice STT challenge (separate `VOICE_STT_PLAN.md`) -- Server face-detection refactor to MediaPipe Tasks (reversed — strategy is client-first pre-filter, not server rewrite) -- KMP client-apps ML (this document now scopes to web-app only; mobile has its own track) +`app.auth.client-side-embedding` (FACE) and `app.auth.puzzle-layer` (PUZZLE) both default +OFF. Sequence: **land dark** (flags OFF, prod unchanged) → host the FP16 model + re-add the +manifest entry + build a JSON enroll endpoint → **canary** (one tenant) → re-enroll devs → +real-device same-person cosine validation across two devices → broad flip. Kill-switch = +flag OFF (no redeploy). Server image path is the fallback. + +**Ordering caveat:** flip the Identity Core flag **before** the web `VITE_CLIENT_SIDE_EMBEDDING` +flag — web-ON + identity-OFF breaks FACE login (no image sent → legacy path fails). Best +fix = drive the web flag from login-config. (Phase-6 runbook: +`docs/superpowers/2026-06-12-client-side-embedding-PHASE6-runbook.md`.) --- -## 9. Risk & Mitigation +## 7. Model Delivery (D3) -| Risk | Mitigation | -|---|---| -| Model file missing in production | Build-time fetch with SHA256; fatal on mismatch. Graceful client fallback (server handles) during transition. | -| Log-only insert fails and breaks enrollment | `try/except` with background task; never raise. Telemetry is best-effort. | -| Tampered `client_embedding` from malicious client | Field is never trusted. Offline analysis can flag divergence but cannot affect auth. | -| VAD false-negatives block legitimate users | Threshold conservative (0.2). VAD-unavailable path always allows upload. User error message is retry-friendly. | -| Doc drift recurs | Single source of truth: this file. Memory files reference it; no parallel "future design" doc. | +FP16 Facenet512 ONNX (~47 MB) exported from our own SHA-pinned `facenet512_weights.h5` +(`tf2onnx`, opset 17 — do NOT ship unlicensed public ONNX). Served from +`app.fivucsas.com/models/` with a content-hashed filename and a `manifest.json` SHA256; +downloaded once, cached (SW CacheFirst + IndexedDB), integrity-checked on read. INT8 is +rejected for onnxruntime-web WASM (missing quant ops). The voice (Silero) + card (YOLO) +models continue to ship via the same manifest pattern. --- -## 10. Verification - -Each phase has an acceptance check in the rollout plan. Summary: +## 8. Legacy appendix — v2.0 (pre-filter only), retained for history -- Phase 1: `grep -n "Primary" docs/plans/CLIENT_SIDE_ML_PLAN.md` returns zero hits in the face-verify row. -- Phase 2: `alembic upgrade head` clean; round-trip POST with `client_embedding` lands a row; missing field still works. -- Phase 3: fresh clone → `npm install && npm run build` produces `dist/models/*.onnx` with correct hashes. -- Phase 4: silent recording blocked; speech recording passes; missing VAD model fails open. -- Phase 5: every client ML class has at least one real consumer. +v2.0 (2026-04-14) was the **pre-filter-only** strategy: the client detected / quality-gated +/ cropped and uploaded a JPEG; the server owned embeddings and verdicts; any client +embedding (D2) was log-only. The v2.0 image path is still in the codebase as the default and +the fallback when the v3.0 flags are OFF, so its behavior is preserved verbatim — v3.0 does +not remove it, it adds the client-authoritative path on top behind a flag. ---- +Earlier v1.0 (2026-04-05) promised client-primary verdicts, a 128↔512 projection matrix, +CDN delivery and a 10-week KMP rollout; that design was retired in v2.0. v3.0 does **not** +resurrect client-side verdicts or the projection matrix — the server still owns every +decision; what v3.0 adds is a client-computed *authoritative embedding* that the server +matches. -*Previous v1.0 (2026-04-05) promised client-primary verdicts, projection matrix, CDN, 10-week KMP rollout. That design is retired in favor of the pre-filter-only strategy above. See `/home/deploy/.claude/plans/resilient-finding-thunder.md` for execution detail.* +The v2.0 voice plan (D4: Silero VAD client-side now; ECAPA-TDNN client embedding deferred) +and the card-detection-on-client decision carry forward unchanged.