refactor(risk): dedup SQL category classifier via unnest arrays#2987
refactor(risk): dedup SQL category classifier via unnest arrays#2987mfbx9da4 wants to merge 3 commits into
Conversation
Follow-up to #2976. Four duplicated CASE expressions in server/internal/risk/queries.sql shared a 40-line categorisation block that the Go classifier in internal/risk/categories already owned. Replace them with a single subquery that joins risk_results against a session-scoped TEMP TABLE risk_category_lookup populated from the canonical Go classifier on every pool connection. How it works: - internal/risk/categories.BootstrapConnection runs on AfterConnect for every pgxpool connection (cmd/gram/deps.go and the testenv pool helper for risk_results tests). It creates risk_category_lookup if needed, truncates it, and inserts rows derived from Definitions. - All four risk queries now resolve the category via: (SELECT rcl.category FROM risk_category_lookup rcl WHERE source-match OR rule_id-match OR LIKE prefix ORDER BY priority LIMIT 1) -- with COALESCE to 'custom' - internal/risk/categories/sqlc_schema.sql declares the temp table for sqlc's static analysis (not migrated; runtime owns table creation). Result: one source of truth. Adding or moving a rule means editing Definitions and restarting the server; SQL never has to be touched. Perf: ListRiskRulesByCategory goes from ~3.2ms to ~8.7ms on the local dev DB (2,410 findings) because of the SubPlan that scans the 14-row lookup table per result row. Well within budget; see PR description for the EXPLAIN ANALYZE comparison.
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.
Tip: disable this comment in your organization's Code Review settings.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
🚀 Preview Environment (PR #2987)Preview URL: https://pr-2987.dev.getgram.ai
Gram Preview Bot |
The previous commit introduced a per-connection TEMP TABLE (risk_category_lookup) populated by a pgxpool AfterConnect hook so that the four risk-overview queries could share one category classifier without each carrying its own CASE expression. That worked but bound a runtime concern (connection bootstrap) to a build concern (sqlc static analysis), and it required a sqlc_schema.sql stub plus matching bootstrap code in cmd/gram/deps.go and internal/testenv. Switch to passing the classifier as five parallel arrays (priority, category, source, rule_id, rule_prefix) per query call. Each query builds the same risk_category_lookup CTE inline via parallel unnest() calls. No connection hook, no schema stub, no AfterConnect hook in tests; the Go classifier in internal/risk/categories remains the single source of truth. Drops the unused Filter type and FilterFor() helper that were leftovers from the previous per-query CASE design.
The dashboard's risk-utils held its own (source, rule_id) -> category classifier built from SOURCE_TO_CATEGORY and DETECTION_RULES. That was a parallel copy of the Go classifier in internal/risk/categories, guaranteed to drift whenever a rule was added on one side and not the other. Replace it with useFindingClassifier(), a hook that reads the canonical classifier through the existing useRiskCategories() React Query binding (/rpc/risk.categories, shipped in #2976). CategoryLabel renders nothing during the first fetch and falls back to "custom" for unmapped rules once data is loaded. Per-rule human-readable titles still come from DETECTION_RULES since the API does not expose them.
Why
Follow-up to #2976. The (source, rule_id) -> category mapping lived in four duplicated CASE expressions in
server/internal/risk/queries.sql(~40 lines each, ~160 lines total) and in a separate TypeScript copy in the dashboard, even though the Go classifier ininternal/risk/categoriesalready owns the canonical mapping. Adding or rebucketing a rule meant editing the Go classifier AND four SQL blocks AND the dashboard'srisk-utils.tsin lockstep, with no compile-time check that they agreed.An earlier iteration of this PR tried a session-scoped TEMP TABLE populated via a pgxpool
AfterConnecthook. It worked, but bound a runtime concern (connection bootstrap) to a build concern (sqlc static analysis), and required asqlc_schema.sqlstub plus matching hook in cmd/gram and the test pool. Switched to passing the classifier in directly as query parameters, no schema declarations or connection hooks.What changed
Server: classifier handed in as five parallel arrays
internal/risk/categories.SQLRows()projectsDefinitionsinto five parallel arrays sized 1-to-1 by row:Each caller in
internal/risk/impl.gopasses these as@cat_priority::int[],@cat_category::text[], etc.All four affected queries (
ListRiskUserCategoryBreakdown,ListRiskRulesByCategory,ListRiskOverviewTimeSeriesFindings,ListRiskResultsByProjectFound) now open with:and resolve categories per row via:
Empty-string sentinels (instead of NULL) because the Go classifier emits one row per definition with the irrelevant fields blanked out; this keeps the WHERE clause uniform without three-valued logic.
Dashboard: classifier reads from /rpc/risk.categories
risk-utils.tsused to maintain its ownSOURCE_TO_CATEGORYmap plus aDETECTION_RULES-derived rule_id lookup, a parallel copy of the Go classifier. Replaced withuseFindingClassifier(), a hook backed by the existinguseRiskCategories()React Query binding. CategoryLabel renders nothing during the first fetch and falls back to "custom" for unmapped rules once data is loaded.DETECTION_RULESstays only as the source of per-rule human-readable titles (the API exposes the classification but not titles).What's gone
internal/risk/categories/bootstrap.go(theAfterConnecthook)internal/risk/categories/sqlc_schema.sql(the sqlc-only schema stub)pgxpool.Config.AfterConnect = categories.BootstrapConnectionlines inserver/cmd/gram/deps.goandserver/internal/testenv/postgresql.goFilterstruct andFilterFor()helper (leftovers from an even earlier design)categories.Filterexhaustruct exclusionSOURCE_TO_CATEGORYmap andRULE_ID_TO_CATEGORYlookupNet change
Definitionsonce, restart, dashboard picks it up automaticallyBefore
After
and one Go file with the canonical
Definitionsslice driving both.Query performance
EXPLAIN ANALYZE on the local dev DB (575 chats / 38,632 chat_messages / 4,851 findings; busiest project = 2,410 findings):
ListRiskRulesByCategory(category='pii', 7d)The lookup is materialized once per query via the CTE; PostgreSQL plans the seq-scan against a 14-row in-memory tuple stream so there is no IO cost. Linear in the result set, not in the lookup-table size.
Test coverage
results_test.go,resultsbychat_test.go,listgetdelete_test.go,overview_test.go) all pass.categories_test.go) is unchanged and still pins the canonical behaviour.pnpm buildandpnpm lintpass.