feat(dashboard): detection rules section under Secure#2992
Open
mfbx9da4 wants to merge 8 commits into
Open
Conversation
Add a Detection Rules page that lists built-in rules by category (matching the policy form groupings) and allows authoring custom regex-based rules. Severity overrides and custom rules are stored client-side until the server endpoints land. Custom rules show up as a selectable section in the policy creation form.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.
Tip: disable this comment in your organization's Code Review settings.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
🦋 Changeset detectedLatest commit: 07ee641 The changes in this PR will be included in the next version bump. Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
…ets default Place Detection Rules after Risk Policies in the Secure nav group, and start the Detection Rules page with every category collapsed.
Secrets stays at high since the cost of leaking a credential is qualitatively different. Everything else (financial, PII, gov IDs, healthcare, prompt injection, off-policy, shadow MCP, destructive tools, CLI destructive, custom) starts at medium and can be tuned per rule via the Detection Rules page.
…hoist BETA badge to Secure group Sheets are siblings of Page.Section.Title and friends, but Page.Section's slot filter only renders known slot children. The CreateCustomRuleSheet and RuleDetailSheet were getting dropped, so the "New Custom Detection Rule" button looked dead. Wrap the page body in a fragment and move the sheets outside the section so the slot filter no longer eats them. Also collapse the per-sub-route BETA badge into the parent Secure nav group: add a stage prop to CollapsibleNavGroup, drop stage from riskOverview/policyCenter/detectionRules, and pass stage="beta" once on the group. Less visual noise inside the expanded section.
Add the risk.customRules.suggest endpoint that takes a single
natural-language prompt ("what do you want to detect?") and returns a
structured suggestion (rule_id, title, description, regex, severity)
via OpenRouter's JSON-schema-constrained completion. The dashboard's
New Custom Detection Rule sheet now opens on the prompt step, calls
the endpoint on submit, and drops the operator into an editable review
form with the suggested fields prefilled.
A "Skip, fill manually" affordance keeps the original form reachable
when the operator already knows the rule shape.
…ndor names from rule copy Two unrelated fixes shipped together because they came up in the same demo loop. 1. SuggestCustomDetectionRule no longer 500s when the org has no provisioned OpenRouter key (or the upstream API errors). The handler logs a warning and returns a deterministic heuristic suggestion (slug-derived rule_id, titleized name, prompt-as-description, empty regex, severity=medium) so the operator always lands in the editable review step. 2. Detection Rules page descriptions no longer name "gitleaks" or "Presidio". Operators think in what is detected (token shape, PII, gov IDs), not how — leaking library names is detail churn that ages poorly if we swap engines.
alx-xo
approved these changes
May 21, 2026
Contributor
alx-xo
left a comment
alx-xo
left a comment
There was a problem hiding this comment.
I like this has it's own page!
I think we could do the same for New Policy as well
Add a rule playground that lives inside each detection rule's detail sheet. Paste a sample, click Run, and the dashboard calls the new risk.rules.test endpoint. The endpoint dispatches by rule_id prefix to the same scanner code the worker uses during chat-message analysis: - secret.* -> ScanWithGitleaks - pii.* -> the configured PIIScanner (Presidio) - prompt_injection.* -> PromptInjectionScanner - custom.* -> regex match using the regex on the rule - shadow_mcp.* / destructive_tool.* -> not supported (no text-only detector) Result is a list of TestDetectionRuleMatch values that mirror the runtime risk_result shape (rule_id, description, match, start/end, source, confidence, tags) so what the operator sees here is what the worker would record on a real chat message. Separately, drop the severity-override UI from the rule detail sheet per upstream call. Severity still renders as a default badge per row and in the detail; editing / overriding will return in a follow-up.
Extend the rule playground inside the detection-rule detail sheet with a second mode. Operators now toggle between "Paste sample" and "Run on a chat": - The chat picker lists chats from the last 7 days, grouped by external user id (falling back to internal user id or "(no external user)" so every chat ends up under a heading). - After selecting a user and a chat, Run on chat loads the chat, truncates to the most-recent 100 messages, and warns when the chat has more than that. - Each message fires the existing risk.rules.test endpoint serially; results stream into the UI as they come back, sorted newest-first. - Each row collapses the message to a 240-char preview and expands on click to show the full text and the per-rule match list. Empty messages short-circuit to a 0-match row without hitting the server. While I was in there, retire the severity-override store entries (severityOverrides, setSeverityOverride, resolveSeverity) and the Override row badge that pointed at them — that feature got punted to a follow-up PR. The default-severity row badge stays.
simplesagar
approved these changes
May 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes AGE-2406.
Demo
https://github.com/speakeasy-api/gram/releases/download/_gh-attach-assets/demo-90s.mp4
Why
The Secure surface only exposed Risk Overview and Risk Policies. Customers want a dedicated place to (a) browse the built-in detection rules grouped by the same categories used in policy creation, (b) tweak the default severity assigned to any rule, and (c) author org-specific regex rules that participate in policies. Authoring a rule by hand — invent a stable id, pick a regex, write a description — is a lot to ask of an operator who just wants to say "catch our internal Acme tokens", so the create flow now opens on a single prompt and asks the LLM to fill the form in.
What changed
Before
After
Secure → Detection Rulespage:New Custom Detection Ruleopens a two-step wizard:Securenav group itself carries the BETA badge now, the sub-routes do not.mediumeverywhere except Secrets (stillhigh).gitleaks/Presidio— vendor names are an implementation detail.Backend
risk.customRules.suggest(POST /rpc/risk.customRules.suggest). Calls OpenRouter viaGetObjectCompletionwith a JSON-schema-constrained response so the dashboard always gets{rule_id, title, description, regex, severity}. When OpenRouter is unreachable or the org has no provisioned key, the handler logs a warning and returns a deterministic heuristic suggestion derived from the prompt, so the wizard never dead-ends.org:admin), auth context, observability, and audit conventions.gram.detection-rules.v1) — the durable store ships with the follow-up that adds the worker side of custom regex evaluation.Test plan
Secure → Detection Rulesrenders with built-in categories in the same order as the policy form, BETA badge onSecureonly.Overridetag appears; reset puts the override back to default.New Custom Detection Ruleopens the prompt step; submitting a description prefills the review form. With no OpenRouter key locally, the heuristic fallback fillsrule_id,title,description; emptyregexfor the operator to write.secret.aws_access_token) and with existing customs.[a-zsurfaces an error.Custom Rulessection with per-rule checkboxes once at least one custom rule exists.