Measure gate vs codex --sandbox: cede dev gate to off-the-shelf#7
Draft
moonweave wants to merge 1 commit into
Draft
Measure gate vs codex --sandbox: cede dev gate to off-the-shelf#7moonweave wants to merge 1 commit into
moonweave wants to merge 1 commit into
Conversation
…helf Run the decisive measurement the philosophy review demanded *before* building the gate-precision redesign (PR #6, now closed). n=14 hand-labeled dev commands; the keelplane verdict comes from the real assess_command_safety (deterministic, not a guess), the codex verdict from --sandbox workspace-write OS rules. Result: codex --sandbox 2 errors vs keelplane command-safety 8 (6 false-stop on plainly-safe dev commands blocked as non-allowlisted, 2 false-pass on substring-evading args). This confirms the critics' prediction that the off-the-shelf stack already wins, so keelplane's self-gate and autonomous code-loop have no dev-domain edge and are ceded; its identity settles on discipline orchestration (design contract + adversarial-verify-against-source). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Runs the decisive measurement the philosophy review demanded before building the gate-precision redesign (PR #6, now closed): keelplane's command-safety gate vs
codex --sandboxon false-pass / false-stop.measurements/gate_vs_sandbox.py— n=14 hand-labeled dev commands. keelplane verdict from the realassess_command_safety(deterministic, not a guess); codex verdict from--sandbox workspace-writeOS rules (network deny + out-of-tree-write block).docs/keelplane-gate-vs-sandbox-measurement.md— decision record.Result
codex --sandboxwins decisively. keelplane false-stops on plainly-safe dev commands (git/pytest/pipblocked as non-allowlisted) and false-passes on substring-evading args (ftp_host:21/exfil,--target /etc/hosts). Confirms the critics' "off-the-shelf already wins."Decision
Cede the dev gate + autonomous loop to off-the-shelf (
codex --sandbox+ thin allowlist + pre-push hook + git diff). keelplane's identity settles on discipline orchestration (design contract + adversarial-verify-against-source), applied to the dev domain. Resolves the direction-vs-domain tension.🤖 Generated with Claude Code