Conversation
…ution reporting - Added `scope` option to CLI for specifying test depth: quick, standard, or thorough. - Updated `runHeadless` and related functions to handle new scope tier logic. - Enhanced execution reporting to include detailed step results and durations. - Implemented auth redirect warning in the browser MCP server. - Introduced new constants for run management in the supervisor module.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
commit: |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
3 issues found across 11 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/cli/src/index.tsx">
<violation number="1" location="apps/cli/src/index.tsx:71">
P2: `--scope` is exposed as a global option, but it only affects the headless execution path. In interactive runs it is ignored, so users can pass `--scope quick` and still get default behavior.</violation>
</file>
<file name="apps/cli/src/data/execution-atom.ts">
<violation number="1" location="apps/cli/src/data/execution-atom.ts:210">
P2: `finalExecuted.id ?? crypto.randomUUID()` does not handle empty-string ids, so runs with `id: ""` all write to the same `.expect/runs/.json` file.</violation>
</file>
<file name="apps/cli/src/utils/run-test.ts">
<violation number="1" location="apps/cli/src/utils/run-test.ts:405">
P2: Use a truthy fallback for `finalExecuted.id` so empty-string IDs don’t collapse to `.expect/runs/.json` and overwrite previous run results.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
| ) | ||
| .option("-t, --target <target>", "what to test: unstaged, branch, or changes", "changes") | ||
| .option( | ||
| "-s, --scope <tier>", |
There was a problem hiding this comment.
P2: --scope is exposed as a global option, but it only affects the headless execution path. In interactive runs it is ignored, so users can pass --scope quick and still get default behavior.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/cli/src/index.tsx, line 71:
<comment>`--scope` is exposed as a global option, but it only affects the headless execution path. In interactive runs it is ignored, so users can pass `--scope quick` and still get default behavior.</comment>
<file context>
@@ -63,6 +67,11 @@ const program = new Command()
)
.option("-t, --target <target>", "what to test: unstaged, branch, or changes", "changes")
+ .option(
+ "-s, --scope <tier>",
+ "test depth: quick (one check, ~30s), standard (primary + follow-ups), thorough (full audit)",
+ "standard",
</file context>
| summary: `${summaryParts.join(", ")} out of ${report.steps.length} step${report.steps.length === 1 ? "" : "s"}`, | ||
| }); | ||
|
|
||
| yield* writeRunResult(finalExecuted.id ?? crypto.randomUUID(), resultOutput); |
There was a problem hiding this comment.
P2: finalExecuted.id ?? crypto.randomUUID() does not handle empty-string ids, so runs with id: "" all write to the same .expect/runs/.json file.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/cli/src/data/execution-atom.ts, line 210:
<comment>`finalExecuted.id ?? crypto.randomUUID()` does not handle empty-string ids, so runs with `id: ""` all write to the same `.expect/runs/.json` file.</comment>
<file context>
@@ -175,6 +179,36 @@ const executeCore = (input: ExecuteInput) =>
+ summary: `${summaryParts.join(", ")} out of ${report.steps.length} step${report.steps.length === 1 ? "" : "s"}`,
+ });
+
+ yield* writeRunResult(finalExecuted.id ?? crypto.randomUUID(), resultOutput);
+
return {
</file context>
| yield* writeRunResult(finalExecuted.id ?? crypto.randomUUID(), resultOutput); | |
| yield* writeRunResult(finalExecuted.id && finalExecuted.id.length > 0 ? finalExecuted.id : crypto.randomUUID(), resultOutput); |
| }); | ||
|
|
||
| const runResultPath = yield* writeRunResult( | ||
| finalExecuted.id ?? crypto.randomUUID(), |
There was a problem hiding this comment.
P2: Use a truthy fallback for finalExecuted.id so empty-string IDs don’t collapse to .expect/runs/.json and overwrite previous run results.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/cli/src/utils/run-test.ts, line 405:
<comment>Use a truthy fallback for `finalExecuted.id` so empty-string IDs don’t collapse to `.expect/runs/.json` and overwrite previous run results.</comment>
<file context>
@@ -366,39 +369,47 @@ export const runHeadless = (options: HeadlessRunOptions) =>
+ });
+
+ const runResultPath = yield* writeRunResult(
+ finalExecuted.id ?? crypto.randomUUID(),
+ resultOutput,
+ );
</file context>
| finalExecuted.id ?? crypto.randomUUID(), | |
| finalExecuted.id || crypto.randomUUID(), |
Test Results✅ Website Test: passed9 passed, 0 failed out of 10 steps — 172s
Session Recording |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
| // agents (Cursor, Claude Code, Codex) can read a single file instead of | ||
| // polling terminal output. Each run gets a unique planId (UUID), enabling | ||
| // parallel agent sessions without file conflicts. | ||
|
|
There was a problem hiding this comment.
Constants EXPECT_STATE_DIR, EXPECT_RUNS_DIR, and EXPECT_RUNS_MAX_KEPT are duplicated in write-run-result.ts instead of being imported from supervisor
| yes?: boolean; | ||
| agent?: AgentBackend; | ||
| target?: Target; | ||
| scope?: ScopeTier; |
There was a problem hiding this comment.
The CommanderOpts interface declares scope/target/output/agent with literal union types (ScopeTier, Target, OutputFormat, AgentBackend), but Commander provides raw strings for all options, making the type annotation misleading about type safety.
| }); | ||
|
|
||
| const totalDurationMs = getTotalElapsedMs(report.steps) || durationMs; | ||
| const summaryParts = [`${passedCount} passed`, `${failedCount} failed`]; |
There was a problem hiding this comment.
The summary construction in execution-atom.ts is missing the skipped count, resulting in inconsistent output compared to run-test.ts
…ers and execution results
- Bumped version from 2.2.0 to 2.3.0.
- Added detailed documentation on scope tiers for testing depth: quick, standard, and thorough.
- Included instructions for reading structured run result files from `.expect/runs/{planId}.json`.
| summary: `${summaryParts.join(", ")} out of ${report.steps.length} step${report.steps.length === 1 ? "" : "s"}`, | ||
| }); | ||
|
|
||
| yield* writeRunResult(finalExecuted.id ?? crypto.randomUUID(), resultOutput); |
There was a problem hiding this comment.
Using the ?? operator doesn't properly handle empty-string IDs, causing multiple runs to overwrite the same file
| }); | ||
|
|
||
| const runResultPath = yield* writeRunResult( | ||
| finalExecuted.id ?? crypto.randomUUID(), |
There was a problem hiding this comment.
The nullish coalescing operator (??) doesn't treat empty strings as falsy, causing empty-string IDs to be used instead of generating UUIDs
1 participant
Summary by cubic
Adds scoped test tiers (quick, standard, thorough) to control test depth from the CLI and writes structured run results to
.expect/runsfor easy consumption by external agents. Also adds an auth-redirect warning when a page opens to a login/auth screen.New Features
--scope <tier>CLI flag (quick,standard,thorough; defaultstandard), validated and passed through to the supervisor.@expect/shared/prompts(focused checks in quick, audits and cross-browser in thorough)..expect/runs/{planId}.jsonwith status, durations, step results, and artifacts; keeps last 20 runs; path is printed in non-JSON mode.opentool when the app redirects to an auth page; suggests marking auth-blocked steps.Refactors
ScopeTierin@expect/shared/models; executor and CLI accept and forward it.NodeServiceslayer in the CLI to enable filesystem effects for result writing.EXPECT_RUNS_DIRandEXPECT_RUNS_MAX_KEPTconstants in@expect/supervisor; updated tests for new tier behavior.packages/expect-skillto v2.3.0 and expanded docs for scope tiers and.expect/runsresult files.Written for commit 95ce694. Summary will update on new commits.