Skip to content

fix Codex HTML output pipeline#51

Open
BobbyCats wants to merge 1 commit into
nexu-io:mainfrom
BobbyCats:fix/codex-html-output-pipeline
Open

fix Codex HTML output pipeline#51
BobbyCats wants to merge 1 commit into
nexu-io:mainfrom
BobbyCats:fix/codex-html-output-pipeline

Conversation

@BobbyCats
Copy link
Copy Markdown

@BobbyCats BobbyCats commented May 16, 2026

背景

在 Codex CLI 作为 agent 时，我复现了几类集中在 agent invoke/output pipeline 的问题：

  • codex exec --json 会读取用户本机 Codex config。只要 config 中存在暂时不可用的 MCP server，本次 HTML 生成就可能被 unrelated MCP transport/auth errors 打断。
  • 当前 Codex JSONL 会输出 thread.started / turn.started / item.completed，其中 assistant payload 可能是 item.type: "agent_message"，并且 usage 会出现在 turn.completed.usage。现有 parser 只识别旧的 assistant_message shape。
  • /api/convert 需要的是 HTML artifact，/api/draft 需要的是 markdown/text。两条路径共用 delta 处理时，assistant preamble 容易污染 HTML source 和 preview metrics。
  • 运行中 elapsed/waiting 文案没有独立 tick，长时间生成时 UI 会显得停住。

改动

  • 让 Codex adapter 使用更 hermetic 的 invocation flags：--ignore-user-config、--ignore-rules、--ephemeral，避免用户本机 MCP/rules/session state 影响 HTML Anything 的一次性 generation。
  • 扩展 Codex JSONL parser，识别 thread.started、turn.started、item.completed.item.type === "agent_message"、旧版 assistant_message、以及 turn.completed.usage。
  • 在 invokeAgent 增加 output: "text" | "html" seam：
    • /api/convert 走 output: "html"
    • /api/draft 走 output: "text"
  • 新增 HtmlStreamExtractor，只在 HTML document start (<!DOCTYPE html> / <html>) 之后向 preview/source 输出，并在 </html> 后截断 trailing prose。
  • 修复 preview/source metrics 的 live elapsed timer，使等待状态在 agent 运行期间持续更新。
  • 增加 node:test 覆盖 HTML extraction 和 Codex parser shapes。

依赖说明

新增 tsx 作为 dev-only test runner，用于直接运行 TypeScript 测试文件；runtime code 没有 import 它，生产 bundle 不依赖它。

边界

这个 PR 没有声称给 Codex CLI 增加 token-level streaming。如果 Codex CLI 在 exec --json 下只在最后输出一个 item.completed，应用仍然只能收到一个 content chunk。这里修的是：Codex invocation 不被 unrelated user config 打断、当前 JSONL shape 能被 parser 正确识别、HTML 输出不会被 preamble 污染、长时间运行时 UI metrics 保持可见。

验证

  • pnpm test：6 tests passed
  • pnpm exec tsc --noEmit
  • pnpm build：passed；保留现有 Turbopack NFT trace warning，没有新增 build failure
  • Manual smoke：Codex /api/convert 能返回以 <!DOCTYPE html> 开头的 HTML delta，没有 MCP transport/auth error；Codex /api/draft 仍返回 markdown/text，不被 HTML extractor 拦截

@lefarcen lefarcen requested a review from Siri-Ray May 16, 2026 16:44
@lefarcen lefarcen added size/L Large change: 300-699 changed lines risk/high High-risk PR: dependencies, infra, security-sensitive, or broad runtime impact type/bugfix Bug fix labels May 16, 2026
This was referenced May 16, 2026
Closed
Closed
Draft
@PerishCode
Copy link
Copy Markdown
Contributor

PerishCode commented May 18, 2026

Thanks for the detailed PR. I am not going to patch this branch directly because it touches several coupled agent-output paths: Codex invocation flags, JSONL parsing, HTML extraction, /api/convert vs /api/draft, and live preview metrics. That is too easy for a maintainer rebase to accidentally change the intended behavior.

Suggested migration to current main:

  • Move app code under the Next package paths:
    • src/app/... -> next/src/app/...
    • src/components/... -> next/src/components/...
    • src/lib/... -> next/src/lib/...
  • Do not add root package.json scripts or root test runners. The root package is now workspace metadata only.
  • Convert scripts/test-agent-output.ts into Vitest coverage inside the Next package, for example next/src/lib/agents/html-stream.test.ts and next/src/lib/agents/argv.test.ts.
  • Keep package changes scoped to next/package.json only if a new dev dependency is still needed. If Vitest covers the tests, tsx should not be needed for this PR.

Recommended regression coverage before review:

  • Codex current JSONL shapes: thread.started, turn.started, item.completed with both agent_message and legacy assistant_message, plus turn.completed.usage.
  • HTML extraction: ignores preamble before <!doctype html> / <html>, emits the document, and cuts trailing prose after </html>.
  • Text mode: /api/draft still returns plain markdown/text and is not filtered by the HTML extractor.
  • Invocation flags: verify the proposed Codex flags against the current installed Codex CLI help/output, since this is version-sensitive.

Recommended validation:

  • pnpm install --frozen-lockfile
  • pnpm exec tsx scripts/guard.ts
  • pnpm -F @html-anything/next typecheck
  • pnpm -F @html-anything/next test
  • pnpm -F @html-anything/next build

Once the parser/extractor behavior is covered in Vitest under next/, the review can focus on the product behavior instead of the workspace migration.

@lefarcen lefarcen mentioned this pull request May 19, 2026
Open
@BobbyCats BobbyCats force-pushed the fix/codex-html-output-pipeline branch from 1f83b30 to ed8b26c Compare May 20, 2026 17:47
@BobbyCats
Copy link
Copy Markdown
Author

BobbyCats commented May 20, 2026

Updated this PR against current main and force-pushed a migrated head commit.

Migration changes:

  • moved the app changes under next/src/...
  • removed the root test runner / root package changes from this PR; this revision has no root package.json or lockfile changes
  • converted regression coverage to Vitest under next/src/lib/agents/argv.test.ts and next/src/lib/agents/html-stream.test.ts
  • kept /api/convert on output: "html" and /api/draft on output: "text"
  • verified the installed Codex CLI help includes --ignore-user-config, --ignore-rules, and --ephemeral

Validation run:

  • pnpm install --frozen-lockfile
  • pnpm exec tsx scripts/guard.ts
  • pnpm -F @html-anything/next typecheck
  • pnpm -F @html-anything/next test — 7 files, 68 tests passed
  • pnpm -F @html-anything/next build — passed; existing Turbopack NFT trace warning is still present

@lefarcen lefarcen added size/M Medium change: 100-299 lines risk/medium Medium risk change and removed risk/high High-risk PR: dependencies, infra, security-sensitive, or broad runtime impact size/L Large change: 300-699 changed lines labels May 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Siri-Ray Siri-Ray Awaiting requested review from Siri-Ray

Assignees

No one assigned

Labels

risk/medium Medium risk change size/M Medium change: 100-299 lines type/bugfix Bug fix

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@BobbyCats @PerishCode @lefarcen