Add CI smoke/integration tests and fake provider for testability#170
Add CI smoke/integration tests and fake provider for testability#170michaelerobertsjr wants to merge 6 commits intoTinyAGI:mainfrom
Conversation
Greptile SummaryThis PR adds a comprehensive automated-testing layer to TinyClaw: a GitHub Actions CI workflow, a restructured three-stage release pipeline (build → verify-smoke → publish), Key findings:
Confidence Score: 3/5
Important Files Changed
Sequence DiagramsequenceDiagram
participant Dev as Developer
participant CI as ci.yml
participant Rel as release.yml (build-and-release)
participant VS as release.yml (verify-smoke)
participant PR as release.yml (publish-release)
participant GH as GitHub Releases
Dev->>CI: push / pull_request
CI->>CI: integration tests (ubuntu, Node 20)
CI->>CI: smoke tests (ubuntu + macos, Node 20)
Dev->>Rel: push vX.Y.Z tag
Rel->>Rel: verify tag == package.json version
Rel->>Rel: npm ci + build + prune dev deps
Rel->>Rel: bundle tarball (src, dist, tests, node_modules…)
Rel->>VS: upload tinyclaw-bundle artifact
VS->>VS: download + extract bundle
VS->>VS: npm ci --include=dev (ubuntu + macos)
VS->>VS: npm run test:smoke
VS-->>PR: smoke pass (tag builds only)
PR->>PR: checkout + download bundle
PR->>PR: extract release notes from annotated tag
PR->>GH: create GitHub Release + attach tarball
Last reviewed commit: a4c19e6 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8cc482a4aa
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- add GitHub CI workflow and release smoke-test gate on Node 20 - add tsx-based integration/smoke test suites with shared test helpers - add internal fake provider and queue retry/dead event emission for deterministic testing - allow install dir override via `TINYCLAW_INSTALL_DIR`
- Restrict fake provider to test mode unless explicitly allowed via env var - Emit chain step completion events on invoke errors and finish team conversations when a branch dead-letters - Add integration coverage for dead team branches, align retry assertion to shared MAX_RETRIES, and clarify install.sh PATH note for custom install dir
Adds an explicit timeout-minutes at the job level (e.g., 15–20 minutes) Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
- export `getFreePort` from fixture helper for reuse - retry queue processor startup up to 5 times when `EADDRINUSE` occurs, reallocating API port - treat `null` like `undefined` in `waitFor` so polling only resolves on real values
michaelerobertsjr
force-pushed
the
feature/ci-integration-smoke-tests
branch
from
5d5e522 to
33ce4e5
Compare
- Remove duplicate `integration` job stub in CI workflow - Scope release workflow permissions per job (`read` for smoke, `write` for release) - Fail install if wrapper symlink is missing/non-executable and report PATH precedence issues
- Add dedicated integration job in `ci.yml` and run smoke matrix separately - Rework `release.yml` to build once, verify smoke from bundle artifacts, then publish on tags - Preserve resolved routed agent ID in dead-branch events and pending-branch cleanup - Surface SSE stream read errors in tests and only suppress expected `AbortError` on close
|
@greptileai can you take another look |
| cp -r lib/ "$BUNDLE_DIR/" | ||
| cp -r docs/ "$BUNDLE_DIR/" | ||
| cp -r .agents/ "$BUNDLE_DIR/" | ||
| cp -r tests/ "$BUNDLE_DIR/" |
There was a problem hiding this comment.
Test files included in the production release tarball
tests/ is being copied into the bundle, so every end-user who downloads the official GitHub release tarball also receives the full test harness (integration helpers, fake providers, SSE fixtures, etc.).
More critically, by the time this bundle step runs, npm prune --omit=dev has already stripped tsx from node_modules/. Any user who extracts the release tarball and tries to run npm run test:smoke or npm run test:integration directly will get an error because tsx is absent from the production node_modules.
The cleanest fix is to keep the release tarball test-free and instead upload a dedicated test-only artifact for the verify-smoke job:
# In build-and-release — create a separate test artifact, don't include tests/ in the main bundle
- name: Upload test sources artifact
uses: actions/upload-artifact@v4
with:
name: tinyclaw-tests
path: |
tests/
package.json
package-lock.json
tsconfig.json
# Remove cp -r tests/ from the bundle tarball stepThen in verify-smoke, download both tinyclaw-bundle and tinyclaw-tests, overlay the tests directory on the extracted bundle, run npm ci --include=dev, and execute the smoke tests. This keeps the release artifact clean while the CI pipeline still has everything it needs.
| verify-smoke: | ||
| needs: build-and-release | ||
| permissions: | ||
| contents: read | ||
| runs-on: ${{ matrix.os }} | ||
| strategy: | ||
| matrix: | ||
| os: [ubuntu-latest, macos-latest] | ||
|
|
||
| steps: | ||
| - name: Download bundle artifact | ||
| uses: actions/download-artifact@v4 | ||
| with: | ||
| name: tinyclaw-bundle | ||
|
|
||
| - name: Extract bundle | ||
| shell: bash | ||
| run: | | ||
| EXTRACT_DIR="$(mktemp -d)" | ||
| tar -xzf tinyclaw-bundle.tar.gz -C "$EXTRACT_DIR" | ||
| echo "BUNDLE_DIR=$EXTRACT_DIR/tinyclaw" >> "$GITHUB_ENV" | ||
|
|
||
| - name: Setup Node.js | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: '20' | ||
|
|
||
| - name: Install dependencies | ||
| working-directory: ${{ env.BUNDLE_DIR }} | ||
| run: | | ||
| PUPPETEER_SKIP_DOWNLOAD=true npm ci --include=dev | ||
|
|
||
| - name: Run smoke tests | ||
| working-directory: ${{ env.BUNDLE_DIR }} | ||
| run: npm run test:smoke |
There was a problem hiding this comment.
verify-smoke job missing timeout-minutes
The verify-smoke job spawns real queue processors, runs HTTP polling loops, and waits on SSE streams — the same class of operations that prompted the timeout-minutes: 15 that was added to the integration job in ci.yml. Without an explicit cap, a hung smoke test on Ubuntu or macOS will hold the runner for the full GitHub Actions default of 6 hours before failing, blocking any tag release.
| verify-smoke: | |
| needs: build-and-release | |
| permissions: | |
| contents: read | |
| runs-on: ${{ matrix.os }} | |
| strategy: | |
| matrix: | |
| os: [ubuntu-latest, macos-latest] | |
| steps: | |
| - name: Download bundle artifact | |
| uses: actions/download-artifact@v4 | |
| with: | |
| name: tinyclaw-bundle | |
| - name: Extract bundle | |
| shell: bash | |
| run: | | |
| EXTRACT_DIR="$(mktemp -d)" | |
| tar -xzf tinyclaw-bundle.tar.gz -C "$EXTRACT_DIR" | |
| echo "BUNDLE_DIR=$EXTRACT_DIR/tinyclaw" >> "$GITHUB_ENV" | |
| - name: Setup Node.js | |
| uses: actions/setup-node@v4 | |
| with: | |
| node-version: '20' | |
| - name: Install dependencies | |
| working-directory: ${{ env.BUNDLE_DIR }} | |
| run: | | |
| PUPPETEER_SKIP_DOWNLOAD=true npm ci --include=dev | |
| - name: Run smoke tests | |
| working-directory: ${{ env.BUNDLE_DIR }} | |
| run: npm run test:smoke | |
| verify-smoke: | |
| needs: build-and-release | |
| timeout-minutes: 20 | |
| permissions: | |
| contents: read | |
| runs-on: ${{ matrix.os }} | |
| strategy: | |
| matrix: | |
| os: [ubuntu-latest, macos-latest] |
| - name: Setup Node.js | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: '20' |
There was a problem hiding this comment.
Missing cache: 'npm' in verify-smoke Node.js setup
Every other job in both workflows (build-and-release, integration, smoke) passes cache: 'npm' to actions/setup-node. Without it, verify-smoke downloads and installs the full npm dependency tree from the network on every run, adding unnecessary latency to every release verification.
| - name: Setup Node.js | |
| uses: actions/setup-node@v4 | |
| with: | |
| node-version: '20' | |
| - name: Setup Node.js | |
| uses: actions/setup-node@v4 | |
| with: | |
| node-version: '20' | |
| cache: 'npm' |
1 participant
TINYCLAW_INSTALL_DIRSummary
This PR adds deterministic automated coverage for TinyClaw’s core runtime and release path. It introduces offline integration and smoke tests around the real queue processor, API, SSE stream, SQLite DB, and installed CLI, then gates CI and releases on those checks so regressions are caught before shipping.
Changes
Added a new GitHub Actions CI workflow with:
tsx + node:test basedtest scripts in package.json*Added a CLI smoke test covering install script behavior, installed wrapper help output, agent list, and minimal fake-provider runtime flow
TINYCLAW_FAKE_PROVIDER_MODETINYCLAW_FAKE_PROVIDER_DELAY_MSscripts/install.shto honorTINYCLAW_INSTALL_DIRtinyclaw.shto supporthelp,-h, and--helpType of Change
Testing
npm run test:integrationnpm run test:smokenpm run buildChecklist