Fleet scaling: surface queue depth + capacity so more container instances spin up when work is queued#1617
Draft
pulzzejaehoon wants to merge 6 commits into
Draft
Fleet scaling: surface queue depth + capacity so more container instances spin up when work is queued#1617pulzzejaehoon wants to merge 6 commits into
pulzzejaehoon wants to merge 6 commits into
Conversation
…v1/me/fleet/capacity)
Expose a claim-faithful, autoscaler-pollable signal of engine queue depth vs
idle container-instance capacity so an operator (or external autoscaler) can
decide whether to spin up more daemon instances. Read-only/derivation only — no
machine-local lease coordination, no in-daemon concurrency state (server
surfaces depth + declares policy; daemon coordinates claims).
- src/lib/engine/fleet-capacity.ts: pure deriveFleetCapacity(queueDepth,
machines) → { queueDepth, online, busy, available, shouldScale,
recommendedInstances }, plus an async getFleetCapacity that sources depth from
listFleetGoalsOrdered (the SAME fleetOwnerId-scoped, occupied-excluded order
the claim path consumes — counts only 'queued' rows) and busy from live leased
runs (isRunLive). shouldScale = queueDepth > available is the server-declared
policy, kept in the pure module.
- Extract the daemon working/idle/stuck/error/offline decision tree from the
fleet-dots route into classifyDaemonMachine (fleet.ts), so the capacity
busy/idle counts and the fleet-dots strip cannot drift (faithful-scope).
- GET /api/v1/me/fleet/capacity via authorizeDaemonOrSession (autoscaler token
or browser session), optional ?projectIds=a,b — mirrors the fleet-queue route.
- Unit tests for the pure derivation (zero machines, all busy, depth 0,
depth>available, capacity>=depth, offline excluded); validated via node
simulation since tsc/vitest can't run in this env.
Agent-config: additive read-only HTTP endpoint, not an agent tool — no new
domain entity/status/workflow/specialist. No PM-agent config change needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e:patch heap
Address review rework on the fleet-capacity goal.
Blocker (system:patch-coverage-below-threshold) — the patch-coverage gate runs
`npm run coverage:patch` and then checks only that coverage/lcov.info EXISTS
(container-loop ignores the command's exit code). The report was missing because
the full v8-instrumented suite (16080 tests) OOM'd at V8's old-space HEAP limit
before vitest could write the report — a heap ceiling, not a physical-RAM limit
(per the env-typecheck-incomplete finding), so the review runner has the RAM, the
run just needed a higher ceiling. Fixes:
- package.json: prefix coverage:patch with NODE_OPTIONS=--max-old-space-size=6144
so the instrumented run reaches the report-writing step. Raising the ceiling
cannot increase a run's actual memory or break a passing run — it only lets a
heap-bound run finish (monotonically safe); matches the repo's inline-env script
style (build/dev:ee).
- Add the missing unit tests so that once a report IS produced, PATCH coverage on
the changed lines clears the 80% floor. Verified via a scoped lcov run:
43/43 changed-instrumented lines covered = 100%.
* fleet-capacity.test.ts: exercise the async getFleetCapacity assembler —
QUEUED-only depth counting, a live-leased machine as busy (with the
isRunLive expired-lease + null-machineId re-filters), the zero-machines
run-query short-circuit, and a recently-failed-but-online machine counted
as available. Mocks @/lib/db + ./fleet-queue; keeps ./fleet REAL so the
shared classifyDaemonMachine/isRunLive predicates are exercised.
* me/fleet/capacity/route.test.ts: 401 path, default all-projects call, and
the comma-separated projectIds parse — mirrors the green fleet-queue route
test (mocks authorizeDaemonOrSession + getFleetCapacity).
The refactored classifyDaemonMachine branches + the fleet-dots route map are
already covered by the existing machines/fleet route test (all five statuses).
Agent-config: still no PM-agent change — these are tests + a build-script tweak,
no agent-visible tool/data/workflow change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… coverage gates Address review rework on the fleet-capacity goal — the two system blockers (system:test-suite-failed and system:patch-coverage-below-threshold) share a single root cause: timing-sensitive suites timing out under the full ~16k-test run, not anything in the fleet-capacity change. The 3 intermittent failures (agent-tools-route beforeEach dynamic-import hook, authorization-fuzzing route import, engine-goals-client jsdom render) all pass in isolation in well under a second each, but under the constrained review container the 50%-worker pool oversubscribes the box and they crossed the old 15s test / default 10s hook ceilings. A timed-out test fails `npm run test` (exit 1) AND aborts the v8-instrumented coverage run before coverage/lcov.info is written — which is exactly the pair of gates that came back red. Fix: raise testTimeout 15s→30s and set hookTimeout 30s (was the 10s default). Raising a timeout only lets a slow run finish; it cannot turn a passing test red, so this is monotonically safe. Verified locally: `npm run coverage:patch` now exits 0, writes coverage/lcov.info (1.1MB), and the full suite reports 16086 passed / 0 failed (87.53% lines, above the 80% floor). No fleet-capacity source change; no PM-agent config change (test-config only). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Poll GET /api/v1/me/fleet/capacity (30s cadence + visibility/realtime handling, alongside the existing fleet-dots/machines/queue polls) and consume the shared FleetCapacity signal. - Add an explicit capacity summary line: "N queued · M idle / K online". - When shouldScale, render an under-provisioned prompt card (styled like the existing setup cards) wired to the /settings/machines Add-machine path, using recommendedInstances in the copy when > 0. - Mirror the pressure as a small amber indicator in the collapsed strip. - Read-only; no claim/lease changes. Gated so an idle fleet or a zero-machine state never shows the scale prompt (the existing "No machines connected" card owns that case). No agent config update needed: read-only operator UI surface, adds no tools and changes no agent-visible domain data. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ect widget Extend the per-project Fleet Activity glance beyond live runs to also show queue depth and idle fleet capacity, reusing the fleet-capacity signal's shared definitions so the two surfaces can never drift. - fleet.ts: extend FleetActivityData with `queueDepth` (this project's runnable-but-unclaimed goals via `listRunnableGoalsWithKeys` — fleetOwnerId scope + occupied-goal exclusion, NOT a new query, bounded by FLEET_QUEUE_DEPTH_CAP) and `machines.busy`/`available`. getProjectFleet now derives capacity through new shared helpers `countMachineCapacity` + `loadDaemonMachineSignals`, kept as the single serializer (route + server render stay byte-identical). - fleet-capacity.ts: refactor deriveFleetCapacity/getFleetCapacity onto the same shared helpers so busy/idle has one definition (faithful-scope). - GET .../engine/fleet: forwards the extended payload (no logic change). - fleet-activity-widget.tsx: header now reads "N running · M queued · online/total (K idle)" with a subtle "under capacity" link to /settings/machines when queueDepth > available. The primary scale prompt stays in the Fleet sidebar. Read-only; no claim/lease changes. Adds widget + helper tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…overage gates The full-suite gate (npm run test) and the patch-coverage gate both aborted on the constrained 8-core review box: 16055/16057 tests passed, but 2 test files died with 'Failed to start forks worker' / 'Timeout waiting for worker to respond' — vitest's hardcoded 60s worker START_TIMEOUT exceeded because a freshly forked worker could not get scheduled through its heavy module-import phase while the other forks saturated the box. Any such abort fails the run and leaves no coverage/lcov.info, blocking the coverage gate too. None of the failing files are touched by this goal. Even at 50% (4 forks here) the spawn window starved. Trimming to 40% (3 forks on an 8-core box) widens the margin so a fresh fork can initialize in time, while a proportional cap still scales up on larger CI hosts. Scheduling only — it cannot turn a passing test red, only let a slow run finish cleanly and emit lcov.info. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integration PR for this goal — its tasks commit onto this branch (one PR per goal). Opened automatically by the engine on first push; left as a draft until the goal completes.