fix(http): externalize streaming cursor so table functions run over HTTP (ungate)#3
Merged
Merged
Conversation
…TTP; ungate
Correct the earlier conclusion: streaming table functions CAN run over the
stateless HTTP transport — they just need their position externalized into the
gob-serialized producer state. Fix the worker and run the FULL suite over all
three transports (no HTTP_GATED_TESTS).
Root cause (verified empirically by instrumenting Process over the http leg):
each Process tick arrived with Done=false, rows=N — the post-Emit `Done bool`
mutation did not survive the HTTP continuation boundary. Over HTTP the worker is
stateless across exchanges; the framework round-trips producer state through a
continuation token (gob-snapshotting the user state each tick, emitting ≤1 data
batch per response — producerBatchLimit=1 — then resuming from the token). A
Done flag flipped after the single Emit observes the pre-Emit snapshot on
resume, re-emits the same rows forever, and pins the worker (50k+ ticks, never
terminating). subprocess/unix keep live state in memory, so they were unaffected
and hid the bug.
Fix — explicit gob-encodable cursor (the reference pattern for all Go streaming
table functions):
- Replace the shared `emitState{Done bool}` with `CursorState{Rows []CVERow;
Offset int}` (exported TYPE — the SDK's registration check counts named
exported fields, so an embedded unexported-named helper would panic at start).
- nextSlice() returns the next bounded slice (rowsPerTick=64) and advances
Offset BEFORE yielding; Process emits it and Finish()es when Offset>=len(Rows).
- The framework snapshots Offset into each continuation token, so HTTP resumes
from the right row and terminates. Empirically: cve()->1, cve_search()->3,
cpe_cves()->3 (incl. NULL score) all return cleanly over http.
CI:
- ci/run-integration.sh: remove HTTP_GATED_TESTS; stage and run the full suite
(incl. cve_api.test) for every transport. Keep the mock NVD server (all
transports), the http httpfs-inject, and the fail-on-silent-skip guard.
Tests/docs:
- functions_test.go: add TestCursorSurvivesContinuation — gob round-trips the
state between every simulated tick and asserts the cursor advances, emits each
row exactly once, and terminates (fast regression guard for the loop). Update
the NewState test to assert Offset==0 instead of !Done.
- Bump github.com/Query-farm/vgi-go v0.1.2 -> v0.2.0 (pulls vgi-rpc-go v0.9.4):
carries the table-state gob-encodability registration check and http
producer-continuation fixes. The cursor fix is required and sufficient on top
of it (v0.1.2 looped identically with a bare Done flag).
- CLAUDE.md + ci/README.md: document the cursor pattern and drop the gate.
Local validation (haybarn-unittest v1.5.4-rc1): subprocess GREEN (25), unix
GREEN (25), http GREEN (29, full suite incl. table functions). go test, vet,
gofmt, golangci-lint all clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the
Done booltable-function state with an explicit gob-encodable cursor (CursorState{ Rows; Offset }) so producer state round-trips correctly across stateless HTTP continuations. Removes thecve_api.testHTTP gate — all SQL tests now run over subprocess + unix + http.🤖 Generated with Claude Code