feat(scanner): persistent RPC connection via keep-alive tuning + heartbeat#5
Merged
Conversation
…tbeat Companion to evmdecoder 0.0.71 (j4ys0n/evmdecoder#49). Production sync collapsed because new TCP connections to the RPC node were black-holed in windows by a stateful network hop while established connections stayed fast; evmdecoder's 4s free-socket timeout meant nearly every request burst dialed a new connection. This change keeps one connection permanently warm: - bump evmdecoder to ^0.0.71 and pass the new eth.http.freeSocketTimeout (ETH_RPC_FREE_SOCKET_TIMEOUT_MS, default 300000) so idle keep-alive sockets survive between request bursts - add an RPC heartbeat (ETH_RPC_HEARTBEAT_INTERVAL_MS, default 15000, overlap-guarded) that refreshes a cached latest-block value and keeps the keep-alive connection from ever idling long enough to be torn down - serve /v1/health from the heartbeat cache instead of issuing a live eth_blockNumber per probe — during RPC outages each probe stalled for the full request timeout and piled more load onto the failing node Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Companion to j4ys0n/evmdecoder#49 (published as 0.0.71). Production catch-up sync collapsed to ~1 block per 5 minutes: new TCP connections from the scanner's container to the reth node were SYN-black-holed in multi-minute windows by a stateful network hop, while established connections stayed LAN-fast (whole blocks processed in ~70ms on a warm connection). evmdecoder's keep-alive agent destroyed idle sockets after 4 seconds, so nearly every request burst dialed a fresh connection and re-rolled the dice — and each failure fed a retry storm whose connection churn kept re-triggering the fault. Blocks whose candidate lookups exhausted retries were silently persisted with zero events while the checkpoint advanced.
This change makes the scanner hold one permanently warm connection instead.
Changes
^0.0.70→^0.0.71and pass the neweth.http.freeSocketTimeoutthroughScannerService(ETH_RPC_FREE_SOCKET_TIMEOUT_MS, default 300 000 ms) — idle keep-alive sockets now survive between request bursts.ETH_RPC_HEARTBEAT_INTERVAL_MS, default 15 000 ms, overlap-guarded, cleared on shutdown): pollseth_blockNumberso the keep-alive connection never idles long enough to be torn down, and maintains a cached latest-block value./v1/healthno longer issues a live RPC call per probe — it serves the heartbeat cache. Previously, during an RPC outage every health probe stalled for the full request timeout (120s) and piled additional requests onto the failing node.createHealthRouternow takes a cache accessor instead of the scanner..env.exampledocuments the new vars; health controller test updated to the new signature.Verification
Against the published evmdecoder 0.0.71 from npm:
yarn lint(tsc --noEmit) cleanyarn test: 20 files, 185 tests passingyarn buildcleanfreeSocketTimeoutinlib/eth/http.js, non-mutatingchunkArrayinlib/utils/obj.js)Deploy notes
docker-compose.ymlon the prod host currently pinsghcr.io/MissionSquad/agentindex-api:1.0.0— build/push this as 1.5.3 and update the compose tag.🤖 Generated with Claude Code