Skip to content

feat(scanner): persistent RPC connection via keep-alive tuning + heartbeat#5

Merged
j4ys0n merged 2 commits into
mainfrom
feat/persistent-rpc-connection
Jul 8, 2026
Merged

feat(scanner): persistent RPC connection via keep-alive tuning + heartbeat#5
j4ys0n merged 2 commits into
mainfrom
feat/persistent-rpc-connection

Conversation

@j4ys0n

@j4ys0n j4ys0n commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Context

Companion to j4ys0n/evmdecoder#49 (published as 0.0.71). Production catch-up sync collapsed to ~1 block per 5 minutes: new TCP connections from the scanner's container to the reth node were SYN-black-holed in multi-minute windows by a stateful network hop, while established connections stayed LAN-fast (whole blocks processed in ~70ms on a warm connection). evmdecoder's keep-alive agent destroyed idle sockets after 4 seconds, so nearly every request burst dialed a fresh connection and re-rolled the dice — and each failure fed a retry storm whose connection churn kept re-triggering the fault. Blocks whose candidate lookups exhausted retries were silently persisted with zero events while the checkpoint advanced.

This change makes the scanner hold one permanently warm connection instead.

Changes

  • evmdecoder ^0.0.70^0.0.71 and pass the new eth.http.freeSocketTimeout through ScannerService (ETH_RPC_FREE_SOCKET_TIMEOUT_MS, default 300 000 ms) — idle keep-alive sockets now survive between request bursts.
  • RPC heartbeat (ETH_RPC_HEARTBEAT_INTERVAL_MS, default 15 000 ms, overlap-guarded, cleared on shutdown): polls eth_blockNumber so the keep-alive connection never idles long enough to be torn down, and maintains a cached latest-block value.
  • /v1/health no longer issues a live RPC call per probe — it serves the heartbeat cache. Previously, during an RPC outage every health probe stalled for the full request timeout (120s) and piled additional requests onto the failing node. createHealthRouter now takes a cache accessor instead of the scanner.
  • .env.example documents the new vars; health controller test updated to the new signature.

Verification

Against the published evmdecoder 0.0.71 from npm:

  • yarn lint (tsc --noEmit) clean
  • yarn test: 20 files, 185 tests passing
  • yarn build clean
  • Confirmed the published package contains the fixes (freeSocketTimeout in lib/eth/http.js, non-mutating chunkArray in lib/utils/obj.js)

Deploy notes

  • docker-compose.yml on the prod host currently pins ghcr.io/MissionSquad/agentindex-api:1.0.0 — build/push this as 1.5.3 and update the compose tag.
  • Defaults are safe: with no new env vars set, behavior is 5-minute socket keep-alive + 15s heartbeat.
  • The underlying network fault (new-connection SYN blackhole between the container and the node) still exists independently of this fix — this makes the scanner resilient to it and stops the connect-churn that appears to trigger it. If sync ever degrades again, run the concurrent laptop-vs-container curl loops to localize the dropping hop (EFG security engine vs reth-side accept path).

🤖 Generated with Claude Code

j4ys0n and others added 2 commits July 7, 2026 23:58
…tbeat

Companion to evmdecoder 0.0.71 (j4ys0n/evmdecoder#49). Production sync
collapsed because new TCP connections to the RPC node were black-holed in
windows by a stateful network hop while established connections stayed fast;
evmdecoder's 4s free-socket timeout meant nearly every request burst dialed a
new connection. This change keeps one connection permanently warm:

- bump evmdecoder to ^0.0.71 and pass the new eth.http.freeSocketTimeout
  (ETH_RPC_FREE_SOCKET_TIMEOUT_MS, default 300000) so idle keep-alive
  sockets survive between request bursts
- add an RPC heartbeat (ETH_RPC_HEARTBEAT_INTERVAL_MS, default 15000,
  overlap-guarded) that refreshes a cached latest-block value and keeps the
  keep-alive connection from ever idling long enough to be torn down
- serve /v1/health from the heartbeat cache instead of issuing a live
  eth_blockNumber per probe — during RPC outages each probe stalled for the
  full request timeout and piled more load onto the failing node

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@j4ys0n j4ys0n merged commit 7756dcf into main Jul 8, 2026
1 check passed
@j4ys0n j4ys0n deleted the feat/persistent-rpc-connection branch July 8, 2026 06:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant