Bandwidth amplification during backend outages (resend queue + ack loss)

[full-bars]

Summary

During URnetwork backend outages, significant volumes of traffic leak through the data plane even though the control plane (API) is unreachable. This was identified during code review on #180.

Suspected mechanism

The resend queue in transfer.go and the multi-client race in ip_remote_multi_client.go interact to amplify traffic when the platform is degraded:

1. Resend queue loops on ack loss (transfer.go ~line 1569–1605)

When the platform is degraded, writes to providers can still succeed (data plane connections may remain up) but acks are never returned. The resend queue re-queues items unconditionally after each write — whether the write succeeded or failed — using the last known scaled RTT as the retry interval:

err := c()
if err != nil {
    glog.V(1).Infof("[s]resend drop = %s", err)
}
// re-queued regardless of whether write succeeded or failed
item.sendCount += 1
item.resendTime = sendTime.Add(self.rttWindow.ScaledRtt())
self.resendQueue.Add(item)

A packet in-flight at the time of the outage can be retransmitted many times before the send sequence closes at CreateContractTimeout (30s default). At a pre-outage RTT of 50ms, a single packet could be sent ~600 times in that window.

2. Multi-client race duplicates packets (ip_remote_multi_client.go ~line 1245–1264)

When no established client exists for a path, MultiRaceClientCount clients race in parallel. Each receives a MessagePoolShareReadOnly copy of the packet and sends it independently:

for _, client := range raceOrderedClients {
    go send(client) // each client sends its own copy
}

During an outage, the window expansion loop (ip_remote_multi_client.go ~line 2106) continues creating new clients every 15s (WindowResizeTimeout). More clients in the window means more copies of each packet sent per resend cycle.

3. No global circuit breaker

There is no mechanism that detects a backend outage and halts packet acceptance or resend attempts across all sequences. Each send sequence independently retries for up to 30 seconds before closing.

Expected vs observed behavior

• Expected: when the API is unreachable, traffic stops or degrades gracefully to near-zero
• Observed: large volumes of traffic continue to flow (and amplify) for up to 30s per sequence per packet

Suggested investigation areas

• Adding a per-packet max resend count to the resend queue
• Coordinating outage detection across sequences (a shared backoff flag similar to CreateContractOobErrorBackoff added in contract,transport: reduce log spam during backend outages #180, but for the data plane)
• Reviewing whether MultiRaceClientCount should be reduced or bypassed when the platform is known to be degraded

Related

• PR contract,transport: reduce log spam during backend outages #180 (log spam reduction, where this was identified)

[full-bars]

Follow-up from deeper investigation.

Corrected numbers

The issue description references CreateContractTimeout (30s upstream) as the sequence window. Two notes on the actual window:

• AckTimeout (60s in both upstream and this repo) is the binding limit — a send sequence closes when it hasn't received an ack within 60s, regardless of contract timeout. So the upstream window is already 60s, not 30s.
• Our fork additionally doubled CreateContractTimeout to 60s, making both timeouts consistent. This doesn't change the effective window for the resend amplification, but it's worth noting for anyone reproducing against stock upstream.

At 50ms scaled RTT the per-packet resend count is closer to 600, not 300. MultiRaceClientCount defaults to 0, which means all ordered clients race — not a fixed small number. Combined: 600 retries × N clients (growing every 15s via WindowResizeTimeout) = 6,000+ copies of a single packet before the sequence closes.

Proposed fix (four changes)

1. MaxResendCount = 16 in SendBufferSettings — hard cap on resend attempts per packet. Reduces ~600 retries to 16 regardless of outage state. No cross-instance coordination needed.

2. MultiRaceClientCount default: 0 → 2 — caps the per-retry fan-out to at most 2 parallel copies instead of all available clients.

3. Exponential backoff when backend is degraded — when lastAuthErrLogNano or lastOobErrLogNano (from contract,transport: reduce log spam during backend outages #180) indicates recent backend errors, apply exponential backoff to resendTime instead of raw ScaledRtt(). Spreads the 16 retries across the full 60s AckTimeout window rather than burning through them in the first 3–4 seconds.

4. Pause window expansion when backend is degraded — gate the expand() call in runWindowSize on !isBackendDegraded(). Prevents the race fan-out multiplier from growing over the course of a long outage.

Estimated reduction: ~200× less data plane traffic during an outage (32 packet copies max vs 6,000+). Normal operation is unaffected — acks arrive quickly and packets exit the resend queue after 1–2 retries.

Implementation is in progress against the fork at full-bars/urnetwork-3.23-fix. Will open a PR against this repo once validated.

[full-bars]

Update — confirmed mechanism + two additional control-plane amplifiers

The resend queue amplification described above is confirmed. After deeper code analysis I found two additional amplifiers that weren't captured here, which together explain the full scale of the observed leak.

Amplifier A — transport reconnect storm

runH1 and runH3 create a Reconnect timer at the start of each loop iteration. When a connection fails quickly (TCP refused, sub-millisecond), the timer has already exceeded its minTimeout and After() returns immediately. The loop fires again without any real delay. Under a sustained outage this produces hundreds of failed connection attempts per second per transport, each making a full parallelEval call across all dialers.

Fix: exponential backoff in the auth error path — 5s → 10s → 20s → 40s → 60s cap per consecutive failure, reset on successful reconnect.

Amplifier B — contract creation storm

Every active SendSequence calls ContractManager.CreateContract every 5 seconds (CreateContractRetryInterval) regardless of backend state. Each call dispatches a goroutine via ClientOob().SendControl → ConnectControl → HttpSerial, which holds a goroutine for up to RequestTimeout (15s) attempting to reach the dead API. With 100+ concurrent flows this sustains thousands of concurrent goroutines during an outage.

Fix: gate the CreateContract call behind isBackendDegraded() — skip it entirely when the backend is known unreachable. Raise the poll interval from 5s to 30s while degraded.

Notes on the degraded detection signal

The existing isBackendDegraded() used the rate-limited log-suppression timestamps (lastAuthErrLogNano, lastOobErrLogNano), which update at most once per minute. This creates a brief window near the 60s boundary where the check incorrectly returns false even during a sustained outage. Fixed by adding a separate lastBackendFailNano atomic updated on every failure (not rate-limited).

Interaction between the three amplifiers

All three fire independently and compound: Amplifier C produces the raw data-plane bytes, Amplifier A produces the connection.bringyour.com request storm, and Amplifier B produces the api.bringyour.com request storm — consistent with the proxy stats in #175 showing ~26M requests to connect and ~3.4M to api during the outage window.