[BUG] Massive Bandwidth Leak and Resource Exhaustion during Server-Side Outage

[full-bars]

Bug Report: Severe Bandwidth & Resource Leak During Server-Side Outages

Summary

The URnetwork provider software (both Docker and standalone binary) exhibits a severe "bandwidth leak" bug during server-side outages. When the central infrastructure is unreachable or authentication fails, the provider enters an uncontrolled high-frequency retry loop.

This loop consumes massive amounts of raw bandwidth and CPU while producing zero credited transfer or productive work.

---

Observed Behavior

• Productive Traffic vs. Burned Bandwidth: Between 00:00 AM and 05:00 AM, the official transfer stats show 0 bytes of change, confirming the network was non-functional. However, vnstat recorded approximately 125 GiB of raw network traffic during that same window.
• High CPU Usage: CPU usage remains pegged at 100–150%, with host load averages exceeding 14.0.
• Log Spam: Flooding with [t]auth error messages (No successful strategy found / Timeout) at a high frequency.
• Critical Resource Leaks: Resident memory (RES) grew from 1.1 GiB to 1.3 GiB in 20 minutes. The process accumulated nearly 5,000 file descriptors, indicating that failed authentication attempts are not being properly cleaned up.

---

Environment

Field	Value
OS	Ubuntu 24.04.4 LTS (aarch64)
Binary Version	2026.3.23+895075980
Docker Image	bringyour/community-provider:g4-latest

---

Supporting Evidence

1. Proxy Host Request Stats (Last 24 Hours)

The provider-side stats for the last 24 hours show a staggering volume of requests. The software hit the central infrastructure nearly 30 million times during this period.

Host	Requests	Usage	Avg. bytes per request
connect.bringyour.com	25,810,722	346.24 GB	14,403.88 B
api.bringyour.com	3,366,462	45.55 GB	14,527.95 B
Total	29,177,184	391.79 GB

---

2. Transfer Stats History

Showing 0 change in unpaid data since ~12:17 AM, confirming zero productive work during the outage window.

Timestamp	Unpaid (GB)	Change (GB)
04/23/2026 05:00:49 AM	560.499	0.000
04/23/2026 04:00:08 AM	560.499	0.000
04/23/2026 03:00:12 AM	560.499	0.000
04/23/2026 02:00:16 AM	560.499	0.000
04/23/2026 01:00:11 AM	560.499	0.000
04/23/2026 12:17:01 AM	560.499	0.000
04/23/2026 12:00:50 AM	560.499	1.857

---

3. vnstat Hourly Data

High GiB consumption during the same period as the outage:

enp0s6  /  hourly (2026-04-23)
        hour        rx      |     tx      |    total    |   avg. rate
    ------------------------+-------------+-------------+---------------
        00:00     10.43 GiB |   15.51 GiB |   25.94 GiB |   61.90 Mbit/s
        01:00      9.19 GiB |   15.20 GiB |   24.39 GiB |   58.19 Mbit/s
        02:00      9.42 GiB |   16.53 GiB |   25.95 GiB |   61.92 Mbit/s
        03:00      6.35 GiB |   10.75 GiB |   17.09 GiB |   40.78 Mbit/s
        04:00      6.22 GiB |   11.05 GiB |   17.26 GiB |   41.19 Mbit/s
        05:00      6.48 GiB |    7.99 GiB |   14.46 GiB |  138.05 Mbit/s
    ------------------------+-------------+-------------+---------------
    Total burned during outage: ~125 GiB in 5 hours on this single server.

---

4. Process Status (top)

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
1612252 ubuntu  20   0 2836788   1.1g  17552 S 116.7   4.9   20,22 urnetwork

---

5. Log Snippet

I0423 12:12:37.038664 40 transport.go:459] [t]auth error = No successful strategy found.
I0423 05:11:28.987004 1612252 transport.go:485] [t]auth error = Timeout.

---

6. Socket State & Resource Leak Snapshot

Analysis of the ~4,600–5,000 connections shows the process is stuck in a massive, high-concurrency retry loop:

Metric	Value	Description
SYN-SENT	~1,600	Aggressively opening new connections simultaneously without rate-limiting
ESTABLISHED	~3,000	Technically open but non-functional/idle connections that are never reaped
FD Count	~4,950	Open file descriptors accumulated from uncleaned failed auth attempts
Memory Growth	+200 MB in 20 min	RES grew from 1.1 GiB to 1.3 GiB with zero productive traffic

---

Reproduction Steps

1. Run a URnetwork provider instance.
2. Block access to URnetwork central authentication servers, or wait for a server-side outage.
3. Observe vnstat, resident memory, FD counts, and CPU usage.

---

Suggested Fix

• Exponential Backoff: Implement a mandatory increasing delay between auth retries.
• Circuit Breaker: Suspend network activity if No successful strategy found persists beyond a threshold.
• Log Throttling: Rate-limit duplicate error messages to prevent log flooding.
• Connection Management: Ensure all sockets are explicitly closed on authentication failure, and implement a reaper for idle/zombie ESTABLISHED connections.

[full-bars]

Follow-up: Fleet-wide impact data from the May 17, 2026 outage

Collecting vnstat data across all provider nodes running during the outage confirms the scale of the problem.

Outage window: May 17, ~18:00-22:00 UTC (simultaneous across all nodes)

Observed pattern across nodes:

Node scale	Peak rate	Outage window burn
Large (high proxy count)	120-190 GiB/hr	430-490 GiB each
Medium-large	70-80 GiB/hr	~230 GiB
Medium	10-25 GiB/hr	50-145 GiB each
Small	4-10 GiB/hr	10-55 GiB each
Minimal (few proxies)	MiB-level	negligible

Total observed: approximately 1.9 TiB burned across the fleet in ~4 hours, with zero productive transfer credited during that window. Cross-referencing against proxy host usage logs accounts for ~2.4 TiB total for the same period — consistent with nodes not captured in vnstat snapshots making up the remainder.

Characteristic signature: rx and tx are nearly perfectly symmetric on all affected nodes, consistent with the resend amplification loop described in issue #181 rather than legitimate user traffic.

For context on real-world impact: a single 4-hour outage consumed an entire month's worth of proxy bandwidth allowance.

[full-bars]

Update — additional investigation and partial fixes

After further investigation on a production fork, I've identified three independent amplifiers that combine to produce the observed leak. Sharing findings here for upstream consideration.

Three amplifiers

Amplifier C — resend queue × race client fan-out (data plane)
As described in #181, packets in the resend queue are retransmitted up to ~600 times before AckTimeout, multiplied by the number of race clients. This is the primary source of raw bytes on the wire.

Amplifier B — contract creation storm (control plane)
Every active SendSequence calls CreateContract every 5 seconds regardless of backend state. Each call launches a 15-second HttpSerial goroutine against the dead API. With 100+ concurrent flows this sustains thousands of concurrent goroutines and a large volume of outbound TCP connections — consistent with the ~5,000 file descriptor count observed in the original report.

Amplifier A — transport reconnect storm (control plane)
runH1 and runH3 reuse a Reconnect timer that has already expired after the first failed attempt, causing the reconnect loop to fire at near-zero delay. Each iteration makes a full parallelEval call trying all dialers. This accounts for the 25M+ requests to connect.bringyour.com observed in the proxy stats above.

---

What's been addressed on my fork

Amplifier C — capped with MaxResendCount = 16 and MultiRaceClientCount = 2, plus exponential backoff on resend timing when isBackendDegraded(). This alone reduces the data-plane leak by ~200×.

Amplifier A — exponential backoff on auth error reconnect: 5s → 10s → 20s → 40s → 60s cap per consecutive failure. Resets on successful reconnect.

Amplifier B — when isBackendDegraded(), the CreateContract call is skipped entirely (no goroutine launched). Retry poll interval raised from 5s to 30s while degraded.

Degraded detection — added a non-rate-limited lastBackendFailNano atomic updated on every failure (auth or OOB), so isBackendDegraded() holds steady during an outage instead of flickering at log-suppression window boundaries.

---

Remaining gaps

• NSSF conditions are not reliably reproducible in synthetic tests (netem, iptables, DNS poisoning all produce different behavior than a real API outage). Real-world validation of the complete fix is still pending.
• Connection reaping: open file descriptors from failed auth attempts are not explicitly cleaned up. The exponential backoff reduces the rate of new FD creation but doesn't reap existing ones.
• [r]drop log spam from the receive path is unaddressed.

Will update this issue once a real outage validates the combined fix.