Skip to content

proxy: half-close upstream connections to avoid noisy log spam#12

Open
Vairelf wants to merge 1 commit into
flant:masterfrom
Vairelf:fix/proxy-half-close
Open

proxy: half-close upstream connections to avoid noisy log spam#12
Vairelf wants to merge 1 commit into
flant:masterfrom
Vairelf:fix/proxy-half-close

Conversation

@Vairelf

@Vairelf Vairelf commented Jun 4, 2026

Copy link
Copy Markdown

Problem

The proxy logs ~1 line per short-lived client connection:

Error writing content: readfrom tcp 127.0.0.1:9999->127.0.0.1:44924: use of closed network connection

This is especially noticeable when Kubernetes exec liveness/readiness probes run redis-cli -p 9999 ping every second — kubelet generates a continuous stream of these errors even though every probe succeeds (PONG).

Root cause

pkg/proxy/proxy.go spawns two goroutines per connection that each io.Copy one direction and then fully close the destination on EOF (defer w.Close()).

redis-cli closes its socket immediately after receiving the reply. The goroutine copying client → upstream sees EOF and closes the upstream socket entirely. The sibling goroutine, still in io.Copy for upstream → client, then trips on a closed socket and surfaces use of closed network connection — even though all data was already delivered.

Fix

Use (*net.TCPConn).CloseWrite() instead of Close() on the copy-done path. This sends a one-way FIN, allowing the sibling goroutine to drain its direction cleanly before the connection is fully torn down. The actual Close() of both sockets is now done unconditionally via defer in proxy(), so resources are still released.

Also suppress the expected net.ErrClosed from the log (e.g. if the listener is shut down).

Verification

Reproduced and verified in the bundled test/docker-compose.yml stack (1 master + 2 replicas + sentinel + proxy):

Branch redis-cli ping Error lines in proxy log
master PONG × 5 5
this PR PONG × 5 0

SET/GET/DEL through the proxy still work; master_resolver failover loop is untouched.

Notes

  • Signature of pipe() widened from io.WriteCloser / io.Reader to net.Conn to access CloseWrite() (fallback to plain Close() для non-TCP net.Conn implementations — e.g. net.Pipe() в тестах).
  • Behaviour for long-lived clients is unchanged.

The pipe goroutines fully closed the opposite connection on EOF,
which raced with the sibling goroutine still reading/writing and
produced "use of closed network connection" errors on every short
client interaction (e.g. kubelet's redis-cli ping liveness probes).

Use TCPConn.CloseWrite to propagate FIN in one direction only, let
the sibling goroutine drain its side cleanly, and suppress the
expected net.ErrClosed from the log.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant