proxy: half-close upstream connections to avoid noisy log spam#12
Open
Vairelf wants to merge 1 commit into
Open
proxy: half-close upstream connections to avoid noisy log spam#12Vairelf wants to merge 1 commit into
Vairelf wants to merge 1 commit into
Conversation
The pipe goroutines fully closed the opposite connection on EOF, which raced with the sibling goroutine still reading/writing and produced "use of closed network connection" errors on every short client interaction (e.g. kubelet's redis-cli ping liveness probes). Use TCPConn.CloseWrite to propagate FIN in one direction only, let the sibling goroutine drain its side cleanly, and suppress the expected net.ErrClosed from the log.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The proxy logs ~1 line per short-lived client connection:
This is especially noticeable when Kubernetes
execliveness/readiness probes runredis-cli -p 9999 pingevery second — kubelet generates a continuous stream of these errors even though every probe succeeds (PONG).Root cause
pkg/proxy/proxy.gospawns two goroutines per connection that eachio.Copyone direction and then fully close the destination on EOF (defer w.Close()).redis-clicloses its socket immediately after receiving the reply. The goroutine copying client → upstream sees EOF and closes the upstream socket entirely. The sibling goroutine, still inio.Copyfor upstream → client, then trips on a closed socket and surfacesuse of closed network connection— even though all data was already delivered.Fix
Use
(*net.TCPConn).CloseWrite()instead ofClose()on the copy-done path. This sends a one-way FIN, allowing the sibling goroutine to drain its direction cleanly before the connection is fully torn down. The actualClose()of both sockets is now done unconditionally viadeferinproxy(), so resources are still released.Also suppress the expected
net.ErrClosedfrom the log (e.g. if the listener is shut down).Verification
Reproduced and verified in the bundled
test/docker-compose.ymlstack (1 master + 2 replicas + sentinel + proxy):redis-cli pingmasterSET/GET/DEL through the proxy still work;
master_resolverfailover loop is untouched.Notes
pipe()widened fromio.WriteCloser/io.Readertonet.Connto accessCloseWrite()(fallback to plainClose()для non-TCPnet.Connimplementations — e.g.net.Pipe()в тестах).