Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 25 additions & 4 deletions src/agent/protocol/grpc/clients/TraceReportClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,8 @@ export default class TraceReportClient implements Client {
if (this.buffer.length >= config.maxBufferSize) {
logger.warn(
`Trace buffer reached maximum size (${config.maxBufferSize}). ` +
`Discarding oldest segment to prevent memory leak. ` +
`This may indicate network connectivity issues with the collector.`
`Discarding oldest segment to prevent memory leak. ` +
`This may indicate network connectivity issues with the collector.`,
);
this.buffer.shift(); // Remove oldest segment
}
Expand All @@ -66,9 +66,30 @@ export default class TraceReportClient implements Client {
}

private reportFunction(callback?: any) {
emitter.emit('segments-sent'); // reset limiter in SpanContext

try {
// Collector is unreachable. Do not call collect():
// gRPC would push the call onto its internal pickQueue and retain all
// SegmentObjectAdapter objects written via stream.write() until the
// deadline fires (default 10 s). With reportFunction running every 1 s,
// ~10 pending calls accumulate simultaneously, each holding up to
// maxBufferSize segment objects, causing unbounded heap growth → OOM.
//
// Do not emit 'segments-sent' either: withholding it keeps
// SpanContext.nActiveSegments above zero, so SpanContext returns
// DummyContext for new spans — natural backpressure that stops further
// segment production while the collector is down.
//
// isConnected calls getConnectivityState(true), which also triggers
// reconnection automatically when the channel is IDLE, so recovery
// is transparent once the collector comes back.
if (!this.isConnected) {
this.buffer.length = 0;
Copy link

Copilot AI Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clearing this.buffer when the channel isn't READY drops all already-finished segments immediately, even for short transient disconnects / initial CONNECTING state, which defeats the purpose of having a bounded buffer and makes recovery unable to flush the backlog once the collector is back. Consider keeping the buffer intact (it’s already capped by maxBufferSize) and simply skipping collect() + not emitting segments-sent while disconnected; only drop segments when the buffer cap is hit (or after a separate time-based policy).

Suggested change
this.buffer.length = 0;

Copilot uses AI. Check for mistakes.
if (callback) callback();
return;
}

emitter.emit('segments-sent'); // reset limiter in SpanContext

if (this.buffer.length === 0) {
if (callback) callback();

Expand Down
Loading