Skip to content

Conversation

@giortzisg
Copy link
Contributor

Description

Since the addition of Telemetry Buffers moves the serialization to a background worker, user provided attributes that can be mutated should be deep copied, to avoid panics during serialization.

@giortzisg giortzisg requested a review from lcian November 27, 2025 14:50
@giortzisg giortzisg self-assigned this Nov 27, 2025
Comment on lines +780 to +781
if event.User.Data != nil {
eventToBuffer.User.Data = deepCopyMapStringString(event.User.Data)

This comment was marked as outdated.

@codecov
Copy link

codecov bot commented Nov 27, 2025

Codecov Report

❌ Patch coverage is 28.84615% with 74 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.83%. Comparing base (34261f3) to head (647e8aa).

Files with missing lines Patch % Lines
deepcopy.go 24.32% 50 Missing and 6 partials ⚠️
client.go 0.00% 18 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1148      +/-   ##
==========================================
- Coverage   86.85%   85.83%   -1.02%     
==========================================
  Files          62       63       +1     
  Lines        6092     6184      +92     
==========================================
+ Hits         5291     5308      +17     
- Misses        587      656      +69     
- Partials      214      220       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

k := iter.Key()
val := iter.Value().Interface()
newVal := deepCopyValue(val)
newMap.SetMapIndex(k, reflect.ValueOf(newVal))
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Nil values in nested maps/slices cause data loss or panic

When deepCopyValue returns nil for a nil value in a nested structure, reflect.ValueOf(nil) produces an invalid reflect.Value. For maps, calling SetMapIndex with an invalid Value silently deletes the key, causing data loss. For slices and arrays, calling Set with an invalid Value causes a panic. This affects user-provided Extra and Context data containing nested collections with nil values.

Additional Locations (2)

Fix in Cursor Fix in Web

Comment on lines 330 to 341
clone.breadcrumbs = make([]*Breadcrumb, len(scope.breadcrumbs))
copy(clone.breadcrumbs, scope.breadcrumbs)
clone.breadcrumbs = make([]*Breadcrumb, 0, len(scope.breadcrumbs))
for _, b := range scope.breadcrumbs {
clone.breadcrumbs = append(clone.breadcrumbs, deepCopyBreadcrumb(b))
}
clone.attachments = make([]*Attachment, len(scope.attachments))
copy(clone.attachments, scope.attachments)
for key, value := range scope.tags {
clone.tags[key] = value
}
for key, value := range scope.contexts {
clone.contexts[key] = cloneContext(value)
}
for key, value := range scope.extra {
clone.extra[key] = value
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if I'm wrong, but:
this was here already and was not causing problems, and it's a separate code path from the buffer/transport, so I don't think this should be changed at all.
I assume this was deliberately done this way because scope forking happens frequently and so you wouldn't want to do an expensive deep copy.

// a proper deep copy: if some context values are pointer types (e.g. maps),
// they won't be properly copied.
func cloneContext(c Context) Context {
res := make(Context, len(c))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above

if event.User.Data != nil {
eventToBuffer.User.Data = deepCopyMapStringString(event.User.Data)
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nested in Event there are other things we might care about, for example a Span can have Data (and apparently tags and extra), same about Log attributes, etc. Do we care about those?

@aldy505
Copy link
Contributor

aldy505 commented Dec 30, 2025

...or we can serialize it up front, and then send them to telemetry buffer after serialization? I would assume doing deep copy would inflict in more RAM usage than just serialize it up front.

@giortzisg
Copy link
Contributor Author

giortzisg commented Jan 7, 2026

...or we can serialize it up front, and then send them to telemetry buffer after serialization?

I was weighting both approaches, but i think that pre-serialization has more drawbacks:

  • the scheduler needs the whole event, to get the trace context to build the envelope (we would need to pass metadata around, along with the serialized event)
  • batching logic for logs/spans would also require metadata and more complicated
  • performance wise, it's better to serialize on the background for:
    • request latency for the user is low
    • when rate limited we drop immediately, skipping serialization completely

I would assume doing deep copy would inflict in more RAM

That is a valid concern but we only keep the copy around till we flush the event (short lifespan) and other SDKs already copy some event data.


if client.telemetryBuffer != nil {
if !client.telemetryBuffer.Add(event) {
eventToBuffer := *event
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The Event.Tags map is not deep-copied before being passed to the telemetry buffer, creating a risk of a concurrent map access panic if the user mutates it.
Severity: HIGH | Confidence: High

🔍 Detailed Analysis

When an event is captured with telemetry enabled, several of its fields are deep-copied to prevent data races during background serialization. However, the Event.Tags map, which is a user-mutable field, is not being deep-copied. If a user modifies the event.Tags map after calling CaptureEvent(), a concurrent map access panic will occur when the background worker attempts to serialize the event for telemetry. This omission is inconsistent with the handling of other mutable fields like Extra and Contexts.

💡 Suggested Fix

In client.go, before adding the event to the telemetry buffer, perform a deep copy of the event.Tags map, similar to how other mutable fields are handled. For example: if event.Tags != nil { eventToBuffer.Tags = deepCopyMapStringString(event.Tags) }.

🤖 Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.

Location: client.go#L766

Potential issue: When an event is captured with telemetry enabled, several of its fields
are deep-copied to prevent data races during background serialization. However, the
`Event.Tags` map, which is a user-mutable field, is not being deep-copied. If a user
modifies the `event.Tags` map after calling `CaptureEvent()`, a concurrent map access
panic will occur when the background worker attempts to serialize the event for
telemetry. This omission is inconsistent with the handling of other mutable fields like
`Extra` and `Contexts`.

Did we get this right? 👍 / 👎 to inform future reviews.
Reference ID: 8279488

}
if event.User.Data != nil {
eventToBuffer.User.Data = deepCopyMapStringString(event.User.Data)
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tags field missing deep copy in telemetry buffer

High Severity

The Tags field is not deep copied when buffering events for background serialization, while other map[string]string fields like Modules and User.Data are. Since eventToBuffer := *event creates a shallow copy, eventToBuffer.Tags still references the same underlying map as event.Tags. If a user mutates the tags map after the event is queued, this could cause a race condition or panic during serialization in the background worker.

Fix in Cursor Fix in Web

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants