Skip to content

Conversation

@chatton
Copy link
Contributor

@chatton chatton commented Jan 8, 2026

Overview

This PR extracts a small interface with methods to call into engine_forkchoiceUpdatedV3, engine_getPayloadV4 and engine_newPayloadV4.

Creates an implementation with the existing logic, and then a tracing wrapper implementation.

In this case, each of these will be child spans as they use the existing context which will have a parent span.

We will get traces like

Executor.InitChain
    -> Engine.ForkchoiceUpdated
Executor.ExecuteTxs
   -> Engine.GetPayload

@github-actions
Copy link
Contributor

github-actions bot commented Jan 8, 2026

The latest Buf updates on your PR. Results from workflow CI / buf-check (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed⏩ skipped✅ passed✅ passedJan 8, 2026, 2:05 PM

@claude
Copy link
Contributor

claude bot commented Jan 8, 2026

Claude finished @chatton's task in 2m 23s —— View job


Code Review Complete

I've completed a comprehensive review of the tracing implementation for EngineClient. Overall, this is a well-structured PR that cleanly adds OpenTelemetry tracing through the decorator pattern. Below are my findings organized by focus area.


1. Code Quality & Architecture ⭐

Strengths:

  • Excellent use of decorator pattern: The EngineRPCClient interface extraction is clean and non-invasive
  • Proper dependency injection: Conditional tracing via tracingEnabled flag is well-implemented
  • Clear separation of concerns: Core RPC logic in engineRPCClient, tracing logic in tracedEngineRPCClient
  • Idiomatic Go: Code follows Go conventions, uses proper error handling patterns

Issues:

Minor: Inconsistent attribute batching (Low Priority)

File: execution/evm/engine_rpc_tracing.go

Gemini's review correctly identified that attributes could be batched more efficiently. There's an inconsistency in the implementation:

  • Line 47-61 (ForkchoiceUpdated): Uses slice building + single SetAttributes call ✅
  • Line 113-119 (NewPayload): Uses slice building + single SetAttributes call ✅
  • Line 82-88 (GetPayload): Makes a single SetAttributes call with all attributes ✅

However, there's a minor style inconsistency: ForkchoiceUpdated could set both conditional attributes in the same pattern as NewPayload for consistency.

Recommendation: Consider standardizing the pattern across all three methods to use the slice-building approach when there are conditional attributes. This is a nice-to-have, not critical.


2. Error Handling & Resilience ✅

Strengths:

  • Error propagation is correct - spans record errors before returning
  • Status codes properly set on errors: span.SetStatus(codes.Error, err.Error())
  • Errors from inner client are passed through unchanged
  • defer span.End() ensures spans are always closed

No issues found - error handling is solid.


3. Security 🔍

Observations:

Potential sensitive data in traces (Medium Priority)

File: execution/evm/engine_rpc_tracing.go

The tracing implementation captures several pieces of data that could be considered sensitive in certain contexts:

  • Block hashes (lines 33-35, 56-57, 84, 98, 116-117): Generally safe, but reveals blockchain state
  • Transaction counts (lines 86, 100): Safe
  • Gas metrics (lines 87, 101): Safe
  • State roots (line 85): Generally safe

Analysis: In the context of a blockchain node, this data is typically public. However:

  1. Consider if your threat model includes trace data being sent to third-party observability platforms
  2. Ensure trace backends have appropriate access controls
  3. Be mindful of correlation attacks if traces are public

Recommendation: Document in code comments or security documentation what trace data is captured and whether it's considered sensitive in your deployment model. This is informational - the current implementation is reasonable for a blockchain node.


4. Performance & Resource Efficiency ⚡

Concerns:

Minor allocation overhead from conditional attributes (Low Priority)

Files: execution/evm/engine_rpc_tracing.go:47-61, 113-119

The conditional attribute building using append could cause slice reallocations:

attributes := []attribute.KeyValue{
    attribute.String("payload_status", result.PayloadStatus.Status),
}
if result.PayloadID != nil {
    attributes = append(attributes, attribute.String("payload_id", result.PayloadID.String()))
}

Impact: Minimal - these are cold paths (RPC calls), and the allocations are tiny. The code prioritizes readability over micro-optimization, which is appropriate.

Recommendation: No change needed unless profiling reveals this as a hotspot (unlikely). The readability is more valuable.

Context overhead

Each traced call adds span creation/end overhead. Given that these wrap RPC calls (which are expensive), the relative overhead is negligible.

No action needed - performance impact is acceptable.


5. Testing & Reliability ⚠️

Critical Gap:

Missing tests for tracing wrapper (High Priority)

Location: No test file for engine_rpc_tracing.go

The tracing wrapper has zero test coverage. While the underlying RPC client logic is tested elsewhere, the tracing behavior should be verified:

Missing test scenarios:

  1. Spans are created with correct names and attributes
  2. Errors are recorded in spans correctly
  3. Span status is set appropriately on success/failure
  4. Context propagation works correctly
  5. Conditional attributes (nil checks) are handled properly

Recommendation: Add execution/evm/engine_rpc_tracing_test.go with tests like:

func TestTracedEngineRPCClient_ForkchoiceUpdated(t *testing.T) {
    // Use a mock EngineRPCClient and an in-memory span recorder
    // Verify span attributes, status, error recording
}

This is high priority because tracing bugs can cause production observability blind spots.

Add tracing tests →


6. Documentation & Developer Experience 📚

Strengths:

  • Interface and type documentation is clear
  • Function comments explain what each method does
  • Constructor comment documents the tracingEnabled parameter

Issues:

Minor: Missing package-level documentation (Low Priority)

Files: execution/evm/engine_rpc_client.go, execution/evm/engine_rpc_tracing.go

Neither new file has package-level documentation explaining the tracing architecture. Developers unfamiliar with the decorator pattern might not immediately understand the relationship between these files.

Recommendation: Add package comments like:

// Package evm provides an execution client for Ethereum Engine API.
//
// Tracing: The engine RPC client supports optional OpenTelemetry tracing via
// the decorator pattern. When tracing is enabled, withTracingEngineRPCClient
// wraps the base EngineRPCClient to emit spans for all Engine API calls.

Documentation of traced attributes

File: execution/evm/engine_rpc_tracing.go

The trace attributes are hardcoded in the implementation without documentation of what gets traced. Consider adding comments documenting the observability contract:

// ForkchoiceUpdated traces engine_forkchoiceUpdatedV3 calls.
// Span attributes:
//   - method: always "engine_forkchoiceUpdatedV3"
//   - head_block_hash, safe_block_hash, finalized_block_hash: input state
//   - payload_status: response status (VALID/INVALID/SYNCING/ACCEPTED)
//   - payload_id: response payload ID (if present)
//   - latest_valid_hash: response validation hash (if present)
func (t *tracedEngineRPCClient) ForkchoiceUpdated(...)

This helps operators understand what to expect in their traces without reading implementation code.

Recommendation: Add attribute documentation to exported methods (nice-to-have).


Summary

Category Status Critical Issues
Code Quality & Architecture ✅ Excellent 0
Error Handling ✅ Solid 0
Security ⚠️ Consider 0 (informational)
Performance ✅ Good 0
Testing ⚠️ Needs Work 1 - Missing tests
Documentation ⭐ Good 0

Critical Action Items:

  1. Add tests for tracedEngineRPCClient (high priority) - see link above
  2. Consider documenting trace attribute contracts (nice-to-have)
  3. Consider security documentation for trace data (informational)

Overall Assessment: This is a clean, well-architected PR that successfully adds tracing without disrupting existing code. The main gap is test coverage for the tracing wrapper. The implementation follows Go best practices and OpenTelemetry conventions correctly.


@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello @chatton, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the observability of interactions with the Ethereum execution engine by integrating OpenTelemetry tracing into the EngineClient. It achieves this through a clean architectural refactoring, introducing an interface for Engine API calls and a decorator pattern for applying tracing. This change allows for detailed monitoring and debugging of critical engine operations without altering the core RPC logic, providing valuable insights into the lifecycle of forkchoice updates and payload processing.

Highlights

  • Engine API Client Abstraction: Introduced an EngineRPCClient interface to abstract the direct RPC calls to the Ethereum Engine API, specifically for ForkchoiceUpdated, GetPayload, and NewPayload methods.
  • OpenTelemetry Tracing Integration: Implemented a tracedEngineRPCClient that wraps the core EngineRPCClient and adds OpenTelemetry spans to each Engine API call. These spans include relevant attributes like method names, block hashes, and payload statuses, enhancing observability.
  • Conditional Tracing Activation: Refactored the EngineClient to utilize the new EngineRPCClient interface, allowing the tracing wrapper to be conditionally applied during client initialization based on a tracingEnabled flag.
  • Dependency Updates: Added go.opentelemetry.io/otel and go.opentelemetry.io/otel/trace as direct dependencies, along with other related indirect OpenTelemetry modules.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces tracing to the EngineClient by extracting an EngineRPCClient interface and implementing a tracing decorator. The changes are well-structured and effectively add observability to the engine RPC calls. My feedback focuses on a minor optimization in the tracing implementation to improve efficiency and code style by batching attribute setting.

Comment on lines 45 to 51
span.SetAttributes(
attribute.String("payload_status", result.PayloadStatus.Status),
attribute.Bool("has_payload_id", result.PayloadID != nil),
)
if result.PayloadStatus.LatestValidHash != nil {
span.SetAttributes(attribute.String("latest_valid_hash", result.PayloadStatus.LatestValidHash.Hex()))
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The attributes for the span are being set in two separate calls to span.SetAttributes. It would be more efficient and readable to gather all attributes into a slice and make a single call.

Suggested change
span.SetAttributes(
attribute.String("payload_status", result.PayloadStatus.Status),
attribute.Bool("has_payload_id", result.PayloadID != nil),
)
if result.PayloadStatus.LatestValidHash != nil {
span.SetAttributes(attribute.String("latest_valid_hash", result.PayloadStatus.LatestValidHash.Hex()))
}
attrs := []attribute.KeyValue{
attribute.String("payload_status", result.PayloadStatus.Status),
attribute.Bool("has_payload_id", result.PayloadID != nil),
}
if result.PayloadStatus.LatestValidHash != nil {
attrs = append(attrs, attribute.String("latest_valid_hash", result.PayloadStatus.LatestValidHash.Hex()))
}
span.SetAttributes(attrs...)

Comment on lines 103 to 108
span.SetAttributes(
attribute.String("payload_status", result.Status),
)
if result.LatestValidHash != nil {
span.SetAttributes(attribute.String("latest_valid_hash", result.LatestValidHash.Hex()))
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The attributes for the span are being set in two separate calls to span.SetAttributes. It's more efficient to collect all attributes and make a single call. This is similar to a previous comment.

Suggested change
span.SetAttributes(
attribute.String("payload_status", result.Status),
)
if result.LatestValidHash != nil {
span.SetAttributes(attribute.String("latest_valid_hash", result.LatestValidHash.Hex()))
}
attrs := []attribute.KeyValue{
attribute.String("payload_status", result.Status),
}
if result.LatestValidHash != nil {
attrs = append(attrs, attribute.String("latest_valid_hash", result.LatestValidHash.Hex()))
}
span.SetAttributes(attrs...)

@codecov
Copy link

codecov bot commented Jan 8, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.67%. Comparing base (41cac58) to head (931d2ac).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2959      +/-   ##
==========================================
- Coverage   58.81%   58.67%   -0.15%     
==========================================
  Files          93       93              
  Lines        8863     8863              
==========================================
- Hits         5213     5200      -13     
- Misses       3062     3074      +12     
- Partials      588      589       +1     
Flag Coverage Δ
combined 58.67% <ø> (-0.15%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Base automatically changed from cian/add-tracing-part-2 to main January 8, 2026 13:05
@chatton chatton marked this pull request as ready for review January 8, 2026 13:35
@chatton chatton requested a review from tac0turtle January 8, 2026 14:44
@chatton chatton added this pull request to the merge queue Jan 8, 2026
Merged via the queue into main with commit 453a8a4 Jan 8, 2026
43 of 46 checks passed
@chatton chatton deleted the cian/add-tracing-part-3 branch January 8, 2026 15:59
alpe added a commit that referenced this pull request Jan 9, 2026
* main:
  feat(tracing): add tracing to EngineClient (#2959)
  chore: inject W3C headers into engine client and eth client (#2958)
  feat: adding tracing for Executor and added initial configuration (#2957)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants