[Infrastructure] Integrate OpenTelemetry (OTel) Tracing for the Safety Pipeline

[Vishisht16]

🎯 Objective

Instrument the 3-stage cascade pipeline with OpenTelemetry spans so enterprise users can monitor HumaneProxy in Datadog, Grafana, or Jaeger.

💡 Why this matters

When an AI agent fails or is slow, developers need to know why. If HumaneProxy adds latency because Stage 3 (Groq API) is slow, it needs to be visible in the company's distributed tracing dashboard. OpenTelemetry is the industry standard for this.

✅ Acceptance Criteria

• Add opentelemetry-api and opentelemetry-sdk as optional dependencies (pip install humane-proxy[telemetry]).
• Wrap proxy.check_async() and the individual stages (Heuristics, Embeddings, Reasoning LLM) in OpenTelemetry spans.
• Add span attributes (e.g., humane_proxy.session_id, humane_proxy.final_score, humane_proxy.stage_reached).
• Ensure this feature is strictly opt-in via config.yaml (telemetry.enabled: true). If disabled, there should be zero performance overhead.

📚 Resources

• OpenTelemetry Python Documentation

[ariktadas144]

Hi @Vishisht16 , I would love to work on this as part of GSSoC'26.

I have been reviewing the codebase and the tracing requirements, and I would like to implement the OpenTelemetry instrumentation with opt-in configuration support and tests.

Could you please assign this to me?

[Vishisht16]

Hello @ariktadas144, can you walk me through how you'll do it?

[lakshitadidwania2106]

Hi @Vishisht16 ! im a GSSOC'26 contributor specialized in web development and agents. I’d like to work on this feature.

1> I will Integrate OpenTelemetry (OTel) Tracing for the Safety Pipeline

-> Add OpenTelemetry instrumentation to the 3-stage safety pipeline.
->Make telemetry fully optional via config.yaml (telemetry.enabled).
->Wrap proxy.check_async() and all stages (Heuristics, Embeddings, LLM) with spans.
->Add meaningful span attributes like session_id, final_score, and stage_reached.
->Add optional dependency support using humane-proxy[telemetry].

->Validate traces using tools like Jaeger or Grafana Tempo.

i would love it if u can assign this to me so that I can start working on it asap, thank you :)

[Vishisht16]

@lakshitadidwania2106 let's give the first commenter a chance to explain their approach. It'll be better to create a new issue after testing the repo to fix bugs or request a feature. That would be assigned to you.

[ariktadas144]

Hi @Vishisht16, this is my implementation plan for OpenTelemetry tracing in the 3-stage Safety Pipeline.

I’ll implement observability at two levels to ensure full end-to-end traceability:

1. Request entry layer

• Instrument /chat (FastAPI interceptor) and proxy.check_async() with a root span (pipeline.request)
• Capture request-scoped metadata such as session_id, request_id and latency for full request lifecycle visibility

2. Pipeline-level tracing

• Wrap pipeline.classify with a parent span (pipeline.classify)
• Add structured child spans for each stage:
  • stage1.heuristics
  • stage2.embeddings
  • stage3.reasoning_llm
• Each span will include consistent attributes: session_id, category, final_score, stage_reached and triggers_count
• Add stage-level timing to directly surface bottlenecks (especially Stage 3 latency / external API calls)

3. Production-safe design

• Fully opt-in via telemetry.enabled: true in config.yaml
• Strict no-op execution path when disabled to guarantee zero runtime overhead
• Isolated under optional dependency group: humane-proxy[telemetry]

4. Export & validation

• OTLP exporter configured for compatibility with Jaeger / Grafana Tempo / Datadog
• I’ll validate:
  • correct parent-child span hierarchy across all stages
  • attribute propagation consistency across pipeline boundaries
  • safe toggle behavior (telemetry on/off does not affect classification logic or latency)

Once implemented, this will provide full request-to-decision observability for debugging safety decisions, latency analysis, and production monitoring.

If this approach looks good, I’ll proceed with a clean PR implementing it with proper isolation and tests.

[Vishisht16]

@ariktadas144 this mostly looks good.
A few things to keep in mind before you start:

1. Use OTel's NoOpTracer when telemetry is disabled instead of if enabled checks.
2. Make sure you only include session_id, score, category, stage_reached, triggers_count and message_hash. We default to not storing message text privacy.store_message_text: false so that must also be respected.
3. pipeline.py also has a classify_sync() where we'll need spans, which you missed.
4. Add a telemetry block to config.yaml defaulting to false and a HUMANE_PROXY_TELEMETRY_ENABLED env override in config.py.
5. Attach images of how you validated with Jaeger, Grafana Tempo or Datadog but add CI tests using InMemorySpanExporter.
6. Also, please add a short section in README.md explaining how users can enable and use this feature.

NOTE: Our CONTRIBUTING.md forbids updating pyproject.toml without approval. Please read CONTRIBUTING.md thoroughly before working on any open-source repo. I'm explicitly allowing it this time only to add [telemetry] optional dependency and include it in all extra. However, please include an explicit note for such changes beforehand, just so I have an idea of what changes I can expect.

I hope this much is clear. If you have any questions, please let me know before working on the issue.

[ariktadas144]

Hi @Vishisht16, thanks for the detailed guidance. I have aligned the implementation plan with all constraints.

I’ll implement OpenTelemetry tracing across the safety pipeline with strict opt-in behavior and zero-overhead mode when disabled.

Core approach:

• Use NoOpTracerProvider when telemetry is disabled (no hot-path conditionals)
• Initialize telemetry only inside _lifespan via setup_telemetry(config)
• Instrument both classify() and classify_sync() under root span: pipeline.classify

Span hierarchy:
http.request → pipeline.classify → stage1/stage2/stage3

Allowed span attributes only:
session_id, score, category, stage_reached, triggers_count, message_hash
(fully respecting privacy.store_message_text: false)

Config & setup:

• Add telemetry block in config.yaml (disabled by default)
• Add HUMANE_PROXY_TELEMETRY_ENABLED env override
• Add optional [telemetry] dependency group (only as approved) without touching any other unrelated packaging config in pyproject.toml
• Ensure OTLP exporter degrades gracefully if backend is unavailable

Testing:

• CI with InMemorySpanExporter:
  • enabled vs disabled telemetry
  • sync vs async parity
  • span hierarchy and attribute validation
• Fully deterministic tests (no timing-based assertions)

Documentation:

• README section for enabling telemetry + OTLP setup
• Example Jaeger/Tempo trace structure
• Attach validation screenshots in PR

I'll also ensure:

• No duplicate instrumentation
• Correct async propagation (Stage 3 included)
• Consistent naming convention: humane_proxy.*
• Strict scope isolation

If this looks good, please assign this to me and I’ll start implementation immediately.

[Vishisht16]

@ariktadas144
This looks much better now. You can start working on it.
Also, raise a PR only after 15th of May if you GSSoC points are a concern for you as contributions aren't being tracked currently.

[ariktadas144]

Hi @Vishisht16, noted, thanks for the clarification.

I’ll proceed with the implementation and ensure everything is ready for a clean PR after 15th May.