Skip to content

Conversation

@devin-ai-integration
Copy link
Contributor

@devin-ai-integration devin-ai-integration bot commented Aug 27, 2025

📥 Pull Request

📘 Description
Implements full Haystack 2.x auto-instrumentation for AgentOps to resolve the reported issue with AzureOpenAIChatGenerator not being tracked. This adds:

  • Auto-instrumentation: New HaystackInstrumentor that wraps OpenAIGenerator and AzureOpenAIChatGenerator methods to capture LLM calls, prompts, and responses
  • Azure example: Demonstrates instrumentation with AzureOpenAIChatGenerator including proper validation
  • CI integration: Both Haystack examples added to examples integration test workflow
  • Updated docs: Added supported generators list and links to both examples

The instrumentor follows existing agentic library patterns, registering as "haystack" in AGENTIC_LIBRARIES to automatically activate when haystack-ai 2.x is imported.

🧪 Testing

  • ✅ Pre-commit checks passed locally
  • OpenAI example validated locally: Generated session link showing 2 spans captured with proper token tracking (48 total tokens, $0.0000216 cost)
  • ⚠️ Azure example requires manual testing - depends on AZURE_OPENAI_* environment variables (gracefully skips if missing)
  • ⚠️ CI workflow not yet triggered - examples integration test workflow hasn't run due to repository settings

⚠️ Human Review Checklist

  • Verify instrumentor correctly captures Haystack LLM calls (test with actual haystack-ai package usage)
  • Review content extraction logic in _first_non_empty_text() and _extract_prompt() for Haystack data structure compatibility
  • Test Azure example with real Azure OpenAI credentials
  • Confirm examples run successfully in CI when workflow is enabled
  • Validate span attributes align with AgentOps conventions

Link to Devin run: https://app.devin.ai/sessions/684e8b0399eb48c2b09cf5e9a9b96bf8
Requested by: @areibman

devin-ai-integration bot and others added 5 commits August 27, 2025 18:46
… docs links

Co-Authored-By: Alex <meta.alex.r@gmail.com>
…) and use named argument

Co-Authored-By: Alex <meta.alex.r@gmail.com>
…ce_spans and print summary

Co-Authored-By: Alex <meta.alex.r@gmail.com>
…nd_trace to ensure session URL and proper validation

Co-Authored-By: Alex <meta.alex.r@gmail.com>
…penAIChatGenerator); add Azure example; update CI and docs

Co-Authored-By: Alex <meta.alex.r@gmail.com>
@devin-ai-integration
Copy link
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

devin-ai-integration bot and others added 5 commits August 27, 2025 20:05
… compatibility

Co-Authored-By: Alex <meta.alex.r@gmail.com>
Co-Authored-By: Alex <meta.alex.r@gmail.com>
…g to keep CI green

Co-Authored-By: Alex <meta.alex.r@gmail.com>
…docs, and requirements

Co-Authored-By: Alex <meta.alex.r@gmail.com>
@codecov
Copy link

codecov bot commented Aug 27, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@areibman areibman merged commit 3244a3a into main Aug 27, 2025
36 of 50 checks passed
@areibman areibman deleted the devin/1756320374-haystack-integration branch August 27, 2025 22:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants