Add FedifySpanExporter for distributed trace storage

[dahlia]

Summary

This PR implements FedifySpanExporter, a new OpenTelemetry SpanExporter that persists ActivityPub activity traces to a KvStore. This enables distributed tracing support across multiple nodes in a Fedify deployment, which is essential for building debug dashboards that can show complete request flows across web servers and background workers.

Closes #497

Usage

The exporter captures activity data from OpenTelemetry span events (activitypub.activity.received and activitypub.activity.sent) and stores them with full trace context preserved:

import { MemoryKvStore } from "@fedify/fedify";
import { FedifySpanExporter } from "@fedify/fedify/otel";
import {
  BasicTracerProvider,
  SimpleSpanProcessor,
} from "@opentelemetry/sdk-trace-base";

const kv = new MemoryKvStore();
const exporter = new FedifySpanExporter(kv, {
  ttl: Temporal.Duration.from({ hours: 1 }),
});

const provider = new BasicTracerProvider();
provider.addSpanProcessor(new SimpleSpanProcessor(exporter));

The stored traces can then be queried for display in a debug dashboard:

// Get all activities for a specific trace
const activities = await exporter.getActivitiesByTraceId(traceId);

// Get recent traces with summary information
const recentTraces = await exporter.getRecentTraces({ limit: 100 });

Storage strategies

The exporter supports two storage strategies depending on the KvStore capabilities. When the list() method is available (preferred), it stores individual records with keys like [prefix, traceId, spanId]. When only cas() is available, it uses compare-and-swap operations to append records to arrays stored per trace.

Changelog

For a detailed list of added types and interfaces, see the diff in CHANGES.md.

Test plan

• Unit tests for FedifySpanExporter class (22 test cases)
• Tests pass in Deno environment (deno task test)
• Tests pass in Node.js environment (pnpm test)

[gemini-code-assist]

Summary of Changes

Hello @dahlia, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances Fedify's observability by integrating distributed tracing capabilities. It introduces FedifySpanExporter, an OpenTelemetry SpanExporter that stores ActivityPub activity traces in a KvStore. This allows for a comprehensive view of request flows across different nodes and background workers in a distributed Fedify deployment, which is essential for debugging and monitoring complex interactions.

Highlights

• Distributed Tracing: Introduced FedifySpanExporter to enable distributed tracing for ActivityPub activities across multiple nodes.
• Activity Persistence: Allows persisting ActivityPub activity traces to a KvStore, crucial for aggregating data from various services.
• Debug Dashboard Support: Facilitates the creation of debug dashboards by providing a mechanism to store and query complete request flows.
• KvStore Flexibility: Supports KvStore implementations that offer either list() (preferred) or cas() methods for storage.
• New Module: Added a new @fedify/fedify/otel module containing the exporter and related types.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces FedifySpanExporter, a valuable addition for distributed tracing in Fedify applications. The implementation is well-structured and accompanied by a comprehensive set of unit tests, covering various scenarios including distributed simulations and concurrent writes. The documentation is also clear and thorough.

I have a few suggestions to improve correctness and maintainability:

• Use the event timestamp from OpenTelemetry spans for more accurate trace data.
• Address a potential race condition in the trace summary update logic by using atomic operations.
• Refactor a method to reduce code duplication.
• Minor updates to an error message and documentation for consistency and clarity.

Overall, this is a great contribution that significantly enhances Fedify's observability capabilities.

[codecov]

Codecov Report

❌ Patch coverage is 84.53608% with 45 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
packages/fedify/src/otel/exporter.ts	84.53%	45 Missing ⚠️

Files with missing lines	Coverage Δ
packages/fedify/src/otel/exporter.ts	84.53% <84.53%> (ø)

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

This PR introduces FedifySpanExporter, a new OpenTelemetry SpanExporter implementation that enables distributed tracing across multiple nodes in Fedify deployments. The exporter persists ActivityPub activity traces to a KvStore, allowing debug dashboards to aggregate and display complete request flows across web servers and background workers.

Key changes:

• New @fedify/fedify/otel module with FedifySpanExporter class and supporting interfaces
• Support for two storage strategies: individual record keys (when list() is available) or CAS-based array append (when only cas() is available)
• Query methods for retrieving trace activities and summaries for dashboard consumption

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
packages/fedify/tsdown.config.ts	Adds new otel module to build configuration
packages/fedify/package.json	Exports @fedify/fedify/otel module for npm consumers
packages/fedify/deno.json	Exports ./otel path for Deno consumers
packages/fedify/src/otel/mod.ts	Module entry point that re-exports FedifySpanExporter and related types
packages/fedify/src/otel/exporter.ts	Core implementation of FedifySpanExporter with span processing, storage, and query methods
packages/fedify/src/otel/exporter.test.ts	Comprehensive test suite with 22 test cases covering exporter functionality
docs/manual/opentelemetry.md	Documentation for setting up and using FedifySpanExporter in distributed deployments
CHANGES.md	Changelog entries documenting all new types and interfaces added in version 1.10.0
AGENTS.md	Updates repository structure documentation to include otel module

[dahlia]

Thanks for the thorough review! I've addressed most of the comments:

Status	Issue	Commit
✅	Use event.time for accurate timestamps instead of export time	31edf2a
✅	Fix race condition in #updateTraceSummary with CAS retry logic	d9aea3f
✅	Fix getRecentTraces sorting/limiting - removed early break that could return wrong results	f6e11e8
✅	Fix fragile _summaries key filtering using key structure check	f5b7876
✅	Add error logging in export() and document the new log category	be8654c
✅	Fix misleading forceFlush() comment	4e1e573
✅	Document httpSignaturesKeyId as optional	ccf4dba
✅	Extract #setWithCasRetry helper to reduce code duplication	4bcfdf2
⏭️	Add DenoKvStore/WorkersKvStore to error message	Skipped (runtime-specific)

[dahlia]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces FedifySpanExporter, a new OpenTelemetry SpanExporter for persisting ActivityPub traces to a KvStore, enabling distributed tracing. The implementation is thorough, with comprehensive unit tests and excellent documentation.

I've identified a couple of potential issues in the implementation of FedifySpanExporter:

• A race condition in the cas retry logic that could lead to data loss.
• The getRecentTraces method silently fails on KvStore implementations without list() support.
• A minor logic issue in getActivitiesByTraceId that could be confusing.

My review includes suggestions to improve the robustness and correctness of these parts. Overall, this is a great addition to Fedify.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.

[dahlia]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces FedifySpanExporter, a significant new feature for distributed tracing by persisting OpenTelemetry spans to a KvStore. The implementation is robust, with support for different KvStore capabilities (list() vs. cas()) and a comprehensive suite of unit tests covering various scenarios, including distributed simulations and concurrency. The documentation is also updated thoroughly to reflect these new additions. My feedback includes a couple of suggestions for improving robustness and developer experience.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

[dahlia]

Addressed the following review comments:

Commit	Change	Review comments
56b46d4	Treat empty httpSignaturesKeyId as undefined	#1, #2
7136157	Document list() requirement for getRecentTraces()	#1, #2, #3
f921cca	Fix misleading test description for shutdown()	#1

[2chanhaeng]

Awesome PR!
How about separating the package as just @fedify/otel?

[dahlia]

How about separating the package as just @fedify/otel?

@2chanhaeng Separating into @fedify/otel would follow the pattern of other integration packages like @fedify/express or @fedify/redis. However, the current implementation is tightly coupled with Fedify's internal types (TraceActivityEntry, federation context). I'll consider this for a future refactoring. For now, keeping it in the main package simplifies the initial release and allows us to iterate on the API before committing to a separate package boundary. What do you think?

Thank you all for the reviews! I've addressed the applicable feedback and responded to each comment.