feat(backend): add data retention polling with pdp subgraph integration by silent-cipher · Pull Request #286 · FilOzone/dealbot

silent-cipher · 2026-02-17T09:47:22Z

Summary

This PR adds data retention monitoring capabilities to the dealbot backend by integrating with the PDP (Proof of Data Possession) subgraph. It introduces a new job that polls provider data retention statistics every hour (default) and exposes them as Prometheus metrics.

Changes

PDP Subgraph Integration: New PDPSubgraphService to query provider proof-set data from subgraph
Data Retention Service: DataRetentionService that calculates estimated faulted and successful proving periods per provider
Prometheus Metrics: New dataSetChallengeStatus counter metric with labels checkType/providerId/providerStatus/value, where value is success or fault.
Scheduled Job: New data.retention.poll job queue integrated with pg-boss scheduler

silent-cipher · 2026-02-18T17:43:28Z

PR is open for review. However, it shouldn’t be merged until subgraph data is available (FilOzone/pdp-explorer#86).

Copilot

Pull request overview

Adds backend support for polling PDP (Proof of Data Possession) subgraph data on a schedule to derive per-provider data-retention statistics and emit them as Prometheus metrics, integrated into the existing pg-boss job system and configuration/docs.

Changes:

Introduces PDPSubgraphService (+ module, query, response validation) and associated tests.
Adds DataRetentionService (+ module, tests) and wires a new data.retention.poll pg-boss job/schedule into JobsService.
Extends configuration and docs with PDP_SUBGRAPH_ENDPOINT and DATA_RETENTION_POLL_INTERVAL_SECONDS, plus a new Prometheus counter registration.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
docs/environment-variables.md	Documents new env vars and scheduling option for data retention polling.
apps/backend/src/wallet-sdk/wallet-sdk.service.ts	Adds `getBlockNumber()` helper used by data retention polling.
apps/backend/src/wallet-sdk/wallet-sdk.service.spec.ts	Updates test config to include new blockchain config field.
apps/backend/src/pdp-subgraph/types.ts	Adds Joi-based validation/transforms for subgraph response types.
apps/backend/src/pdp-subgraph/types.spec.ts	Adds unit tests for subgraph response validation.
apps/backend/src/pdp-subgraph/queries.ts	Adds GraphQL query for providers and proof sets.
apps/backend/src/pdp-subgraph/pdp-subgraph.service.ts	Implements subgraph fetch with batching, rate limiting, retries, validation.
apps/backend/src/pdp-subgraph/pdp-subgraph.service.spec.ts	Adds tests for subgraph service fetch/retry/validation behavior.
apps/backend/src/pdp-subgraph/pdp-subgraph.module.ts	Exposes `PDPSubgraphService` via Nest module.
apps/backend/src/metrics-prometheus/metrics-prometheus.module.ts	Registers a new counter for data retention / dataset challenge status.
apps/backend/src/jobs/repositories/job-schedule.repository.ts	Adds queue-name mapping for `data_retention_poll` in pg-boss job state queries.
apps/backend/src/jobs/jobs.service.ts	Wires new `data.retention.poll` worker + schedule row + queue mapping + metrics tracking.
apps/backend/src/jobs/jobs.service.spec.ts	Updates job service tests for new dependency and new worker/schedule expectations.
apps/backend/src/jobs/jobs.module.ts	Imports `DataRetentionModule` so the jobs worker can execute the poller.
apps/backend/src/jobs/job-queues.ts	Adds `DATA_RETENTION_POLL_QUEUE` constant.
apps/backend/src/database/entities/job-schedule-state.entity.ts	Extends `JobType` union with `data_retention_poll`.
apps/backend/src/data-retention/data-retention.service.ts	Implements polling logic, delta computation, and Prometheus counter increments.
apps/backend/src/data-retention/data-retention.service.spec.ts	Adds tests for polling behavior, batching, deltas, and edge cases.
apps/backend/src/data-retention/data-retention.module.ts	Exposes `DataRetentionService` via Nest module.
apps/backend/src/config/app.config.ts	Adds env validation + config fields for PDP subgraph and poll interval.
apps/backend/README.md	Documents `PDP_SUBGRAPH_ENDPOINT` and `DATA_RETENTION_POLL_INTERVAL_SECONDS`.
apps/backend/.env.example	Adds example values for new env vars.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

apps/backend/src/config/app.config.ts

apps/backend/src/metrics-prometheus/metrics-prometheus.module.ts

apps/backend/src/data-retention/data-retention.service.ts

apps/backend/src/pdp-subgraph/types.ts

apps/backend/src/data-retention/data-retention.service.spec.ts

apps/backend/README.md

apps/backend/src/data-retention/data-retention.service.spec.ts

SgtPooki

a few comments.

We should add a few tests too:

cover how we handle subgraph lag vs rpc block height
test large deltas and assert we are handling them safely.

apps/backend/src/data-retention/data-retention.service.ts

SgtPooki · 2026-02-20T14:38:16Z

apps/backend/src/data-retention/data-retention.service.ts

+        `Negative delta detected for provider ${address} (faulted: ${faultedDelta}, success: ${successDelta}); skipping counter update`,
+      );
+      return;
+    }


When is this possible? are there re-orgs or subgraph corrections? do we need to reset baseline so that metrics aren't stalled when numbers dip below baseline?

Yes, there can be subgraph corrections. Baseline is reset to current values in 710d49e

SgtPooki · 2026-02-20T14:39:53Z

apps/backend/src/wallet-sdk/wallet-sdk.service.ts

+   * Get the current block number from the RPC provider
+   */
+  async getBlockNumber(): Promise<number> {
+    return await this.rpcProvider.getBlockNumber();


this block number will be different than the subgraph's indexed block-number.

replaced it with subgraph's indexed block height - e761858

SgtPooki · 2026-02-20T14:43:07Z

apps/backend/src/data-retention/data-retention.service.ts

+    const estimatedOverduePeriods = proofSets.reduce((acc, proofSet) => {
+      if (proofSet.maxProvingPeriod === 0n) {
+        return acc;
+      }
+      return acc + (blockNumberBigInt - (proofSet.nextDeadline + 1n)) / proofSet.maxProvingPeriod;
+    }, 0n);


the subgraph could be X blocks behind RPC chain head. we should compare overdue period based on the subgraph's own indexed block height.

Now, I'm using subgraph's indexed block height instead of rpc chain head - e761858

apps/backend/src/data-retention/data-retention.service.ts

…tests

Copilot

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

docs/environment-variables.md

Copilot · 2026-02-23T12:18:30Z

apps/backend/src/data-retention/data-retention.service.ts

+  private readonly providerCumulativeTotals: Map<
+    string,
+    {
+      faultedPeriods: bigint;
+      successPeriods: bigint;
+    }
+  >;
+
+  constructor(
+    private readonly configService: ConfigService<IConfig, true>,
+    private readonly walletSdkService: WalletSdkService,
+    private readonly pdpSubgraphService: PDPSubgraphService,
+    @InjectMetric("dataSetChallengeStatus")
+    private readonly dataSetChallengeStatusCounter: Counter,
+  ) {
+    this.providerCumulativeTotals = new Map();
+  }


The providerCumulativeTotals Map grows indefinitely as provider addresses are added but never removed. If providers are dynamically added/removed from the testing provider list (e.g., providers being approved/unapproved, or configuration changes), stale entries will accumulate in memory over time.

Consider implementing cleanup logic to periodically remove entries for providers that are no longer in the active testing provider list. For example, at the start of pollDataRetention(), you could:

Get the current set of provider addresses

Remove any entries from providerCumulativeTotals whose addresses are not in the current set

This would prevent unbounded memory growth while maintaining correct baseline tracking for active providers.

added stale provider clean up in aa0eff3

apps/backend/src/data-retention/data-retention.service.ts

apps/backend/src/pdp-subgraph/pdp-subgraph.service.ts

silent-cipher added 2 commits February 17, 2026 14:28

feat(backend): add data retention polling with pdp subgraph integration

31d6404

test(backend): add tests for data retention and pdp subgraph services

003f330

FilOzzy added this to FOC Feb 17, 2026

github-project-automation bot moved this to 📌 Triage in FOC Feb 17, 2026

Merge branch 'main' into feat/backend/data-retention-metrics

ae840b6

silent-cipher mentioned this pull request Feb 17, 2026

Data retention fault metrics via subgraph #222

Open

silent-cipher self-assigned this Feb 17, 2026

rjan90 moved this from 📌 Triage to ⌨️ In Progress in FOC Feb 18, 2026

rjan90 added this to the M4.1: mainnet ready milestone Feb 18, 2026

silent-cipher added 3 commits February 18, 2026 22:59

refactor(backend): update data retention batch processing

17c2381

Merge branch 'main' into feat/backend/data-retention-metrics

00d2e66

chore: format

2025424

silent-cipher marked this pull request as ready for review February 18, 2026 17:43

Copilot AI review requested due to automatic review settings February 18, 2026 17:43

Copilot started reviewing on behalf of silent-cipher February 18, 2026 17:44 View session

Copilot AI reviewed Feb 18, 2026

View reviewed changes

silent-cipher added 2 commits February 18, 2026 23:38

chore: address copilot comments

7d5bea9

chore: expect addresses in lowercase from subgraph

b59bdd7

silent-cipher requested a review from SgtPooki February 18, 2026 18:21

SgtPooki requested changes Feb 20, 2026

View reviewed changes

silent-cipher added 5 commits February 23, 2026 12:43

refactor(backend): use subgraph meta endpoint for block number

e761858

test(backend): add fake timers to pdp-subgraph service tests

2a78f29

Merge branch 'main' into feat/backend/data-retention-metrics

d80352a

refactor(backend): increment counter in chunks for large deltas

710d49e

test(backend): add catch handlers to prevent unhandled rejections in …

559a511

…tests

silent-cipher requested review from SgtPooki and Copilot February 23, 2026 12:08

Copilot started reviewing on behalf of silent-cipher February 23, 2026 12:09 View session

Copilot AI reviewed Feb 23, 2026

View reviewed changes

silent-cipher added 3 commits February 23, 2026 18:30

chore: address copilot review comments

53b29dd

feat(backend): add stale provider cleanup

aa0eff3

chore: update docs

aa6213a

Comments

Conversation

silent-cipher commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Uh oh!

silent-cipher commented Feb 18, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SgtPooki left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SgtPooki Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

silent-cipher Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

SgtPooki Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

silent-cipher Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

SgtPooki Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

silent-cipher Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

silent-cipher Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

silent-cipher commented Feb 17, 2026 •

edited

Loading