Feature/ledger backfill 419

[Babigdk]

Ledger Backfill Service for Testnet Contract Events (#419)

Closes #419

Overview

This PR implements a comprehensive ledger backfill service that scans historical contract events from the Stellar Testnet and populates analytics and timelines. The implementation follows best practices for idempotency, resumability, and observability.

✨ Key Features

1. Configurable Ledger Backfill

• Start and end ledger sequence number configuration
• Adjustable batch size for processing
• Support for campaign-specific filtering
• Input validation (non-negative, valid ranges)

2. Progress Checkpoint System

• Persists backfill progress in database
• Tracks exact ledger position after each batch
• Enables resume from failure without reprocessing
• Tracks error information and retry attempts
• Support for pausing and resuming jobs

3. Idempotent Event Storage

• Composite unique key on ContractEvent model
• Unique constraint: (contractId, ledgerSequence, transactionHash, eventIndex)
• Prevents duplicate events in database
• Automatic detection and skipping of existing events
• Tracked metrics (processed, skipped, errors)

4. Resilient Job Processing

• BullMQ queue integration with exponential backoff
• Max 3 retry attempts per batch
• Dead Letter Queue (DLQ) support for permanently failed jobs
• Checkpoint updates after each batch for resume capability

📦 Database Changes

New Models

BackfillCheckpoint

• Tracks progress for each backfill job
• Fields: id, jobId, contractId, startLedger, endLedger, lastProcessedLedger
• Status tracking: pending → processing → completed/failed/paused
• Running counters: processed, skipped, errors
• Resume support: resumeCount, maxRetries
• Metadata storage for batch configuration

ContractEvent

• Stores contract events fetched from blockchain
• Composite unique key prevents duplicates
• Event metadata: type, topics, data payload
• Processing tracking: processedAt, processedBy
• Ledger reference: ledgerSequence, transactionHash, eventIndex

🔌 API Endpoints

All endpoints require admin role and API authentication.

POST /v1/admin/ledger/backfill

Trigger a new backfill job

Request:

{
  "contractId": "CCONTRACTID...",
  "startLedger": 1000,
  "endLedger": 2000,
  "campaignId": "campaign_123",
  "batchSize": 100
}

[drips-wave]

@Babigdk Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[Babigdk]

Resolved

[Cedarich]

please fix workflow

[Cedarich]

Please fix lint error, workflow failed again