Skip to content

Exactly-once ingest with idempotent checkpointing #101

@Miracle656

Description

@Miracle656

Background

On restart mid-batch the indexer can double-insert or skip events. Ingest should be exactly-once via idempotent writes + durable checkpoints.

What to build

Wrap each batch in a transaction that upserts events by unique eventId and advances a persisted cursor atomically, so a crash/restart resumes exactly where it left off.

Key files

  • src/indexer.ts
  • src/indexer/checkpoint.ts (new)

Acceptance criteria

  • Kill mid-batch → no dupes, no gaps on resume
  • Cursor advances atomically with the batch
  • Test simulates crash between insert and commit

Drips Wave · Complexity: Advanced · 200 points

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions