perf(webapp,run-store): point-lookup batch idempotency keys by ericallam · Pull Request #4255 · triggerdotdev/trigger.dev

ericallam · 2026-07-14T10:59:30Z

Summary

Batch triggers that use per-item idempotency keys could take seconds instead of milliseconds when the target task had a large run history. This keeps the idempotency lookup fast regardless of how many runs a task has accumulated.

Root cause

The batch path checks which items already have runs by looking up their idempotency keys with a single WHERE runtimeEnvironmentId = ? AND taskIdentifier = ? AND idempotencyKey IN (...) query. On a very large TaskRun table Postgres underestimates the row count of a specific (environment, task) pair, so once the IN list grows past a handful of keys it stops doing per-key index probes and instead scans every run for that (environment, task) and filters the keys in memory. The cost is then flat and large regardless of how many keys are being checked, and a routine ANALYZE does not correct the estimate at that table size.

Fix

Look each idempotency key up on its own, batched into a UNION ALL of point lookups (chunked, run with bounded concurrency). Each branch is an equality on all three columns of the unique index, so the planner can only do a per-key index probe and can never fall back to the range scan. Same results, same columns, confined to the batch trigger path.

Batch triggers that pass per-item idempotency keys could take seconds when the target task had a large run history, because the bulk `idempotencyKey IN (...)` lookup degraded to scanning every run for the (environment, task) pair and filtering in memory. Look each key up individually via a chunked UNION ALL of point lookups so the planner always uses a per-key index probe.

changeset-bot · 2026-07-14T10:59:35Z

⚠️ No Changeset found

Latest commit: 4d8d5ab

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

coderabbitai · 2026-07-14T10:59:58Z

Walkthrough

Adds a typed run-store API for idempotency-key point lookups, with PostgreSQL and routed-store implementations. Batch trigger preparation now deduplicates keys, chunks queries, and performs bounded concurrent lookups before flattening cached-run matches. A change note documents the batch idempotency lookup performance fix.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The description explains the change, but it does not follow the required template sections.	Add the required Closes #, checklist, Testing, Changelog, and Screenshots sections, and fill in the testing steps and changelog summary.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the main change: faster point-lookup of batch idempotency keys.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch perf/batch-idempotency-point-lookups

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

🧹 Nitpick comments (1)

internal-packages/run-store/src/runOpsStore.ts (1)
471-478: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Execute sub-store queries concurrently.

Queries to the #new and #legacy sub-stores are currently awaited sequentially, which adds unnecessary latency to the caching lookup. Use Promise.all to execute them in parallel, which aligns with the fan-out pattern used in other RoutingRunStore methods (e.g., #findRunsOpen, findSnapshotCompletedWaitpointIds).
⚡ Proposed fix to parallelize the lookups
-    const newRows = await this.#new.findRunsByIdempotencyKeys(
-      args,
-      RoutingRunStore.#ownPrimary(this.#new, client)
-    );
-    const legacyRows = await this.#legacy.findRunsByIdempotencyKeys(
-      args,
-      RoutingRunStore.#ownPrimary(this.#legacy, client)
-    );
+    const [newRows, legacyRows] = await Promise.all([
+      this.#new.findRunsByIdempotencyKeys(
+        args,
+        RoutingRunStore.#ownPrimary(this.#new, client)
+      ),
+      this.#legacy.findRunsByIdempotencyKeys(
+        args,
+        RoutingRunStore.#ownPrimary(this.#legacy, client)
+      ),
+    ]);

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: e985e51a-9343-4959-994e-204f6b097c07

📥 Commits

Reviewing files that changed from the base of the PR and between a1ca646 and 4d8d5ab.

📒 Files selected for processing (5)

.server-changes/fix-batch-idempotency-point-lookups.md
apps/webapp/app/v3/services/batchTriggerV3.server.ts
internal-packages/run-store/src/PostgresRunStore.ts
internal-packages/run-store/src/runOpsStore.ts
internal-packages/run-store/src/types.ts

📜 Review details

⏰ Context from checks skipped due to timeout. (17)

GitHub Check: webapp / 🧪 Unit Tests: Webapp (11, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (12, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (7, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (6, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (8, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (5, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (10, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (3, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (2, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (9, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (1, 12)
GitHub Check: webapp / 🧪 Unit Tests: Webapp (4, 12)
GitHub Check: typecheck / typecheck
GitHub Check: e2e-webapp / 🧪 E2E Tests: Webapp
GitHub Check: internal / 🧪 Unit Tests: Internal
GitHub Check: code-quality / code-quality
GitHub Check: Analyze (javascript-typescript)

🧰 Additional context used

📓 Path-based instructions (12)

**/*.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

**/*.{ts,tsx}: Use types over interfaces for TypeScript
Avoid using enums; prefer string unions or const objects instead

**/*.{ts,tsx}: Prefer static imports over dynamic import(); use dynamic imports only for unresolvable circular dependencies, genuine performance code splitting, or conditional runtime loading.
Import Trigger.dev tasks from @trigger.dev/sdk; never use @trigger.dev/sdk/v3 or deprecated client.defineJob.
Add agentcrumbs while writing code using approved namespaces; mark lines with // @Crumbs or blocks with `// `#region` `@crumbs, and strip them before merging.

Files:

internal-packages/run-store/src/runOpsStore.ts
internal-packages/run-store/src/types.ts
internal-packages/run-store/src/PostgresRunStore.ts
apps/webapp/app/v3/services/batchTriggerV3.server.ts

**/*.{ts,tsx,js,jsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use function declarations instead of default exports

Files:

internal-packages/run-store/src/runOpsStore.ts
internal-packages/run-store/src/types.ts
internal-packages/run-store/src/PostgresRunStore.ts
apps/webapp/app/v3/services/batchTriggerV3.server.ts

**/*.ts

📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)

**/*.ts: When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs
Do not use high-cardinality attributes in OTEL metrics such as UUIDs/IDs (envId, userId, runId, projectId, organizationId), unbounded integers (itemCount, batchSize, retryCount), timestamps (createdAt, startTime), or free-form strings (errorMessage, taskName, queueName)
When exporting OTEL metrics via OTLP to Prometheus, be aware that the exporter automatically adds unit suffixes to metric names (e.g., 'my_duration_ms' becomes 'my_duration_ms_milliseconds', 'my_counter' becomes 'my_counter_total'). Account for these transformations when writing Grafana dashboards or Prometheus queries

Files:

internal-packages/run-store/src/runOpsStore.ts
internal-packages/run-store/src/types.ts
internal-packages/run-store/src/PostgresRunStore.ts
apps/webapp/app/v3/services/batchTriggerV3.server.ts

internal-packages/**/*.{ts,tsx}

📄 CodeRabbit inference engine (AGENTS.md)

For internal packages, use typecheck for verification and never use build as the correctness check.

Files:

internal-packages/run-store/src/runOpsStore.ts
internal-packages/run-store/src/types.ts
internal-packages/run-store/src/PostgresRunStore.ts

{packages/core,apps/webapp}/**/*.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use zod for validation in packages/core and apps/webapp

Files:

apps/webapp/app/v3/services/batchTriggerV3.server.ts

apps/webapp/**/*.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)

apps/webapp/**/*.{ts,tsx}: Access environment variables through the env export of env.server.ts instead of directly accessing process.env
Use subpath exports from @trigger.dev/core package instead of importing from the root @trigger.dev/core path

Do not reintroduce the removed v1 execution path; RunEngineVersion.V1 branches may only reject or finalize gracefully so v3 clients receive a clean 4xx, never a 5xx.

Files:

apps/webapp/app/v3/services/batchTriggerV3.server.ts

apps/webapp/app/**/*.{ts,tsx}