Skip to content

Add Durable Object inline schema sync#127

Merged
mmkal merged 7 commits into
mainfrom
do-inline-sync-tests
May 26, 2026
Merged

Add Durable Object inline schema sync#127
mmkal merged 7 commits into
mainfrom
do-inline-sync-tests

Conversation

@mmkal
Copy link
Copy Markdown
Collaborator

@mmkal mmkal commented May 16, 2026

Summary

Adds an opt-in runtime schema sync primitive for SQLite clients. Import only the sync primitive from sqlfu/api/sync, pass an already-created client plus inline desired schema SQL, and sqlfu plans/applies the same SQLite schema diff used by the command API.

import {createDurableObjectClient} from 'sqlfu';
import {sync} from 'sqlfu/api/sync';

const definitions = `
  create table posts (
    id integer primary key,
    slug text not null,
    body text
  );

  create unique index posts_slug_key on posts (slug);
`;

export class BlogObject extends DurableObject {
  constructor(ctx: DurableObjectState, env: Env) {
    super(ctx, env);
    const client = createDurableObjectClient(ctx.storage);
    sync(client, {definitions});
  }
}

The runtime entry intentionally lives under sqlfu/api/sync, not sqlfu/api: it is a cherry-picked subset of the API surface and does not load the Node command facade. Durable Object tests cover both first-start initialization and a redeploy-style path where the same persisted object storage starts with the old schema, then new constructor definitions add a column and a unique index while preserving existing rows.

Media

Runtime sync walkthrough screenshot:

https://github.com/user-attachments/assets/46a94a8a-ee84-4da2-bd63-e6b6c792c0b2

Runtime sync walkthrough video:

pr-127-inline-sync-demo.mp4

Before / After

Before:

// Runtime DO schema changes needed generated migration bundles.
import {migrate} from './migrations/.generated/migrations.ts';

migrate(createDurableObjectClient(ctx.storage));

The existing sqlfu/api CLI-style sync path loads the Node command facade and is not suitable for Worker runtime imports.

After:

import {sync} from 'sqlfu/api/sync';

sync(createDurableObjectClient(ctx.storage), {
  definitions: `
    create table posts (
      id integer primary key,
      slug text not null,
      body text
    );
  `,
});

sync() supports an explicit scratchSchema: 'scratch-db' | 'prefix' strategy. Normal sync clients default to an attached in-memory scratch database. Durable Object clients default to the prefix strategy because Miniflare/workerd rejects temp.sqlite_schema and normal scratch database creation there. The prefix strategy materializes desired definitions as __sqlfu_sync_* objects in main, inspects and unprefixes that desired model, plans the normal SQLite schema diff, applies the real diff in a transaction, and removes the prefixed scratch objects.

This also fixes a planner gap exposed by the test: tables classified as simple add column changes now still collect explicit index additions/modifications/removals, so adding a column and a new index in the same desired schema applies both.

Bugbot-reported scratch-prefix edge cases now have direct coverage: index names that are substrings of table names, and SQLite LIKE wildcard behavior around the scratch prefix cleanup.

A follow-up task for fully inlinable Durable Objects (inlineSqlfu(...) with inline definitions, migrations, and queries) is filed in tasks/durable-object-inline-sqlfu.md.

Verification

Passing:

pnpm --filter sqlfu build:runtime
pnpm --filter sqlfu exec vitest run test/api-sync.test.ts test/adapters/durable-object.test.ts
pnpm --filter sqlfu exec vitest run test/import-surface.test.ts -t 'built api entry|built api sync entry|built cloudflare entry'
pnpm --filter sqlfu exec vitest run test/pkg.test.ts
pnpm --filter sqlfu typecheck

Known unrelated local failure still present in this worktree:

pnpm --filter sqlfu exec vitest run test/import-surface.test.ts
# strict-tier entry "sqlfu/analyze" has disallowed import: could not resolve "node:sqlite"
# at dist/vendor/typesql/sqlfu.js
Package size — packed 242.1 kB (+2.8 kB, +1.2%)

Package size

main this PR Δ
packed 239.4 kB 242.1 kB +1.2%
unpacked 991.8 kB 1006.7 kB +1.5%
files 185 187 +2

dist/vendor/*.js bundles

main this PR Δ
vendor/sha256.js 4.3 kB 4.3 kB 0
vendor/sql-formatter/*.js 58.3 kB 58.3 kB 0
vendor/sqlfu-sqlite-parser/*.js 17.2 kB 17.2 kB 0
vendor/standard-schema/*.js 2.8 kB 2.8 kB 0
vendor/typesql/*.js 134.6 kB 134.6 kB 0

Measured with npm pack --dry-run --json on sqlfu (0.0.3-7 on main vs 0.0.3-7 on this PR).


Note

Medium Risk
Applies destructive-capable schema migrations directly on live SQLite (including DO storage) at runtime; mistakes could alter production object data, though behavior mirrors existing CLI sync and is covered by integration tests.

Overview
Adds a runtime schema sync entrypoint at sqlfu/api/sync so Workers and Durable Objects can align SQLite storage to inline definitions SQL without the Node sqlfu/api command facade.

sync(client, { definitions, scratchSchema?, allowDestructive? }) inspects the live database, materializes the desired schema in a scratch space, runs the existing SQLite diff planner, and applies the result in a transaction. Scratch strategy: default attached in-memory DB (scratch-db); Durable Object clients auto-select prefix mode (__sqlfu_sync_* objects in main) because attach/temp scratch is unreliable there. Migration bookkeeping tables (sqlfu_migrations, d1_migrations) are excluded from comparison.

Supporting changes: inspectSqliteSchema is dual sync/async and schema-aware (quoted schema + pragma schema args); add-column diffs now also emit explicit index add/drop/recreate (fixes column + index in one redeploy). Docs, package exports, packed-import tests, DO Miniflare tests (init + persisted redeploy), and api-sync regressions for prefix edge cases.

Reviewed by Cursor Bugbot for commit 78531d5. Bugbot is set up for automated code reviews on this repo. Configure here.

@pkg-pr-new
Copy link
Copy Markdown

pkg-pr-new Bot commented May 16, 2026

Open in StackBlitz

npm i https://pkg.pr.new/sqlfu@127

commit: 78531d5

Comment thread packages/sqlfu/src/api/sync.ts Outdated
Comment thread packages/sqlfu/src/api/sync.ts Outdated
@mmkal
Copy link
Copy Markdown
Collaborator Author

mmkal commented May 21, 2026

to avoid muddying the API and having too many things to remember, do you think we could bake this behaviour into the existing sync(...) function under sqlfu/api

basically we could figure out somehow if we're in an environment that doesn't support spawning a scratch database and go through this slightly messier path of sqlfu_temp_ prefixing

Comment thread packages/sqlfu/src/api/sync.ts
Comment thread packages/sqlfu/src/api/sync.ts
Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 78531d5. Configure here.

Comment thread packages/sqlfu/src/api/sync.ts
Comment thread packages/sqlfu/src/api/sync.ts
@mmkal mmkal merged commit e2bd6cc into main May 26, 2026
9 checks passed
@mmkal mmkal deleted the do-inline-sync-tests branch May 26, 2026 14:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant