Consolidate SQLite parser-shaped helpers

[mmkal]

Summary

Adds a small internal SQLite parser facade backed by the existing vendored sqlfu-sqlite-parser tokenizer. The facade exposes sqlfu-shaped helpers for tokenization, first-keyword detection, keyword presence, CREATE statement classification, and CREATE identifier spans.

The implementation routes parser-shaped call sites through that facade:

• runtime sync CREATE statement ordering, unsupported virtual-table detection, and CREATE name / on-table rewrites
• schemadiff scratch SQL application ordering and virtual-table detection
• sqlReturnsRows first-keyword / returning detection

Before / after

Before, these paths each had their own regex/comment-stripping logic, so behavior depended on the local scanner. Schemadiff could see create virtual table inside a comment or string literal, and runtime sync name rewriting could classify create /* comment */ index ... but still fail when the old rewrite regex tried to find the object name.

After, valid SQLite text is classified from tokenizer output, and runtime sync rewrites object names using token spans. Comments, strings, quoted identifiers, and bare identifiers containing $ are handled consistently. The row-return helper keeps a tolerant fallback for PostgreSQL-ish ad hoc SQL such as select value::json, show timezone, and fetch all from cursor_name, preserving the previous UI behavior while avoiding false positives for normal SQLite strings/comments.

Tests

• pnpm --filter sqlfu exec vitest run test/sqlfu-sqlite-parser/tokenizer.test.ts test/sqlite-parser.test.ts test/sqlite-text.test.ts test/api-sync.test.ts test/schemadiff/plumbing.test.ts
• pnpm --filter sqlfu typecheck
• pnpm exec oxfmt --check packages/sqlfu/src/vendor/sqlfu-sqlite-parser/tokenizer.ts packages/sqlfu/src/sqlite-parser.ts packages/sqlfu/src/sqlite-text.ts packages/sqlfu/src/api/sync.ts packages/sqlfu/src/schemadiff/sqlite/index.ts packages/sqlfu/test/sqlfu-sqlite-parser/tokenizer.test.ts packages/sqlfu/test/sqlite-parser.test.ts packages/sqlfu/test/sqlite-text.test.ts packages/sqlfu/test/api-sync.test.ts packages/sqlfu/test/schemadiff/plumbing.test.ts

Notes

Named parameter scanning is intentionally left alone in this PR. The existing query-parameter scanner already has focused coverage for comments, strings, and PostgreSQL-style casts; this branch keeps the first parser-facade pass scoped to statement/header and keyword classification.

Package size — packed 244.5 kB (+1.2 kB, +0.5%)

Package size

	main	this PR	Δ
packed	243.3 kB	244.5 kB	+0.5%
unpacked	1011.8 kB	1019.1 kB	+0.7%
files	187	189	+2

dist/vendor/*.js bundles

	main	this PR	Δ
vendor/sha256.js	4.3 kB	4.3 kB	0
vendor/sql-formatter/*.js	58.3 kB	58.3 kB	0
vendor/sqlfu-sqlite-parser/*.js	17.2 kB	17.2 kB	+0.1%
vendor/standard-schema/*.js	2.8 kB	2.8 kB	0
vendor/typesql/*.js	134.6 kB	134.6 kB	0

Measured with npm pack --dry-run --json on sqlfu (0.0.3-7 on main vs 0.0.3-7 on this PR).

---

Note

Medium Risk
Changes how schema definition SQL is classified and rewritten in runtime sync and schemadiff; behavior is broader but well-covered by new tests, with intentional fallbacks for non-SQLite ad hoc SQL.

Overview
Introduces an internal sqlite-parser facade on the vendored tokenizer so runtime sync, schemadiff, and sqlReturnsRows share one path for CREATE classification, keyword detection, and identifier spans instead of per-caller regex and comment stripping.

Runtime sync and schemadiff now order CREATE statements by tokenized kind (so an index listed before its table still applies correctly), detect virtual tables without false positives in comments/strings, and rewrite CREATE names via span replacement—including comments between keywords and bare identifiers with $. The tokenizer treats $ as part of identifiers (not only bind parameters). Row-return heuristics keep tolerant fallbacks for PostgreSQL-style ad hoc SQL (::json, show, fetch).

^{Reviewed by Cursor Bugbot for commit d21ff18. Bugbot is set up for automated code reviews on this repo. Configure here.}

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/sqlfu@137

commit: d21ff18

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d21ff18. Configure here.}

[cursor]

Qualified names partially rewritten

Medium Severity

When a CREATE statement uses a dotted object name, readIdentifier records a span for only the final segment but leaves the earlier schema. text in place. Runtime sync then splices prefixed or scratch-qualified names into the middle of that qualifier, producing invalid or mis-targeted DDL instead of failing cleanly.

Additional Locations (1)

• packages/sqlfu/src/api/sync.ts#L164-L212

 

^{Reviewed by Cursor Bugbot for commit d21ff18. Configure here.}