Skip to content

Consolidate SQLite parser-shaped helpers #137

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mmkal merged 6 commits into from
May 27, 2026
Merged

Consolidate SQLite parser-shaped helpers #137

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
905499f
Document SQLite parser facade architecture task
mmkal May 26, 2026
ab0c731
Add SQLite parser facade
mmkal May 26, 2026
63c1eea
Use SQLite parser spans for runtime sync rewrites
mmkal May 26, 2026
d21ff18
Fix parser facade identifier and row-return regressions
mmkal May 27, 2026
be0aeb2
Inline create-kind checks in parser facade PR
mmkal May 27, 2026
8f28611
Remove unused sqlite tokenizer facade
mmkal May 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ Note: this library is in pre-pre-pre-alpha. There are zero users. It is EXTREMEL

Prefer lowercase SQL keywords.

Do not parse SQL with fragile regexes; use `packages/sqlfu/src/sqlite-parser.ts` for SQLite parser-shaped work.

## Docs examples

For small docs, examples, and guides, put query examples in a catch-all
Expand Down
265 changes: 92 additions & 173 deletions packages/sqlfu/src/api/sync.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@ import {quoteIdentifier} from '../schemadiff/sqlite/identifiers.js';
import {inspectSqliteSchema} from '../schemadiff/sqlite/inspect.js';
import {planSchemaDiff} from '../schemadiff/sqlite/plan.js';
import type {SqliteInspectedDatabase} from '../schemadiff/sqlite/types.js';
import {
classifySqliteCreateStatement,
replaceSqliteIdentifierSpan,
type SqliteIdentifierSpan,
} from '../sqlite-parser.js';
import {splitSqlStatements} from '../sqlite-text.js';
import type {SyncClient} from '../types.js';

Expand Down Expand Up @@ -92,195 +97,112 @@ function applyDefinitionsToPrefixedSchema(client: SyncClient, definitions: strin
}

function orderedSchemaStatements(sql: string): string[] {
const statements = splitSqlStatements(sql);
const statements = splitSqlStatements(sql).map((statement) => ({
statement,
createStatement: classifySqliteCreateStatement(statement),
}));
return [
...statements.filter((statement) => isCreateTableStatement(statement)),
...statements.filter((statement) => isCreateIndexStatement(statement)),
...statements.filter((statement) => isCreateViewStatement(statement)),
...statements.filter((entry) => entry.createStatement?.kind === 'table'),
...statements.filter((entry) => entry.createStatement?.kind === 'index'),
...statements.filter((entry) => entry.createStatement?.kind === 'view'),
...statements.filter(
(statement) =>
!isCreateTableStatement(statement) && !isCreateIndexStatement(statement) && !isCreateViewStatement(statement),
(entry) =>
entry.createStatement?.kind !== 'table' &&
entry.createStatement?.kind !== 'index' &&
entry.createStatement?.kind !== 'view',
),
];
].map((entry) => entry.statement);
Copy link
Copy Markdown

@cursor cursor Bot May 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicated statement ordering logic across two files

Low Severity

The classify-then-order-by-kind pattern (split → classifySqliteCreateStatement → filter into table/index/view/rest buckets) is duplicated almost identically between orderedSchemaStatements in sync.ts and applySchemaSql in schemadiff/sqlite/index.ts. If the ordering needs to change (e.g. adding explicit trigger ordering), both sites must be updated in lockstep. This could be extracted into a shared helper in sqlite-parser.ts or sqlite-text.ts.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 8f28611. Configure here.

}

function toPrefixedSchemaStatement(statement: string): string {
if (/^create\s+virtual\s+table\b/iu.test(stripLeadingComments(statement))) {
throw new Error('runtime sync does not support sqlite virtual tables yet');
}
if (isCreateTableStatement(statement)) {
return prefixCreateObjectName(statement, 'table');
}
if (isCreateIndexStatement(statement)) {
return prefixCreateIndexStatement(statement);
const createStatement = classifySqliteCreateStatement(statement);
if (!createStatement) {
throw new Error(
'runtime sync definitions only support create table, create index, create view, and create trigger',
);
}
if (isCreateViewStatement(statement)) {
return prefixCreateObjectName(statement, 'view');
switch (createStatement.kind) {
case 'virtual-table':
throw new Error('runtime sync does not support sqlite virtual tables yet');
case 'table':
case 'view':
return replaceRequiredIdentifier(statement, createStatement.name, (name) =>
quoteIdentifier(`${syncObjectPrefix}${name}`),
);
case 'index':
case 'trigger':
return replaceRequiredIdentifiers(statement, [
{identifier: createStatement.name, replacement: (name) => quoteIdentifier(`${syncObjectPrefix}${name}`)},
{identifier: createStatement.onName, replacement: (name) => quoteIdentifier(`${syncObjectPrefix}${name}`)},
]);
}
if (isCreateTriggerStatement(statement)) {
return prefixCreateTriggerStatement(statement);
}
throw new Error('runtime sync definitions only support create table, create index, create view, and create trigger');
}

function toAttachedSchemaStatement(statement: string, schemaName: string): string {
const stripped = stripLeadingComments(statement);
if (/^create\s+virtual\s+table\b/iu.test(stripped)) {
const createStatement = classifySqliteCreateStatement(statement);
if (!createStatement) {
throw new Error(
'runtime sync definitions only support create table, create index, create view, and create trigger',
);
}
if (createStatement.kind === 'virtual-table') {
throw new Error('runtime sync does not support sqlite virtual tables yet');
}
if (/^create\s+(?:temp|temporary)\s+/iu.test(stripped)) {
if (createStatement.temporary) {
throw new Error('runtime sync scratch-db definitions do not support temp schema objects');
}
if (isCreateTableStatement(statement)) {
return qualifyCreateObjectName(statement, 'table', schemaName);
}
if (isCreateIndexStatement(statement)) {
return qualifyCreateIndexStatement(statement, schemaName);
switch (createStatement.kind) {
case 'table':
case 'view':
return replaceRequiredIdentifier(
statement,
createStatement.name,
(name) => `${quoteIdentifier(schemaName)}.${quoteIdentifier(name)}`,
);
case 'index':
case 'trigger':
return replaceRequiredIdentifiers(statement, [
{
identifier: createStatement.name,
replacement: (name) => `${quoteIdentifier(schemaName)}.${quoteIdentifier(name)}`,
},
{identifier: createStatement.onName, replacement: (name) => quoteIdentifier(name)},
]);
}
if (isCreateViewStatement(statement)) {
return qualifyCreateObjectName(statement, 'view', schemaName);
}
if (isCreateTriggerStatement(statement)) {
return qualifyCreateTriggerStatement(statement, schemaName);
}
throw new Error('runtime sync definitions only support create table, create index, create view, and create trigger');
}

function isCreateTableStatement(statement: string): boolean {
return /^create\s+(?:(?:temp|temporary)\s+)?table\b/iu.test(stripLeadingComments(statement));
}

function isCreateIndexStatement(statement: string): boolean {
return /^create\s+(?:unique\s+)?index\b/iu.test(stripLeadingComments(statement));
}

function isCreateViewStatement(statement: string): boolean {
return /^create\s+(?:(?:temp|temporary)\s+)?view\b/iu.test(stripLeadingComments(statement));
}

function isCreateTriggerStatement(statement: string): boolean {
return /^create\s+(?:(?:temp|temporary)\s+)?trigger\b/iu.test(stripLeadingComments(statement));
}

function stripLeadingComments(statement: string): string {
return statement.replace(/^(?:\s+|--[^\n]*(?:\n|$)|\/\*[\s\S]*?\*\/)+/u, '');
}

function prefixCreateObjectName(statement: string, kind: 'table' | 'view'): string {
return replaceMatchedIdentifier(
statement,
new RegExp(
`^(${leadingCommentPattern}create\\s+(?:(?:temp|temporary)\\s+)?${kind}\\s+(?:if\\s+not\\s+exists\\s+)?)(?<name>${identifierPattern})`,
'iu',
),
'name',
(name) => quoteIdentifier(`${syncObjectPrefix}${name}`),
);
}

function prefixCreateIndexStatement(statement: string): string {
return replaceMatchedIdentifier(
replaceMatchedIdentifier(
statement,
new RegExp(
`^(${leadingCommentPattern}create\\s+(?:unique\\s+)?index\\s+(?:if\\s+not\\s+exists\\s+)?)(?<name>${identifierPattern})(?<tail>\\s+on\\s+)(?<table>${identifierPattern})`,
'iu',
),
'name',
(name) => quoteIdentifier(`${syncObjectPrefix}${name}`),
),
new RegExp(
`^(${leadingCommentPattern}create\\s+(?:unique\\s+)?index\\s+(?:if\\s+not\\s+exists\\s+)?${identifierPattern}\\s+on\\s+)(?<table>${identifierPattern})`,
'iu',
),
'table',
(name) => quoteIdentifier(`${syncObjectPrefix}${name}`),
);
}

function prefixCreateTriggerStatement(statement: string): string {
return replaceMatchedIdentifier(
replaceMatchedIdentifier(
statement,
new RegExp(
`^(${leadingCommentPattern}create\\s+(?:(?:temp|temporary)\\s+)?trigger\\s+(?:if\\s+not\\s+exists\\s+)?)(?<name>${identifierPattern})`,
'iu',
),
'name',
(name) => quoteIdentifier(`${syncObjectPrefix}${name}`),
),
new RegExp(
`^(${leadingCommentPattern}create\\s+(?:(?:temp|temporary)\\s+)?trigger\\s+(?:if\\s+not\\s+exists\\s+)?${identifierPattern}[\\s\\S]*?\\bon\\s+)(?<table>${identifierPattern})`,
'iu',
),
'table',
(name) => quoteIdentifier(`${syncObjectPrefix}${name}`),
);
}

function qualifyCreateObjectName(statement: string, kind: 'table' | 'view', schemaName: string): string {
return replaceMatchedIdentifier(
statement,
new RegExp(
`^(${leadingCommentPattern}create\\s+${kind}\\s+(?:if\\s+not\\s+exists\\s+)?)(?<name>${identifierPattern})`,
'iu',
),
'name',
(name) => `${quoteIdentifier(schemaName)}.${quoteIdentifier(name)}`,
);
}

function qualifyCreateIndexStatement(statement: string, schemaName: string): string {
return replaceMatchedIdentifier(
statement,
new RegExp(
`^(${leadingCommentPattern}create\\s+(?:unique\\s+)?index\\s+(?:if\\s+not\\s+exists\\s+)?)(?<name>${identifierPattern})`,
'iu',
),
'name',
(name) => `${quoteIdentifier(schemaName)}.${quoteIdentifier(name)}`,
);
}

function qualifyCreateTriggerStatement(statement: string, schemaName: string): string {
return replaceMatchedIdentifier(
statement,
new RegExp(
`^(${leadingCommentPattern}create\\s+trigger\\s+(?:if\\s+not\\s+exists\\s+)?)(?<name>${identifierPattern})`,
'iu',
),
'name',
(name) => `${quoteIdentifier(schemaName)}.${quoteIdentifier(name)}`,
);
}

function replaceMatchedIdentifier(
function replaceRequiredIdentifier(
statement: string,
pattern: RegExp,
groupName: string,
identifier: SqliteIdentifierSpan | null,
replacement: (name: string) => string,
): string {
const match = pattern.exec(statement);
const rawIdentifier = match?.groups?.[groupName];
const beforeIdentifier = match?.[1];
if (!match || !rawIdentifier || !beforeIdentifier) {
if (!identifier) {
throw new Error(`runtime sync could not parse schema definition statement: ${statement}`);
}
const start = match.index + beforeIdentifier.length;
return `${statement.slice(0, start)}${replacement(parseIdentifierName(rawIdentifier))}${statement.slice(start + rawIdentifier.length)}`;
return replaceSqliteIdentifierSpan(statement, identifier, replacement(identifier.name));
}

function parseIdentifierName(rawIdentifier: string): string {
if (rawIdentifier.startsWith('"') && rawIdentifier.endsWith('"')) {
return rawIdentifier.slice(1, -1).replaceAll('""', '"');
}
if (rawIdentifier.startsWith('`') && rawIdentifier.endsWith('`')) {
return rawIdentifier.slice(1, -1).replaceAll('``', '`');
}
if (rawIdentifier.startsWith('[') && rawIdentifier.endsWith(']')) {
return rawIdentifier.slice(1, -1);
function replaceRequiredIdentifiers(
statement: string,
replacements: {
identifier: SqliteIdentifierSpan | null;
replacement: (name: string) => string;
}[],
): string {
let output = statement;
const sorted = replacements
.map((replacement) => {
if (!replacement.identifier) {
throw new Error(`runtime sync could not parse schema definition statement: ${statement}`);
}
return {identifier: replacement.identifier, value: replacement.replacement(replacement.identifier.name)};
})
.sort((left, right) => right.identifier.start - left.identifier.start);

for (const replacement of sorted) {
output = replaceSqliteIdentifierSpan(output, replacement.identifier, replacement.value);
}
return rawIdentifier;
return output;
}

function unprefixInspectedSchema(schema: SqliteInspectedDatabase): SqliteInspectedDatabase {
Expand Down Expand Up @@ -352,16 +274,18 @@ function removeSyncPrefixes(value: string): string {
}

function cleanupPrefixedObjects(client: SyncClient): void {
const rows = client.all<{type: 'index' | 'table' | 'trigger' | 'view'; name: string}>({
sql: `
const rows = client
.all<{type: 'index' | 'table' | 'trigger' | 'view'; name: string}>({
sql: `
select type, name
from main.sqlite_schema
where type in ('index', 'table', 'trigger', 'view')
and name not like 'sqlite\\_%' escape '\\'
order by type, name
`,
args: [],
}).filter((row) => row.name.startsWith(syncObjectPrefix));
args: [],
})
.filter((row) => row.name.startsWith(syncObjectPrefix));
const typeOrder = ['trigger', 'view', 'index', 'table'] as const;
for (const type of typeOrder) {
for (const row of rows.filter((candidate) => candidate.type === type)) {
Expand All @@ -373,9 +297,7 @@ function cleanupPrefixedObjects(client: SyncClient): void {
function excludeRuntimeSyncTables(schema: SqliteInspectedDatabase): SqliteInspectedDatabase {
return {
...schema,
tables: Object.fromEntries(
Object.entries(schema.tables).filter(([name]) => !runtimeSyncExcludedTables.has(name)),
),
tables: Object.fromEntries(Object.entries(schema.tables).filter(([name]) => !runtimeSyncExcludedTables.has(name))),
triggers: Object.fromEntries(
Object.entries(schema.triggers).filter(([, trigger]) => !runtimeSyncExcludedTables.has(trigger.onName)),
),
Expand Down Expand Up @@ -408,6 +330,3 @@ function isObject(value: unknown): value is Record<string, unknown> {

const syncObjectPrefix = '__sqlfu_sync_';
const runtimeSyncExcludedTables = new Set(['sqlfu_migrations', 'd1_migrations']);
const leadingCommentPattern = String.raw`(?:\s+|--[^\n]*(?:\n|$)|\/\*[\s\S]*?\*\/)*`;
const identifierPattern =
String.raw`(?:"(?:[^"]|"")+"|` + '`(?:[^`]|``)+`' + String.raw`|\[[^\]]+\]|[a-z_][a-z0-9_$]*)`;
Loading
Loading