…Spider 2.0-lite Implement a single `text2sql` evaluation method supporting three major text-to-SQL benchmarks with shared SQL Extractor + Scorer infrastructure. Key features: - SQL Extractor: extracts SQL from LLM responses (```sql blocks, plain SELECT, etc.) - SQL Scorer: two scoring modes — Execution Accuracy (EX, default) and Exact Match (EM) - EX mode executes predicted + gold SQL against SQLite databases, compares result sets - Read-only SQLite execution (mode=ro + PRAGMA query_only) - Fallback to EM when database files unavailable Datasets: - Spider 1.0: 10 example rows, 2 databases (concert_singer, pets_1) - BIRD: 10 example rows with evidence field, 2 databases (california_schools, financial) - Spider 2.0-lite: 10 example rows, 1 database (book_store) Only lite version (85 SQLite questions) — full requires BigQuery/Snowflake credentials Tests: 67 tests covering extractor, scorer (EM + EX), SQL normalization, result set matching, SQLite execution, PRESETS, and all three example datasets. Closes #89, #90, #91, #92, #93, #94 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

feat(text2sql): speed benchmark vs reference frameworks #92

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions