Skip to content

fix(benchmark): correct postgres expected answers and remove inapplicable questions#218

Merged
jafreck merged 1 commit intomainfrom
fix/postgres-benchmark-answers
Mar 16, 2026
Merged

fix(benchmark): correct postgres expected answers and remove inapplicable questions#218
jafreck merged 1 commit intomainfrom
fix/postgres-benchmark-answers

Conversation

@jafreck
Copy link
Owner

@jafreck jafreck commented Mar 16, 2026

Summary

Fixes incorrect expected answers in the Postgres benchmark task definitions, removes questions that don't apply to the Postgres repo, and drops Q5.1 (semantic similarity) from both lore-self and postgres since it relies on embedding search rather than structural tools.

Changes

  • Q1.6 (analyze.c dead-code question): Updated expected answer from 'None' to 'addNSItemForReturning'
  • Q10.2 (top callers in utility.c): Updated with accurate caller counts from the SCIP index
  • Removed Q2.1 (type hierarchy for TupleTableSlotOps): Not a meaningful type-hierarchy question for a C codebase
  • Removed Q3.5 (external dependency mapping): Relies on build-system analysis outside Lore's current scope
  • Removed Q5.1 (semantic similarity) from lore-self and postgres: Relies on embedding-based semantic search, which we want to exclude from the structural benchmark question set

…able questions

- Update Q1.6 expected answer from 'None' to 'addNSItemForReturning'
- Update Q10.2 with accurate caller counts from SCIP index
- Remove Q2.1 (type hierarchy) and Q3.5 (external deps) — not applicable to postgres repo
@codecov
Copy link

codecov bot commented Mar 16, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.49%. Comparing base (af76cab) to head (d1c9792).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #218   +/-   ##
=======================================
  Coverage   87.49%   87.49%           
=======================================
  Files          85       85           
  Lines        9475     9475           
  Branches     2932     2932           
=======================================
  Hits         8290     8290           
  Misses       1185     1185           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@jafreck jafreck merged commit 508f9be into main Mar 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant