Skip to content

fix: parse embedded ranking JSON#12

Open
alibababye wants to merge 1 commit into
OpenNSWM-Lab:mainfrom
alibababye:fix/ranking-json-extraction
Open

fix: parse embedded ranking JSON#12
alibababye wants to merge 1 commit into
OpenNSWM-Lab:mainfrom
alibababye:fix/ranking-json-extraction

Conversation

@alibababye

Copy link
Copy Markdown

Problem

RankingService._parse_llm_response could parse fenced JSON, but the fallback for prose-wrapped responses only matched objects without nested braces. Ranking responses are expected to contain nested criterion objects such as novelty.score, so a response like Here is the result: {...} could parse the first inner object instead of the full payload and leave the criteria at default values.

Solution

This updates the fallback JSON extraction to capture the full object and adds a regression test for nested ranking JSON embedded in explanatory text.

Testing

Ran in the sandbox:

python3 tests/test_ranking_variance.py

Handle nested ranking JSON objects when an LLM wraps the response in explanatory prose. This keeps criterion scores from silently falling back to defaults.
@alibababye alibababye force-pushed the fix/ranking-json-extraction branch from 20e19d8 to 95c95d9 Compare June 22, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant