Outline a structured approach to evaluating results

[digitalWestie]

Can we ask more insightful questions by customising the context? This will need tested

We want to be able to compare runs so we can answer the above question. A start might be looking at some frameworks like RubyLLM_Eval (if not to use the gem, but to learn approach).

[digitalWestie]

We'll need to make sure we are capturing run conditions. This relates to #41

[digitalWestie]

Take a look at https://github.com/reprompt-dev/reprompt

[digitalWestie]

Evaluation

Reliability

Across comparable runs we want to establish metrics for reliability e.g.:

• success/fail
• retries used
• schema/parse failures
• rate-limit/API errors
• completion time

Then we can compare with figures like:

= 95% successful runs
<= 5% runs requiring max retries
0 fatal schema failures

Requirements

To do this we need to capture exit reason more accurately e.g. (completed|failed|cancelled|timeout), at the moment exit_code (currently only inferred), so need to store it explicitly. Could do with storing error_class + normalised error_code (e.g. rate_limit, auth_error, schema_error, timeout).

As for retries, schema, and parse failures, Sensemaker just logs these out. If we want to capture these stats we could write a report file that can be picked up alongside output (failing or no=).

Output Quality

This probably needs to be a manual process conduct with random checks. E.g. sample 30 categorised comments or 10 summary claims and verify against source comments. Potentially we could setup llm judge to automate.

• Category quality: are they distinct, non-overlapping, representative?
• Categorisation quality: are the sampled comments correctly assigned?
• Summary faithfulness: no invented claims, grounded in source comments
• Clarity/actionability: useful to end users
• Coverage: important themes not missed

Requirements

This is mostly covered in our output, but we'd need somewhere to record our quality scores. Raises the question of where we do so. Is this part of Sensemaker integration? Or is it stored somewhere else just for dev purposes?

Consistency

Across 3 (or n) repeated runs per model:

• topic overlap (normalised so minor differences aren't an issue)
• % comments where the category set changes
• summary section consistency (are any major conclusions changed?)

Requirements

Same as output quality really.

[digitalWestie]

continued ...

Efficiency

• wall-clock time
• token usage (if available)
• estimated cost/run
• cost per 1k comments (?)

Requirements

We do have started at / finished at already for our job runs, but perhaps could be more specific about what those represent, potentially break down the steps if multiple scripts invoked.

Token usage is returned for each API call, so Sensemaker tools will need to keep a record. Later this could be stored in db if needed. Cost is can be derived if there's a could source of prices and price history for tokens/provider.

[digitalWestie]

Ahandy comparison site for llm stats https://llm-stats.com/models/compare/claude-sonnet-4-5-20250929-vs-llama-3.1-70b-instruct