Skip to content

Add Lichess loss analysis script#28

Merged
metaember merged 1 commit into
masterfrom
add-loss-analysis-script
Feb 23, 2026
Merged

Add Lichess loss analysis script#28
metaember merged 1 commit into
masterfrom
add-loss-analysis-script

Conversation

@metaember

Copy link
Copy Markdown
Owner

Summary

  • Adds scripts/analyze_losses.py — downloads bot losses/draws from Lichess and categorizes errors
  • For each bot move: runs custom engine, Stockfish at same depth, and Stockfish deep (ground truth)
  • Classifies mistakes as depth issues (SF also misses at same depth) vs eval issues (SF finds it, engine doesn't)
  • Verifies resignations were justified, flags draws where wins were missed

Usage

uv run --project scripts scripts/analyze_losses.py --since-hours 12 --verbose

Initial findings (12 games, last 12 hours)

  • 82 mistakes, 16 blunders across 8 losses + 4 draws
  • 59% eval issues vs 41% depth issues — evaluation function is the primary weakness
  • 1 unjustified resignation (position was +274cp)
  • 3 draws with missed wins (up to +504cp advantage squandered)

Test plan

  • Ran on 1 game with --verbose to verify per-move output
  • Ran on all 12 losses/draws from last 12 hours successfully
  • JSON report output verified

Python script that downloads the bot's losses/draws from Lichess,
replays each game, and categorizes errors as depth issues (engine
needed deeper search) vs eval issues (evaluation function is wrong).

Features:
- Per-move analysis comparing custom engine vs Stockfish at same depth
- Resignation verification (was the position actually lost?)
- Draw analysis (were there missed winning opportunities?)
- Configurable thresholds, depths, and time filters
- JSON + terminal report output
@metaember metaember merged commit cfebb98 into master Feb 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant