Skip to content

feat: add multi-dimensional quality scorer for structured outputs#16

Open
lustsazeus-lab wants to merge 1 commit intoMint-Claw:mainfrom
lustsazeus-lab:main
Open

feat: add multi-dimensional quality scorer for structured outputs#16
lustsazeus-lab wants to merge 1 commit intoMint-Claw:mainfrom
lustsazeus-lab:main

Conversation

@lustsazeus-lab
Copy link

Summary

Implements a multi-dimensional quality scorer for structured outputs (JSON, markdown, code, text) as specified in issue #1.

Features

  • 5-Dimension Scoring:

    • Completeness (30%): How complete the submission is
    • Format Compliance (20%): Follows expected format
    • Coverage (25%): Covers required aspects
    • Clarity (15%): Readable and clear content
    • Validity (10%): Valid and accurate content
  • Auto-format Detection: Detects JSON, Markdown, Code, Text automatically

  • Output Schema:

{
  "weighted_score": 0.85,
  "quality_rating": "excellent",
  "scores": {
    "completeness": 0.9,
    "format_compliance": 1.0,
    "coverage": 0.8,
    "clarity": 0.7,
    "validity": 0.9
  },
  "feedback": [...],
  "pass_threshold": true
}
  • Performance: Batch scoring supports 100+ submissions efficiently

Tests

28 comprehensive tests covering:

  • Format detection
  • Individual dimension scoring
  • Integration tests
  • Edge cases

Usage

from scorer import QualityScorer

scorer = QualityScorer()
result = scorer.score(your_content)

print(f"Score: {result.weighted_score}")
print(f"Rating: {result.quality_rating.value}")

Bounty: (+ potential bonus for NLP feedback)

- Added scorer.py with 5-dimension scoring:
  - Completeness (0.30)
  - Format Compliance (0.20)
  - Coverage (0.25)
  - Clarity (0.15)
  - Validity (0.10)
- Auto-detects format: JSON, Markdown, Code, Text
- Returns weighted_score, quality_rating, scores, feedback, pass_threshold
- Includes batch scoring for 100+ submissions
- Added comprehensive test suite (28 tests)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant