Skip to content

feat: prompt scorer, tracer adjustments, logging, and error handling#11

Merged
adivate2021 merged 10 commits into
mainfrom
ahh/0.2.0
Sep 18, 2025
Merged

feat: prompt scorer, tracer adjustments, logging, and error handling#11
adivate2021 merged 10 commits into
mainfrom
ahh/0.2.0

Conversation

@abhishekg999

Copy link
Copy Markdown
Collaborator

No description provided.

@delve-auditor

delve-auditor Bot commented Sep 18, 2025

Copy link
Copy Markdown

No security or compliance issues detected. Reviewed everything up to caca58b.

Security Overview
  • 🔎 Scanned files: 23 changed file(s)
Detected Code Changes
Change Type Relevant files
Configuration changes ► pom.xml
    Bump Java SDK version 0.1.9 → 0.2.0
Enhancement ► src/main/java/com/judgmentlabs/judgeval/data/TraceEvaluationRun.java
    Add TraceEvaluationRun model for trace-based evaluations
► src/main/java/com/judgmentlabs/judgeval/tracer/Tracer.java
    Add asyncTraceEvaluate, trace eval serialization, logging, and ObjectMapper; improve asyncEvaluate error handling
► src/main/java/com/judgmentlabs/judgeval/tracer/OpenTelemetryKeys.java
    Add PENDING_TRACE_EVAL attribute key
► src/main/java/com/judgmentlabs/judgeval/internal/api/models/OtelTraceSpan.java
    Add OtelTraceSpan model (new)
Refactor ► src/main/java/com/judgmentlabs/judgeval/data/ExampleEvaluationRun.java
    Refactor to use ScorerConfig list only; remove mixed/local scorer handling and builders; simplify validation and builders
► src/main/java/com/judgmentlabs/judgeval/data/Example.java
    Set default exampleId to a generated UUID and import UUID
► src/main/java/com/judgmentlabs/judgeval/scorers/BaseScorer.java
    Change toTransport signature to return ScorerConfig; import model type
► src/main/java/com/judgmentlabs/judgeval/scorers/APIScorer.java
    Update toTransport return type to ScorerConfig
API model changes ► src/main/java/com/judgmentlabs/judgeval/internal/api/models/FetchPromptScorersRequest.java
    Rename/modify request to accept names (List) instead of a single name
► src/main/java/com/judgmentlabs/judgeval/internal/api/models/FetchPromptScorersResponse.java
    Rename/modify response to return list of PromptScorer objects (scorers) instead of a single scorer
► src/main/java/com/judgmentlabs/judgeval/internal/api/models/ScorerData.java
    Add id field and change evaluationModel type to String; update equals/hashCode and getters/setters
Internal API client updates ► src/main/java/com/judgmentlabs/judgeval/internal/api/JudgmentSyncClient.java
    Rename fetchScorer → fetchScorers and update endpoint to /fetch_scorers/ and response type
► src/main/java/com/judgmentlabs/judgeval/internal/api/JudgmentAsyncClient.java
    Rename fetchScorer → fetchScorers and update endpoint to /fetch_scorers/ and response type
New scorer framework & features ► src/main/java/com/judgmentlabs/judgeval/scorers/api_scorers/prompt_scorer/BasePromptScorer.java
    Introduce base class for prompt scorers with fetch/push helpers, push handling for trace flag, and common setters/getters
► src/main/java/com/judgmentlabs/judgeval/scorers/api_scorers/prompt_scorer/PromptScorer.java
    Add PromptScorer implementation using BasePromptScorer (non-trace prompt scorer)
► src/main/java/com/judgmentlabs/judgeval/scorers/api_scorers/prompt_scorer/TracePromptScorer.java
    Add TracePromptScorer implementation (trace-aware prompt scorer)
Removals / cleanup ► src/main/java/com/judgmentlabs/judgeval/scorers/api_scorers/PromptScorer.java
    Remove legacy PromptScorer (replaced by new package implementations)
► src/main/java/com/judgmentlabs/judgeval/internal/api/models/Tool.java
    Delete Tool model (removed)
► src/main/java/com/judgmentlabs/judgeval/internal/api/models/TraceSpan.java
    Delete TraceSpan model (replaced by OtelTraceSpan)
► src/main/java/com/judgmentlabs/judgeval/internal/api/models/TraceUsage.java
    Delete TraceUsage model
Script update ► scripts/generate_client.py
    Update client generation paths to use /fetch_scorers/ endpoint name
Other ► src/main/java/com/judgmentlabs/judgeval/internal/api/models/FetchPromptScorersRequest.java and FetchPromptScorersResponse.java (renames)
    Adjust JSON properties and equality/hashCode implementations to support list-based fetch semantics

Reply to this PR with @delve-auditor followed by a description of what change you want and we'll auto-submit a change to this PR to implement it.

@abhishekg999 abhishekg999 marked this pull request as ready for review September 18, 2025 02:37

@adivate2021 adivate2021 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@adivate2021 adivate2021 merged commit cb94928 into main Sep 18, 2025
2 checks passed
@adivate2021 adivate2021 deleted the ahh/0.2.0 branch September 18, 2025 05:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants