Skip to content

feat(eval): add 25 new per-skill A/B evaluation reports#101

Merged
dmartinol merged 1 commit into
mainfrom
per_skill_eval/reports_1
May 12, 2026
Merged

feat(eval): add 25 new per-skill A/B evaluation reports#101
dmartinol merged 1 commit into
mainfrom
per_skill_eval/reports_1

Conversation

@GuyZivRH
Copy link
Copy Markdown
Contributor

Add evaluation reports for 28 skills across 6 personas (ocp-admin, rh-ai-engineer, rh-basic, rh-developer, rh-sre, rh-virt) from the ABEvalFlow pipeline. Updates existing playbook-generator and remediation reports with latest runs, fixes misplaced job-template-remediation-validator to its own folder, and corrects LLM label from 4.6 to 4.5.

Summary

Pack(s) affected

  • rh-sre
  • rh-developer
  • ocp-admin
  • rh-virt
  • rh-ai-engineer
  • Other / repo-wide

Change type

  • New skill
  • New agent
  • New pack
  • Update existing skill / agent
  • MCP server config (mcps.json)
  • Docs / README
  • CI / tooling

CLAUDE.md compliance

  • Agents orchestrate skills; no direct MCP/tool calls in agents
  • Skills are single-purpose task executors
  • Skills encapsulate all tool access (MCP tools invoked only inside skills)
  • Document consultation: file is read with the Read tool, then declared to the user
  • No credentials hardcoded; env vars used via ${VAR} references
  • Human-in-the-loop confirmation added for any destructive or critical operations

Validation

  • make validate passes locally
  • New/changed skills have valid YAML frontmatter (name, description)
  • New/changed agents have valid YAML frontmatter (name, description)

Add evaluation reports for 28 skills across 6 personas (ocp-admin,
rh-ai-engineer, rh-basic, rh-developer, rh-sre, rh-virt) from the
ABEvalFlow pipeline. Updates existing playbook-generator and remediation
reports with latest runs, fixes misplaced job-template-remediation-validator
to its own folder, and corrects LLM label from 4.6 to 4.5.

Co-authored-by: Cursor <cursoragent@cursor.com>
@GuyZivRH GuyZivRH requested a review from dmartinol May 12, 2026 11:31
Copy link
Copy Markdown
Collaborator

@dmartinol dmartinol left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! /lgtm

@dmartinol dmartinol merged commit 3a56977 into main May 12, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants