Skip to content

Create evaluations #167

@gr2m

Description

@gr2m

whenever we change the prompt in tigent.yml, the system prompt, or any relevant code, we want to make sure we don't break current behavior.

Ideas we had so far

  1. Add tests to the code with webhook fixtures and desired outcome in form of labels
  2. Create a dedicated evaluations repository on the @tigent org with issues and labels, then use the issues as input (dynamically requested when running evaluations) and the applied labels as output
  3. Use the vercel/ai repository itself by adding evaluations which is an array of numbers and use issues/PRs on the ai repository as evals

Metadata

Metadata

Assignees

No one assigned

    Labels

    ai/corecore functions like generateText, streamText, etc. Provider utils, and provider spec.featureNew feature or requestpull request welcomeFor issues with clear instructions that welcome community contirbutions

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions