Skip to content

feat: Map OpenAI Evals YAML format → waza graders #14

Description

@spboyer

Migrated from spboyer/waza#386

Summary

Map OpenAI Evals' modelgraded YAML schema to waza prompt grader config with concrete translation examples.

OpenAI Format (examples from their registry)

fact.yaml: prompt + choice_strings (ABCDE) + input_outputs
closedqa.yaml: prompt + eval_type (cot_classify) + choice_scores (Y:1.0, N:0.0)
battle.yaml: prompt + choice_scores + dual input_outputs for head-to-head comparison

Deliverable

Field-level mapping table and working translations of fact.yaml, closedqa.yaml, battle.yaml → waza eval.yaml.

Parent: #385

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Fields

    No fields configured for Feature.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions