Skip to content

feat: Go-module-style grader/eval references #15

Description

@spboyer

Migrated from spboyer/waza#387

Summary

Design how eval.yaml can reference remote graders via Go-module-style paths.

Concept

graders:
  - ref: github.com/waza-evals/fact@v1.0
  - ref: github.com/myorg/custom-grader@v2.1
    config:
      threshold: 0.8

Design Questions

  • How does waza get resolve and cache remote graders?
  • Version pinning semantics (exact, range, latest)?
  • Offline/air-gapped fallback from local cache?
  • How to handle transitive dependencies (grader A depends on dataset B)?
  • Authentication for private repos?

Parent: #385

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestepic:evaluationE3: Evaluation Frameworkgo:needs-researchNeeds investigationgradersneeds-author-feedbackNeeds additional details from issue author to validate or implementquestionFurther information is requestedrelease:backlogNot yet targetedtype:spikeResearch/investigation — produces a plan, not code

    Fields

    No fields configured for Feature.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions