Skip to content

feat: Grader plugin extensibility (WASM/external programs) #18

Description

@spboyer

Migrated from spboyer/waza#390

Summary

Design how custom graders beyond config-driven types can be distributed via the registry.

Context

From waza-vs-openai-evals.md Row 15: OpenAI Evals allows Python class inheritance for custom eval logic. Waza is config-driven only (code, program graders). This is waza's #2 competitive gap.

Options

  1. WASM plugins — compile grader logic to WASM, distribute via registry. Sandboxed, cross-platform.
  2. External program protocol — any binary that accepts stdin JSON and outputs grader results. Already partially supported via program grader.
  3. Go plugin interfaceplugin.Open() for native Go graders. Fast but platform-specific.
  4. Embedded scripting — extend code grader beyond Python (JavaScript, Lua, Starlark).

Parent: #385

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestepic:evaluationE3: Evaluation Frameworkgo:needs-researchNeeds investigationgradersneeds-author-feedbackNeeds additional details from issue author to validate or implementquestionFurther information is requestedrelease:backlogNot yet targetedsecuritytype:spikeResearch/investigation — produces a plan, not code

    Fields

    No fields configured for Feature.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions