Skip to content

feat(ci): PR Validation for AI Artifact Registry Drift Detection #468

@katriendg

Description

@katriendg

Parent: #431

Issue Description

Wire the existing Validate-ArtifactRegistry.ps1 into the PR validation workflow so that any pull request adding, removing, or renaming an AI artifact file without updating the registry manifest is caught before merge.

Motivation

The AI Artifacts Registry (ai-artifacts-registry.json) is the single source of truth for persona tagging, maturity levels, and dependency metadata. If a contributor adds a new prompt, deletes an agent, or renames an instruction file without updating the registry, the result is silent drift: the packaging scripts, installer, and collection manifests all operate on stale data.

The Validate-ArtifactRegistry.ps1 script already performs dual-direction validation:

  • Registry → Disk: Each registry entry maps to a file on disk. A removed or renamed file causes a "file not found" error.
  • Disk → Registry: Each artifact file on disk should appear in the registry. A new or renamed file without a registry entry is reported as an orphan warning.

The script also excludes instructions/hve-core/* from orphan detection, since those are repo-specific and not distributed.

The missing piece is wiring this validator into the PR gate with -WarningsAsErrors so orphan warnings become hard failures.

Current State

Component Status
Validate-ArtifactRegistry.ps1 Exists, full validation logic including orphan detection
npm run lint:registry Exists, runs the validator
Pester test suite Exists, 17 fixtures covering structure, persona refs, entries, dependencies, orphans, CI annotations, exit codes, renamed-file mismatch
Reusable workflow ✅ Implemented
PR validation job ✅ Implemented
Test fixtures for renamed-file mismatch ✅ Implemented

Deliverables

1. Reusable Workflow

Create .github/workflows/artifact-registry-validation.yml as a reusable workflow_call workflow following the same patterns as frontmatter-validation.yml:

  • Inputs: soft-fail (boolean, default false), warnings-as-errors (boolean, default true)
  • Steps: checkout, create logs directory, run Validate-ArtifactRegistry.ps1, upload results artifact, fail-on-error gate
  • Permissions: contents: read

2. PR Validation Integration

Add an artifact-registry-validation job to .github/workflows/pr-validation.yml:

artifact-registry-validation:
  name: Artifact Registry Validation
  uses: ./.github/workflows/artifact-registry-validation.yml
  permissions:
    contents: read
  with:
    soft-fail: false
    warnings-as-errors: true

This catches three drift scenarios on every PR:

Scenario Detection Mechanism Existing Logic
File added, registry not updated Orphan detection (Find-OrphanArtifacts) Yes, reports as warning; -WarningsAsErrors promotes to error
File removed, registry not updated File existence check (Test-ArtifactFileExistence) Yes, reports as error
File renamed, registry not updated Both: old key fails existence, new file is orphan Yes, combination of above two checks

3. Test Fixtures for Renamed-File Mismatch

Add a fixture that simulates the renamed-file scenario (old registry key pointing to a non-existent file while a new file exists on disk but is unregistered). This validates the dual-direction detection works in concert.

Add Pester test cases to Validate-ArtifactRegistry.Tests.ps1:

  • File added but not in registry (orphan detected, promoted to error with -WarningsAsErrors)
  • File removed but still in registry (existence error)
  • File renamed: old key errors on existence AND new file detected as orphan
  • Files under instructions/hve-core/ remain excluded from orphan detection

Acceptance Criteria

  • Reusable workflow artifact-registry-validation.yml exists and can be called independently
  • PR validation workflow includes the registry validation job
  • A PR adding a new .agent.md, .prompt.md, .instructions.md, or SKILL.md without a registry entry fails validation
  • A PR removing an artifact file without removing its registry entry fails validation
  • A PR renaming an artifact file without updating the registry entry fails validation (both old-key error and new-file orphan)
  • Files under .github/instructions/hve-core/ are excluded from orphan detection (existing behavior preserved)
  • Pester tests cover the renamed-file mismatch scenario with fixtures
  • Validation results are uploaded as a CI artifact for debugging

Technical Notes

  • No changes to Validate-ArtifactRegistry.ps1 are needed. The existing -WarningsAsErrors flag already promotes orphan warnings to errors.
  • The reusable workflow follows the frontmatter-validation.yml pattern: checkout → logs directory → run script → upload artifact → fail gate.
  • The pester-tests job already runs in PR validation and will pick up new test files automatically via its test discovery pattern.
  • Code coverage for the registry validator is already tracked by the existing Pester/Codecov pipeline.

Dependencies

Additional Context

  • Validator script: scripts/linting/Validate-ArtifactRegistry.ps1
  • Existing tests: scripts/tests/linting/Validate-ArtifactRegistry.Tests.ps1
  • Test fixtures: scripts/tests/Fixtures/ArtifactRegistry/
  • PR validation workflow: .github/workflows/pr-validation.yml
  • Pattern reference: .github/workflows/frontmatter-validation.yml
  • Registry file: .github/ai-artifacts-registry.json

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestinfrastructureRepository infrastructure and tooling

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions