Skip to content

feat(scripts): add structured JSON log output to Validate-Collections.ps1#1431

Merged
katriendg merged 1 commit into
microsoft:mainfrom
MauroDruwel:feat/992-collections-json-output
May 3, 2026
Merged

feat(scripts): add structured JSON log output to Validate-Collections.ps1#1431
katriendg merged 1 commit into
microsoft:mainfrom
MauroDruwel:feat/992-collections-json-output

Conversation

@MauroDruwel

@MauroDruwel MauroDruwel commented Apr 23, 2026

Copy link
Copy Markdown
Contributor

Pull Request

Description

Validate-Collections.ps1 was the only collections-facing linting script that did not write structured results to logs/. All other pipeline scripts persist JSON reports after each run; this gap meant collection validation results were lost at terminal close, blocking any tooling or CI step that reads from logs/.

Changes made:

  • Added -OutputPath parameter to Validate-Collections.ps1 (default: logs/collection-validation-results.json).
  • Refactored $fileErrors from plain strings to typed hashtables (ErrorType + Message), enabling specific error codes in JSON output: MissingRequiredField, InvalidIdFormat, DuplicateCollectionId, InvalidCollectionMaturity, RepoSpecificPath, PathNotFound, MissingSuffix, MissingItemKind, InvalidMaturity, IntraCollectionDuplicate.
  • Introduced inner helper Add-ValidationResult that accumulates a typed results list alongside every existing Write-Host call (console output unchanged).
  • Added Export-CollectionValidationReport function for path resolution, directory creation, and JSON serialization.
  • Moved Create logs directory step in plugin-validation.yml to before Validate collection metadata to make workflow intent explicit.
  • Expanded Pester test coverage: updated ErrorType assertion to specific type, added severity differentiation, directory creation, multiple distinct ErrorType values in one run, and clean-run Results key presence tests.

Output schema:

{
  "Timestamp": "2026-04-23T17:00:00.000Z",
  "TotalCollections": 14,
  "ErrorCount": 0,
  "Results": [
    {
      "Collection": "hve-core",
      "Severity": "Warning",
      "ErrorType": "MissingCompanionCollectionMd",
      "Message": "missing companion 'hve-core.collection.md'"
    }
  ]
}

Related Issue(s)

Closes #992.

Type of Change

Select all that apply:

Code & Documentation:

  • Bug fix (non-breaking change fixing an issue)
  • New feature (non-breaking change adding functionality)
  • Breaking change (fix or feature causing existing functionality to change)
  • Documentation update

Infrastructure & Configuration:

  • GitHub Actions workflow
  • Linting configuration (markdown, PowerShell, etc.)
  • Security configuration
  • DevContainer configuration
  • Dependency update

AI Artifacts:

  • Reviewed contribution with prompt-builder agent and addressed all feedback
  • Copilot instructions (.github/instructions/*.instructions.md)
  • Copilot prompt (.github/prompts/*.prompt.md)
  • Copilot agent (.github/agents/*.agent.md)
  • Copilot skill (.github/skills/*/SKILL.md)

Other:

  • Script/automation (.ps1, .sh, .py)
  • Other (please describe):

Testing

Command Result
npm run lint:collections-metadata ✅ pass, writes logs/collection-validation-results.json
npm run lint:ps ✅ pass
npm run test:ps -- -TestPath scripts/tests/collections/ ✅ pass, 6 tests (2 pre-existing + 4 new), no regressions

Baseline runs confirmed pre-existing test suite intact; post-change runs confirmed new tests pass and JSON file is created with correct schema.

Checklist

Required Checks

  • Documentation is updated (if applicable)
  • Files follow existing naming conventions
  • Changes are backwards compatible (if applicable)
  • Tests added for new functionality (if applicable)

AI Artifact Contributions

  • Used /prompt-analyze to review contribution
  • Addressed all feedback from prompt-builder review
  • Verified contribution follows common standards and type-specific requirements

Required Automated Checks

The following validation commands must pass before merging:

  • Markdown linting: npm run lint:md
  • Spell checking: npm run spell-check
  • Frontmatter validation: npm run lint:frontmatter
  • Skill structure validation: npm run validate:skills
  • Link validation: npm run lint:md-links
  • PowerShell analysis: npm run lint:ps
  • Plugin freshness: npm run plugin:generate
  • Docusaurus tests: npm run docs:test

Security Considerations

  • This PR does not contain any sensitive or NDA information
  • Any new dependencies have been reviewed for security issues
  • Security-related scripts follow the principle of least privilege

Additional Notes

This PR was rebased on top of the main branch which includes #1430 (structured JSON output for Validate-Marketplace.ps1). The merge conflict in package.json was resolved by combining both -OutputPath flags: lint:collections-metadata and lint:marketplace both now pass their respective output paths.

@MauroDruwel MauroDruwel marked this pull request as ready for review April 23, 2026 19:13
@MauroDruwel MauroDruwel requested a review from a team as a code owner April 23, 2026 19:13

@katriendg katriendg left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for another one @MauroDruwel!
We have some changes requested to finetune a few things, looking very good overall.

Note here as well if you could update your PR description to match the PULL_REQUEST_TEMPLATE.md? Using the prompt is easiest.

Comment thread scripts/collections/Validate-Collections.ps1
Comment thread scripts/collections/Validate-Collections.ps1 Outdated
Comment thread scripts/tests/collections/Validate-Collections.Tests.ps1
@MauroDruwel MauroDruwel force-pushed the feat/992-collections-json-output branch 2 times, most recently from 82988c7 to acf886c Compare April 30, 2026 20:19
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@MauroDruwel MauroDruwel force-pushed the feat/992-collections-json-output branch from acf886c to be29d6f Compare April 30, 2026 20:24
@MauroDruwel

MauroDruwel commented Apr 30, 2026

Copy link
Copy Markdown
Contributor Author

All three review comments addressed @katriendg 😊:

  • Workflow step order: Create logs directory moved before Validate collection metadata in plugin-validation.yml
  • Typed ErrorType values: Replaced generic CollectionValidationError with 10 specific identifiers (e.g. MissingRequiredField, OrphanArtifact, CanonicalOnlyArtifact, etc.)
  • Test coverage: Fixed existing assertion + added 4 new tests covering JSON schema, severity differentiation, directory auto-creation, and multiple ErrorType emission

Also rebased onto main (resolved conflicts in package.json and Validate-Collections.ps1) and updated PR description to match the template.

@MauroDruwel MauroDruwel requested a review from katriendg April 30, 2026 20:28
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.82609% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 87.43%. Comparing base (5bedf80) to head (be29d6f).

Files with missing lines Patch % Lines
scripts/collections/Validate-Collections.ps1 97.82% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1431      +/-   ##
==========================================
+ Coverage   87.41%   87.43%   +0.02%     
==========================================
  Files          68       68              
  Lines       10302    10335      +33     
==========================================
+ Hits         9005     9036      +31     
- Misses       1297     1299       +2     
Flag Coverage Δ
pester 84.79% <97.82%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
scripts/collections/Validate-Collections.ps1 93.83% <97.82%> (+0.53%) ⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@katriendg katriendg left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you so much @MauroDruwel. This is good for merging in!

@katriendg katriendg merged commit 57ea279 into microsoft:main May 3, 2026
58 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add JSON log output to Validate-Collections.ps1

4 participants