Summary

Deep evaluation of the Codecov integration for both Pester (PowerShell) and pytest (Python/Hypothesis) pipelines reveals several gaps that reduce accuracy and independent visibility of coverage reporting.

Confirmed Gaps

1. Missing pester project status in codecov.yml

Severity: Medium-High

The pester flag is defined with paths and carryforward, and pester-tests.yml correctly uploads logs/coverage.xml with flags: pester. However, coverage.status.project has no pester entry — only default and pytest:

# Current: only pytest has a flag-specific project status
coverage:
  status:
    project:
      default:
        target: auto
        threshold: 1%
      pytest:
        target: auto
        threshold: 1%
        flags:
          - pytest
      # ❌ No pester entry — coverage uploads but has no independent PR status check

Impact: Pester coverage regresses silently. It only contributes to the default aggregate status, where regressions in PowerShell scripts can be masked by Python coverage gains (and vice versa).

Fix: Add a pester project status entry:

coverage:
  status:
    project:
      pester:
        target: auto
        threshold: 1%
        flags:
          - pester

2. --cov=. overrides [tool.coverage.run] source in pytest workflow

Severity: Medium

pytest-tests.yml runs:

uv run pytest --cov=. --cov-report=xml:coverage.xml --cov-report=term-missing -v

The --cov=. CLI argument overrides the source = ["scripts"] setting in pyproject.toml:

[tool.coverage.run]
source = ["scripts"]

This means coverage is measured against all Python files in the working directory (including tests, conftest, strategies modules), not just the intended scripts/ source. The fail_under = 85 check runs against this inflated scope.

The Codecov pytest flag filters server-side via paths: [".github/skills/**/scripts/**"], so Codecov's displayed metric may be correct — but the local CI coverage threshold applies to the wrong scope.

Fix: Change --cov=. to --cov so coverage.py reads the source from pyproject.toml:

uv run pytest --cov --cov-report=xml:coverage.xml --cov-report=term-missing -v

3. No flag-specific patch coverage

Severity: Low-Medium

coverage.status.patch only has a default entry:

patch:
  default:
    target: 80%
    informational: true

There are no per-flag (pester, pytest) patch coverage statuses. If one ecosystem has strong patch coverage and the other has none, the aggregate still passes.

Fix: Add flag-specific patch entries:

patch:
  pester:
    target: 80%
    informational: true
    flags:
      - pester
  pytest:
    target: 80%
    informational: true
    flags:
      - pytest

4. No scheduled baseline coverage runs

Severity: Low

Both flags use carryforward: true but coverage only runs on PRs via pr-validation.yml. The weekly-validation.yml workflow runs only ms.date freshness checks. Direct merges to main (rare but possible) produce no coverage data, and carryforward values grow stale over time.

Fix: Extend weekly-validation.yml with three new jobs:

• pester-coverage — Calls pester-tests.yml with soft-fail: true, changed-files-only: false, and code-coverage: true to upload a full Pester baseline.
• discover-python-projects — Finds all pyproject.toml files and outputs a matrix of Python project directories (same logic as pr-validation.yml).
• pytest-coverage — Matrix job calling pytest-tests.yml for each discovered Python project with soft-fail: true and changed-files-only: false to upload full pytest baselines.

All coverage jobs use id-token: write for Codecov OIDC authentication.

Investigation Notes

What works correctly

• pester-tests.yml uploads JaCoCo XML to Codecov with flags: pester and use_oidc: true ✅
• pytest-tests.yml uploads Cobertura XML to Codecov with flags: pytest and use_oidc: true ✅
• pr-validation.yml wires both workflows with coverage enabled ✅
• Pester config (pester.config.ps1) correctly resolves coverage paths from scripts and skills directories ✅
• Hypothesis-based tests run through pytest-cov and contribute coverage data normally ✅
• Both flags have carryforward: true for stable PR status when only one ecosystem changes ✅

Files involved

Acceptance Criteria

• codecov.yml has independent project status entries for both pester and pytest flags
• pytest-tests.yml coverage source matches pyproject.toml [tool.coverage.run] source
• Flag-specific patch coverage entries added for pester and pytest
• weekly-validation.yml runs Pester and pytest coverage on a weekly schedule to keep carryforward data fresh