Summary

The repository has no CI workflow for running Python tests or collecting Python code coverage. Existing test infrastructure covers PowerShell (Pester) with Codecov integration but has no equivalent for pytest. This is the most complex issue in the Python build system series, touching a new workflow, the orchestrator, and Codecov configuration. It satisfies the OSSF Best Practices automated_integration_testing MUST criterion for silver badge compliance.

Context

The current testing architecture runs Pester tests via npm run test:ps in CI, with coverage results uploaded to Codecov using a pester flag. The codecov.yml configuration file at the repository root defines coverage flags and reporting thresholds. Adding Python testing requires a parallel structure: a new reusable workflow for pytest, orchestrator integration, and a new Codecov coverage flag.

Issue #634 (closed, v3.0.0) previously expanded CI coverage including Pester testing and coverage reporting. This issue extends that pattern to Python.

This workflow has HIGH OSSF impact. Without automated testing for Python code, the automated_integration_testing MUST criterion is at risk when Python becomes a supported language.

Architecture: Multi-Skill Working Directory

The reusable workflow should accept a working-directory input parameter, following the same pattern as the Python linting workflow (#889):

on:
  workflow_call:
    inputs:
      working-directory:
        type: string
        required: false
        default: ".github/skills/experimental/powerpoint"
      soft-fail:
        type: boolean
        required: false
        default: false
      changed-files-only:
        type: boolean
        required: false
        default: false

The orchestrator calls this workflow once per Python skill directory. Each skill runs uv sync --locked and uv run pytest independently, with coverage uploaded per skill. The --locked flag ensures CI uses the committed lockfile without re-resolving, preventing supply chain attacks from updating dependencies between commit and CI run.

Testing and Mocking Framework

Standardize on the following dev dependencies for all Python skills:

[dependency-groups]
dev = ["pytest>=8.0", "ruff>=0.4", "pytest-mock>=3.14"]

• pytest-mock provides the mocker fixture, which wraps unittest.mock with automatic cleanup and concise mocker.patch() syntax. Use for call assertions, spies, and autospec.
• monkeypatch (built-in pytest fixture) is sufficient for simple attribute and environment variable patching where call assertions are not needed.
• Avoid direct unittest.mock usage: pytest-mock wraps the same API with better fixture integration.

Changes Required

The new workflow should:

• Accept soft-fail, changed-files-only, and working-directory inputs
• Install uv and run uv sync --locked in the specified working directory
• Run pytest with coverage collection (--cov flag or coverage.py integration)
• Generate coverage report in a format Codecov accepts (XML/JSON)
• Upload coverage to Codecov with the pytest flag
• Output test results in a structured format

Codecov configuration changes:

coverage:
  status:
    project:
      pester:
        flags:
          - pester
      pytest:
        flags:
          - pytest
flags:
  pester:
    paths:
      - scripts/
  pytest:
    paths:
      - .github/skills/**/scripts/**
    carryforward: true

The glob **/scripts/** covers all skill directories. With carryforward enabled, coverage from skills whose tests did not run in a given commit is preserved from the previous upload.

Acceptance Criteria

• .github/workflows/pytest-tests.yml exists as a reusable workflow
• Workflow accepts soft-fail, changed-files-only, and working-directory standard inputs
• pytest runs against Python test files and reports results
• pytest-mock is included in dev dependencies for Python skills
• Coverage data is collected during test execution
• Coverage report is uploaded to Codecov with a pytest flag
• pr-validation.yml calls the new workflow as a job in the validation matrix
• codecov.yml defines a pytest coverage flag with appropriate path filters and carryforward
• Coverage reporting does not interfere with existing Pester coverage
• Workflow passes when no Python test files exist in changed paths
• Test failures cause the workflow to fail (unless soft-fail is enabled)
• Adding a new Python skill requires only a new orchestrator job entry, no workflow changes

OSSF Impact

Dependencies

• Benefits from uv being pre-installed (feat(devcontainer): Add Python development extensions and uv package manager #887, feat(ci): Add uv and Python package sync to copilot-setup-steps #888) but can install uv within the workflow
• Benefits from npm scripts (feat(scripts): Add npm run scripts for Python linting and testing #886) but can invoke pytest directly
• Python linting workflow (feat(ci): Add Python linting CI workflow with ruff #889) should ideally land first as a simpler CI change

Related

• PR feat(skills): add PowerPoint automation skill with YAML-driven deck generation #868 — PowerPoint automation skill (first Python skill)
• feat(ci): Add Python linting CI workflow with ruff #889 — Python linting CI workflow with ruff
• feat(scripts): Add npm run scripts for Python linting and testing #886 — npm scripts for Python
• feat(ci): Add Python to CodeQL analysis language matrix #884 — CodeQL Python integration
• feat(skills): Mandate unit testing, extend CI coverage, and document language support process #634 — Prior CI coverage expansion including Pester (closed)