Add dependabot grpc migration workflow#573
Add dependabot grpc migration workflow#573llucax wants to merge 16 commits intofrequenz-floss:v0.x.xfrom
Conversation
|
This is a draft because it is still missing the workflow, but I want to do a test with the microgrid-api. |
The upgrade script is temporarily pointing to my fork, this will be fixed when frequenz-floss/frequenz-repo-config-python#573 is merged.
|
Tested in frequenz-floss/frequenz-api-microgrid#405. |
|
Not enabling auto-merge as I will need to update the script URL once the PR is approved, then I will use a hash in this repo. |
There was a problem hiding this comment.
Pull request overview
This PR introduces an automated migration path for Dependabot gRPC/protobuf updates in generated API repositories, ensuring runtime dependency floors in pyproject.toml stay in sync with build-time pins (and updates required checks / docs accordingly).
Changes:
- Add a
grpc-migration.yamlworkflow (API repos) plus new Dependabot groups (grpc-compatible,grpcio-major,protobuf-major) and updateauto-dependabot.yamlto skip those PRs. - Add the
dependabot-grpc-fixer.pyscript and a comprehensive fixture-based test suite for it. - Extend repo migration support (
cookiecutter/migrate.py), update docs/release notes, and refresh golden fixtures / ruleset required checks.
Reviewed changes
Copilot reviewed 89 out of 89 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| tests_golden/integration/test_cookiecutter_generation/model/frequenz-model-test/.github/workflows/auto-dependabot.yaml | Golden fixture: skip new grpc/protobuf Dependabot groups in auto-merge workflow |
| tests_golden/integration/test_cookiecutter_generation/model-proprietary/frequenz-model-test/.github/workflows/auto-dependabot.yaml | Golden fixture: skip new grpc/protobuf Dependabot groups in auto-merge workflow |
| tests_golden/integration/test_cookiecutter_generation/lib/frequenz-test-python/.github/workflows/auto-dependabot.yaml | Golden fixture: skip new grpc/protobuf Dependabot groups in auto-merge workflow |
| tests_golden/integration/test_cookiecutter_generation/lib-proprietary/frequenz-test-python/.github/workflows/auto-dependabot.yaml | Golden fixture: skip new grpc/protobuf Dependabot groups in auto-merge workflow |
| tests_golden/integration/test_cookiecutter_generation/app/frequenz-app-test/.github/workflows/auto-dependabot.yaml | Golden fixture: skip new grpc/protobuf Dependabot groups in auto-merge workflow |
| tests_golden/integration/test_cookiecutter_generation/app-proprietary/frequenz-app-test/.github/workflows/auto-dependabot.yaml | Golden fixture: skip new grpc/protobuf Dependabot groups in auto-merge workflow |
| tests_golden/integration/test_cookiecutter_generation/api/frequenz-api-test/.github/workflows/grpc-migration.yaml | Golden fixture: new gRPC migration workflow |
| tests_golden/integration/test_cookiecutter_generation/api/frequenz-api-test/.github/workflows/auto-dependabot.yaml | Golden fixture: skip grpc/protobuf groups in auto-merge workflow |
| tests_golden/integration/test_cookiecutter_generation/api/frequenz-api-test/.github/dependabot.yml | Golden fixture: add grpc/protobuf groups and exclusions |
| tests_golden/integration/test_cookiecutter_generation/api-proprietary/frequenz-api-test/.github/workflows/grpc-migration.yaml | Golden fixture: new gRPC migration workflow |
| tests_golden/integration/test_cookiecutter_generation/api-proprietary/frequenz-api-test/.github/workflows/auto-dependabot.yaml | Golden fixture: skip grpc/protobuf groups in auto-merge workflow |
| tests_golden/integration/test_cookiecutter_generation/api-proprietary/frequenz-api-test/.github/dependabot.yml | Golden fixture: add grpc/protobuf groups and exclusions |
| tests_golden/integration/test_cookiecutter_generation/actor/frequenz-actor-test/.github/workflows/auto-dependabot.yaml | Golden fixture: skip grpc/protobuf groups in auto-merge workflow |
| tests_golden/integration/test_cookiecutter_generation/actor-proprietary/frequenz-actor-test/.github/workflows/auto-dependabot.yaml | Golden fixture: skip grpc/protobuf groups in auto-merge workflow |
| tests/integration/AGENTS.md | Integration-test subtree guide |
| tests/cookiecutter/scripts/test_dependabot-grpc-fixer.py | Unit/integration tests for the gRPC fixer script (incl. golden-based drift test) |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/updated_dependencies_not_array/input/pyproject.toml | Fixture input: invalid metadata shape (non-array) |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/updated_dependencies_not_array/input/dependabot-metadata.json | Fixture input: invalid metadata shape (non-array) |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/updated_dependencies_not_array/expected/stderr.contains.txt | Fixture expected stderr substring |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/updated_dependencies_not_array/expected/pyproject.toml | Fixture expected file output (unchanged) |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/update_runtime_bounds/input/pyproject.toml | Fixture input: update runtime floors and upper bounds |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/update_runtime_bounds/input/dependabot-metadata.json | Fixture input: realistic metadata with grpc/protobuf updates |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/update_runtime_bounds/expected/stdout.txt | Fixture expected stdout |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/update_runtime_bounds/expected/pyproject.toml | Fixture expected rewritten pyproject.toml |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/root_scoped_metadata_only/input/pyproject.toml | Fixture input: only root/pip entries should apply |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/root_scoped_metadata_only/input/dependabot-metadata.json | Fixture input: mixed directories/ecosystems |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/root_scoped_metadata_only/expected/stdout.txt | Fixture expected stdout |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/root_scoped_metadata_only/expected/pyproject.toml | Fixture expected rewritten pyproject.toml |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/prerelease_version/input/pyproject.toml | Fixture input: prerelease-like versions |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/prerelease_version/input/dependabot-metadata.json | Fixture input: prerelease-like versions |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/prerelease_version/expected/stdout.txt | Fixture expected stdout |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/prerelease_version/expected/pyproject.toml | Fixture expected rewritten pyproject.toml |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/non_object_dependency_item/input/pyproject.toml | Fixture input: non-object items in metadata |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/non_object_dependency_item/input/dependabot-metadata.json | Fixture input: non-object items in metadata |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/non_object_dependency_item/expected/stderr.contains.txt | Fixture expected stderr substring |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/non_object_dependency_item/expected/pyproject.toml | Fixture expected file output (unchanged) |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/no_tracked_metadata/input/pyproject.toml | Fixture input: metadata has no tracked deps |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/no_tracked_metadata/input/dependabot-metadata.json | Fixture input: metadata has no tracked deps |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/no_tracked_metadata/expected/stderr.contains.txt | Fixture expected stderr substring |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/no_tracked_metadata/expected/pyproject.toml | Fixture expected file output (unchanged) |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_runtime_constraint/input/pyproject.toml | Fixture input: missing runtime constraint in pyproject.toml |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_runtime_constraint/input/dependabot-metadata.json | Fixture input: update requiring missing constraint |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_runtime_constraint/expected/stderr.contains.txt | Fixture expected stderr substring |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_runtime_constraint/expected/pyproject.toml | Fixture expected file output (unchanged) |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_pyproject/input/dependabot-metadata.json | Fixture input: missing pyproject.toml |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_pyproject/expected/stderr.contains.txt | Fixture expected stderr substring |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_new_version_for_tracked_dep/input/pyproject.toml | Fixture input: tracked dep missing newVersion |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_new_version_for_tracked_dep/input/dependabot-metadata.json | Fixture input: tracked dep missing newVersion |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_new_version_for_tracked_dep/expected/stderr.contains.txt | Fixture expected stderr substring |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_new_version_for_tracked_dep/expected/pyproject.toml | Fixture expected file output (unchanged) |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/invalid_updated_dependencies_json/input/pyproject.toml | Fixture input: invalid JSON |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/invalid_updated_dependencies_json/input/dependabot-metadata.json | Fixture input: invalid JSON |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/invalid_updated_dependencies_json/expected/stderr.contains.txt | Fixture expected stderr substring |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/invalid_updated_dependencies_json/expected/pyproject.toml | Fixture expected file output (unchanged) |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/invalid_new_version_format/input/pyproject.toml | Fixture input: unparseable version |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/invalid_new_version_format/input/dependabot-metadata.json | Fixture input: unparseable version |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/invalid_new_version_format/expected/stderr.contains.txt | Fixture expected stderr substring |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/invalid_new_version_format/expected/pyproject.toml | Fixture expected file output (unchanged) |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/ignore_non_runtime_updates/input/pyproject.toml | Fixture input: only build-time updates should no-op |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/ignore_non_runtime_updates/input/dependabot-metadata.json | Fixture input: only build-time updates should no-op |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/ignore_non_runtime_updates/expected/stdout.txt | Fixture expected stdout |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/ignore_non_runtime_updates/expected/pyproject.toml | Fixture expected file output (unchanged) |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/empty_updated_dependencies_json/input/pyproject.toml | Fixture input: empty metadata |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/empty_updated_dependencies_json/expected/stderr.contains.txt | Fixture expected stderr substring |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/empty_updated_dependencies_json/expected/pyproject.toml | Fixture expected file output (unchanged) |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/downgrade_refused/input/pyproject.toml | Fixture input: downgrade scenario |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/downgrade_refused/input/dependabot-metadata.json | Fixture input: downgrade scenario |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/downgrade_refused/expected/stderr.contains.txt | Fixture expected stderr substring |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/downgrade_refused/expected/pyproject.toml | Fixture expected file output (unchanged) |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/double_runtime_constraint/input/pyproject.toml | Fixture input: duplicate runtime constraints |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/double_runtime_constraint/input/dependabot-metadata.json | Fixture input: duplicate runtime constraints |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/double_runtime_constraint/expected/stderr.contains.txt | Fixture expected stderr substring |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/double_runtime_constraint/expected/pyproject.toml | Fixture expected file output (unchanged) |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/conflicting_root_metadata/input/pyproject.toml | Fixture input: conflicting root metadata entries |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/conflicting_root_metadata/input/dependabot-metadata.json | Fixture input: conflicting root metadata entries |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/conflicting_root_metadata/expected/stderr.contains.txt | Fixture expected stderr substring |
| tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/conflicting_root_metadata/expected/pyproject.toml | Fixture expected file output (unchanged) |
| src/frequenz/repo/config/AGENTS.md | Package subtree guide |
| github-rulesets/python/Protect version branches.json | Add required check context for gRPC migration job |
| docs/user-guide/advanced-usage.md | Document gRPC migration workflow usage and requirements |
| cookiecutter/{{cookiecutter.github_repo_name}}/.github/workflows/grpc-migration.yaml | New template workflow for gRPC/protobuf runtime floor migration |
| cookiecutter/{{cookiecutter.github_repo_name}}/.github/workflows/auto-dependabot.yaml | Template: skip grpc/protobuf Dependabot group PRs from auto-merge workflow |
| cookiecutter/{{cookiecutter.github_repo_name}}/.github/dependabot.yml | Template: add grpc/protobuf groups + exclude from patch/minor groups (API only) |
| cookiecutter/scripts/dependabot-grpc-fixer.py | New migration script to rewrite runtime bounds based on Dependabot metadata |
| cookiecutter/migrate.py | Add migration step to install workflow + update dependabot config + ruleset |
| cookiecutter/hooks/post_gen_project.py | Post-generation cleanup to remove empty grpc workflow file for non-API repos |
| cookiecutter/AGENTS.md | Cookiecutter subtree guide |
| RELEASE_NOTES.md | Release note entry for new gRPC migration workflow |
| AGENTS.md | Update repo-level AGENTS knowledge base |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -0,0 +1,359 @@ | |||
| """Tests for the dependabot_gprc_fixer package.""" | |||
| # These need a migration script to fix dependabot missing updating | ||
| # the runtime dependencies |
Use a much more concise style and split information to sub-files to avoid filling up the context window unnecessarily. Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
This script will be used to automatically fix wrong dependabot updates for grouped grpcio*/protobuf dependencies. These dependencies have 2 parts for API projects, a build-time part and a run-time part. The build-time dependencies need to be older than the runtime dependencies, and dependabot only bumps the build-time dependencies, which leaves repositories in a bad state. On top of that, runtime dependencies have special compatibility rules for protobuf. Two consecutive major versions are guaranteed to be compatible. This new script bumps the minimum runtime dependencies to match the dependabot-updated build-time dependencies and make sure the supported major versions are also bumped appropriately, including the comments. Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
This will be used to add some unit tests for the internals of the script, which can't be easily imported with just "import" because it is not intended to be used as a module. Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
This commit uses the exiting generated API golden tests (cookiecutter generate templates) to validate the script works on the current template. This ensures that things continue to work even if the template pyproject.toml file is changed and uses a different format to specify the grpc/protobuf dependencies. Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
Golden test fixtures for the new dependabot-grpc-fixer.py script will be stored in tests/cookiecutter/scripts/fixtures/fixer_cases with one sub-directory per test case, where each test case is composed of the expected inputs and outputs. Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
Add many test cases to cover for most happy and error paths. Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
All other tests are calling the fixer.main() function (or other internal functions), but none is really testing the fixer works as a script, and its arguments work as expected. Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
The dependabot-grpc-fixer is meant to be exercised as a script, not through a dynamically imported module. Run the remaining tests via runpy so they cover the supported entrypoint instead of the script internals. Drop the replace_range() unit test and the shared module loader to reduce coupling to implementation details while keeping the existing script-level coverage. Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
Keep the generic Python version-branch protection usable by non-API repositories, and add a separate API-specific variant that also requires the gRPC runtime-floor migration check. Update the GitHub setup docs to make API repositories import the API-specific ruleset instead of the generic Python one. Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
For API repositories the existing monolithic 'grpc' group is replaced
with three groups so the upcoming grpc-migration workflow can target
them precisely:
- 'grpc-compatible' for patch and minor updates of grpcio,
grpcio-tools and protobuf together.
- 'grpcio-major' for major updates of grpcio and grpcio-tools.
- 'protobuf-major' for major updates of protobuf.
The split is needed because grpcio releases historically lag behind
protobuf, so a single 'grpc' group for major updates would block one
on the other.
The grpc/protobuf packages are also excluded from the patch and minor
groups, as they need to be handled by the dedicated grpc-migration
workflow that syncs the runtime '>=' floors with the new build pins.
Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
The 'auto-dependabot' workflow now skips PRs from the three new grpc groups so they are exclusively handled by the upcoming new migration workflow. Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
The new 'grpc-migration' workflow handles Dependabot PRs from the three new grpc/protobuf groups by syncing the runtime '>=' floors in 'pyproject.toml' with the new build-time pins. The workflow is only rendered for API projects (the only template type that pins grpc/protobuf as build-time dependencies), so the post-generation hook removes the leftover empty file for the other project types. Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
Add a 'migrate_grpc_workflow_setup' step that brings existing API projects in line with the new grpc/protobuf handling. The step is a no-op for non-API projects. For API projects it: * Updates '.github/dependabot.yml' to use the new 'grpc-compatible'/'grpcio-major'/'protobuf-major' Dependabot groups, replacing any verbatim old 'grpc:' group and inserting the three groups otherwise. It also adds the grpc/protobuf packages to the 'patch' and 'minor' groups' 'exclude-patterns'. Customized 'grpc:' groups are left untouched and reported as a manual step. * Updates '.github/workflows/auto-dependabot.yaml' to skip PRs from the three new grpc Dependabot groups. * Installs (or replaces) '.github/workflows/grpc-migration.yaml' with the latest workflow contents. Existing experimental versions shipped to some repositories are unconditionally replaced. * Adds the 'Fix gRPC/protobuf runtime floors' required status check to the 'Protect version branches' GitHub ruleset via the 'gh' CLI. Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
Add a new 'gRPC migration workflow' section to the advanced usage guide modeled after the 'Black formatting migration workflow' section. The new section explains: * Why the workflow exists (grpc/protobuf cross-version guarantee versus Dependabot's build-only updates). * That the workflow ships with the cookiecutter template for API projects and uses the 'dependabot-grpc-fixer.py' script from this repository. * How to recreate the caller workflow, dependabot groups and auto-dependabot interaction in API projects that were not generated from the template. Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
Dependabot fails to update grpc/protobuf dependencies correctly, as it limits itself to update build dependencies, but leave the runtime dependencies untouched.
This doesn't work because protobuf/grpc need to be generated with a version that is equals or older to the runtime version, so if we only bump the build dependency, a project might still depend on an older version at runtime, and we should not allow that.
This PR adds an auto-migration workflow to fix this, while also ensuring the major version for protobuf is bumped accordingly when a major version bump occurs.