Skip to content

Add dependabot grpc migration workflow#573

Open
llucax wants to merge 16 commits intofrequenz-floss:v0.x.xfrom
llucax:fix-grpc-group
Open

Add dependabot grpc migration workflow#573
llucax wants to merge 16 commits intofrequenz-floss:v0.x.xfrom
llucax:fix-grpc-group

Conversation

@llucax
Copy link
Copy Markdown
Contributor

@llucax llucax commented Apr 28, 2026

Dependabot fails to update grpc/protobuf dependencies correctly, as it limits itself to update build dependencies, but leave the runtime dependencies untouched.

This doesn't work because protobuf/grpc need to be generated with a version that is equals or older to the runtime version, so if we only bump the build dependency, a project might still depend on an older version at runtime, and we should not allow that.

This PR adds an auto-migration workflow to fix this, while also ensuring the major version for protobuf is bumped accordingly when a major version bump occurs.

@github-actions github-actions Bot added part:tests Affects the unit, integration and performance (benchmarks) tests part:cookiecutter Affects the generation of projects using cookiecutter labels Apr 28, 2026
@llucax
Copy link
Copy Markdown
Contributor Author

llucax commented Apr 28, 2026

This is a draft because it is still missing the workflow, but I want to do a test with the microgrid-api.

@github-actions github-actions Bot added the part:template Affects the cookiecutter template files label Apr 28, 2026
@llucax llucax marked this pull request as ready for review April 28, 2026 14:05
@llucax llucax requested a review from a team as a code owner April 28, 2026 14:05
@llucax llucax requested review from simonvoelcker and removed request for a team April 28, 2026 14:05
tiyash-basu-frequenz pushed a commit to tiyash-basu-frequenz/frequenz-api-microgrid that referenced this pull request Apr 28, 2026
The upgrade script is temporarily pointing to my fork, this will be
fixed when
frequenz-floss/frequenz-repo-config-python#573
is merged.
@llucax llucax requested a review from Marenz April 29, 2026 12:44
@llucax
Copy link
Copy Markdown
Contributor Author

llucax commented Apr 29, 2026

@llucax
Copy link
Copy Markdown
Contributor Author

llucax commented Apr 29, 2026

Not enabling auto-merge as I will need to update the script URL once the PR is approved, then I will use a hash in this repo.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces an automated migration path for Dependabot gRPC/protobuf updates in generated API repositories, ensuring runtime dependency floors in pyproject.toml stay in sync with build-time pins (and updates required checks / docs accordingly).

Changes:

  • Add a grpc-migration.yaml workflow (API repos) plus new Dependabot groups (grpc-compatible, grpcio-major, protobuf-major) and update auto-dependabot.yaml to skip those PRs.
  • Add the dependabot-grpc-fixer.py script and a comprehensive fixture-based test suite for it.
  • Extend repo migration support (cookiecutter/migrate.py), update docs/release notes, and refresh golden fixtures / ruleset required checks.

Reviewed changes

Copilot reviewed 89 out of 89 changed files in this pull request and generated 8 comments.

Show a summary per file
File Description
tests_golden/integration/test_cookiecutter_generation/model/frequenz-model-test/.github/workflows/auto-dependabot.yaml Golden fixture: skip new grpc/protobuf Dependabot groups in auto-merge workflow
tests_golden/integration/test_cookiecutter_generation/model-proprietary/frequenz-model-test/.github/workflows/auto-dependabot.yaml Golden fixture: skip new grpc/protobuf Dependabot groups in auto-merge workflow
tests_golden/integration/test_cookiecutter_generation/lib/frequenz-test-python/.github/workflows/auto-dependabot.yaml Golden fixture: skip new grpc/protobuf Dependabot groups in auto-merge workflow
tests_golden/integration/test_cookiecutter_generation/lib-proprietary/frequenz-test-python/.github/workflows/auto-dependabot.yaml Golden fixture: skip new grpc/protobuf Dependabot groups in auto-merge workflow
tests_golden/integration/test_cookiecutter_generation/app/frequenz-app-test/.github/workflows/auto-dependabot.yaml Golden fixture: skip new grpc/protobuf Dependabot groups in auto-merge workflow
tests_golden/integration/test_cookiecutter_generation/app-proprietary/frequenz-app-test/.github/workflows/auto-dependabot.yaml Golden fixture: skip new grpc/protobuf Dependabot groups in auto-merge workflow
tests_golden/integration/test_cookiecutter_generation/api/frequenz-api-test/.github/workflows/grpc-migration.yaml Golden fixture: new gRPC migration workflow
tests_golden/integration/test_cookiecutter_generation/api/frequenz-api-test/.github/workflows/auto-dependabot.yaml Golden fixture: skip grpc/protobuf groups in auto-merge workflow
tests_golden/integration/test_cookiecutter_generation/api/frequenz-api-test/.github/dependabot.yml Golden fixture: add grpc/protobuf groups and exclusions
tests_golden/integration/test_cookiecutter_generation/api-proprietary/frequenz-api-test/.github/workflows/grpc-migration.yaml Golden fixture: new gRPC migration workflow
tests_golden/integration/test_cookiecutter_generation/api-proprietary/frequenz-api-test/.github/workflows/auto-dependabot.yaml Golden fixture: skip grpc/protobuf groups in auto-merge workflow
tests_golden/integration/test_cookiecutter_generation/api-proprietary/frequenz-api-test/.github/dependabot.yml Golden fixture: add grpc/protobuf groups and exclusions
tests_golden/integration/test_cookiecutter_generation/actor/frequenz-actor-test/.github/workflows/auto-dependabot.yaml Golden fixture: skip grpc/protobuf groups in auto-merge workflow
tests_golden/integration/test_cookiecutter_generation/actor-proprietary/frequenz-actor-test/.github/workflows/auto-dependabot.yaml Golden fixture: skip grpc/protobuf groups in auto-merge workflow
tests/integration/AGENTS.md Integration-test subtree guide
tests/cookiecutter/scripts/test_dependabot-grpc-fixer.py Unit/integration tests for the gRPC fixer script (incl. golden-based drift test)
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/updated_dependencies_not_array/input/pyproject.toml Fixture input: invalid metadata shape (non-array)
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/updated_dependencies_not_array/input/dependabot-metadata.json Fixture input: invalid metadata shape (non-array)
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/updated_dependencies_not_array/expected/stderr.contains.txt Fixture expected stderr substring
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/updated_dependencies_not_array/expected/pyproject.toml Fixture expected file output (unchanged)
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/update_runtime_bounds/input/pyproject.toml Fixture input: update runtime floors and upper bounds
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/update_runtime_bounds/input/dependabot-metadata.json Fixture input: realistic metadata with grpc/protobuf updates
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/update_runtime_bounds/expected/stdout.txt Fixture expected stdout
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/update_runtime_bounds/expected/pyproject.toml Fixture expected rewritten pyproject.toml
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/root_scoped_metadata_only/input/pyproject.toml Fixture input: only root/pip entries should apply
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/root_scoped_metadata_only/input/dependabot-metadata.json Fixture input: mixed directories/ecosystems
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/root_scoped_metadata_only/expected/stdout.txt Fixture expected stdout
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/root_scoped_metadata_only/expected/pyproject.toml Fixture expected rewritten pyproject.toml
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/prerelease_version/input/pyproject.toml Fixture input: prerelease-like versions
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/prerelease_version/input/dependabot-metadata.json Fixture input: prerelease-like versions
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/prerelease_version/expected/stdout.txt Fixture expected stdout
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/prerelease_version/expected/pyproject.toml Fixture expected rewritten pyproject.toml
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/non_object_dependency_item/input/pyproject.toml Fixture input: non-object items in metadata
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/non_object_dependency_item/input/dependabot-metadata.json Fixture input: non-object items in metadata
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/non_object_dependency_item/expected/stderr.contains.txt Fixture expected stderr substring
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/non_object_dependency_item/expected/pyproject.toml Fixture expected file output (unchanged)
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/no_tracked_metadata/input/pyproject.toml Fixture input: metadata has no tracked deps
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/no_tracked_metadata/input/dependabot-metadata.json Fixture input: metadata has no tracked deps
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/no_tracked_metadata/expected/stderr.contains.txt Fixture expected stderr substring
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/no_tracked_metadata/expected/pyproject.toml Fixture expected file output (unchanged)
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_runtime_constraint/input/pyproject.toml Fixture input: missing runtime constraint in pyproject.toml
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_runtime_constraint/input/dependabot-metadata.json Fixture input: update requiring missing constraint
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_runtime_constraint/expected/stderr.contains.txt Fixture expected stderr substring
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_runtime_constraint/expected/pyproject.toml Fixture expected file output (unchanged)
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_pyproject/input/dependabot-metadata.json Fixture input: missing pyproject.toml
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_pyproject/expected/stderr.contains.txt Fixture expected stderr substring
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_new_version_for_tracked_dep/input/pyproject.toml Fixture input: tracked dep missing newVersion
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_new_version_for_tracked_dep/input/dependabot-metadata.json Fixture input: tracked dep missing newVersion
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_new_version_for_tracked_dep/expected/stderr.contains.txt Fixture expected stderr substring
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/missing_new_version_for_tracked_dep/expected/pyproject.toml Fixture expected file output (unchanged)
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/invalid_updated_dependencies_json/input/pyproject.toml Fixture input: invalid JSON
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/invalid_updated_dependencies_json/input/dependabot-metadata.json Fixture input: invalid JSON
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/invalid_updated_dependencies_json/expected/stderr.contains.txt Fixture expected stderr substring
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/invalid_updated_dependencies_json/expected/pyproject.toml Fixture expected file output (unchanged)
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/invalid_new_version_format/input/pyproject.toml Fixture input: unparseable version
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/invalid_new_version_format/input/dependabot-metadata.json Fixture input: unparseable version
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/invalid_new_version_format/expected/stderr.contains.txt Fixture expected stderr substring
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/invalid_new_version_format/expected/pyproject.toml Fixture expected file output (unchanged)
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/ignore_non_runtime_updates/input/pyproject.toml Fixture input: only build-time updates should no-op
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/ignore_non_runtime_updates/input/dependabot-metadata.json Fixture input: only build-time updates should no-op
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/ignore_non_runtime_updates/expected/stdout.txt Fixture expected stdout
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/ignore_non_runtime_updates/expected/pyproject.toml Fixture expected file output (unchanged)
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/empty_updated_dependencies_json/input/pyproject.toml Fixture input: empty metadata
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/empty_updated_dependencies_json/expected/stderr.contains.txt Fixture expected stderr substring
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/empty_updated_dependencies_json/expected/pyproject.toml Fixture expected file output (unchanged)
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/downgrade_refused/input/pyproject.toml Fixture input: downgrade scenario
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/downgrade_refused/input/dependabot-metadata.json Fixture input: downgrade scenario
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/downgrade_refused/expected/stderr.contains.txt Fixture expected stderr substring
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/downgrade_refused/expected/pyproject.toml Fixture expected file output (unchanged)
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/double_runtime_constraint/input/pyproject.toml Fixture input: duplicate runtime constraints
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/double_runtime_constraint/input/dependabot-metadata.json Fixture input: duplicate runtime constraints
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/double_runtime_constraint/expected/stderr.contains.txt Fixture expected stderr substring
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/double_runtime_constraint/expected/pyproject.toml Fixture expected file output (unchanged)
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/conflicting_root_metadata/input/pyproject.toml Fixture input: conflicting root metadata entries
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/conflicting_root_metadata/input/dependabot-metadata.json Fixture input: conflicting root metadata entries
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/conflicting_root_metadata/expected/stderr.contains.txt Fixture expected stderr substring
tests/cookiecutter/scripts/fixtures/dependabot-grpc-fixer/conflicting_root_metadata/expected/pyproject.toml Fixture expected file output (unchanged)
src/frequenz/repo/config/AGENTS.md Package subtree guide
github-rulesets/python/Protect version branches.json Add required check context for gRPC migration job
docs/user-guide/advanced-usage.md Document gRPC migration workflow usage and requirements
cookiecutter/{{cookiecutter.github_repo_name}}/.github/workflows/grpc-migration.yaml New template workflow for gRPC/protobuf runtime floor migration
cookiecutter/{{cookiecutter.github_repo_name}}/.github/workflows/auto-dependabot.yaml Template: skip grpc/protobuf Dependabot group PRs from auto-merge workflow
cookiecutter/{{cookiecutter.github_repo_name}}/.github/dependabot.yml Template: add grpc/protobuf groups + exclude from patch/minor groups (API only)
cookiecutter/scripts/dependabot-grpc-fixer.py New migration script to rewrite runtime bounds based on Dependabot metadata
cookiecutter/migrate.py Add migration step to install workflow + update dependabot config + ruleset
cookiecutter/hooks/post_gen_project.py Post-generation cleanup to remove empty grpc workflow file for non-API repos
cookiecutter/AGENTS.md Cookiecutter subtree guide
RELEASE_NOTES.md Release note entry for new gRPC migration workflow
AGENTS.md Update repo-level AGENTS knowledge base

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cookiecutter/migrate.py Outdated
Comment thread cookiecutter/migrate.py
Comment thread docs/user-guide/advanced-usage.md Outdated
@@ -0,0 +1,359 @@
"""Tests for the dependabot_gprc_fixer package."""
Comment on lines +33 to +34
# These need a migration script to fix dependabot missing updating
# the runtime dependencies
llucax added 10 commits May 4, 2026 14:45
Use a much more concise style and split information to sub-files to
avoid filling up the context window unnecessarily.

Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
This script will be used to automatically fix wrong dependabot updates
for grouped grpcio*/protobuf dependencies.

These dependencies have 2 parts for API projects, a build-time part and
a run-time part. The build-time dependencies need to be older than the
runtime dependencies, and dependabot only bumps the build-time
dependencies, which leaves repositories in a bad state.

On top of that, runtime dependencies have special compatibility rules
for protobuf. Two consecutive major versions are guaranteed to be
compatible.

This new script bumps the minimum runtime dependencies to match the
dependabot-updated build-time dependencies and make sure the supported
major versions are also bumped appropriately, including the comments.

Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
This will be used to add some unit tests for the internals of the
script, which can't be easily imported with just "import" because it is
not intended to be used as a module.

Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
This commit uses the exiting generated API golden tests (cookiecutter
generate templates) to validate the script works on the current
template.

This ensures that things continue to work even if the template
pyproject.toml file is changed and uses a different format to specify
the grpc/protobuf dependencies.

Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
Golden test fixtures for the new dependabot-grpc-fixer.py script will be
stored in tests/cookiecutter/scripts/fixtures/fixer_cases with one
sub-directory per test case, where each test case is composed of the
expected inputs and outputs.

Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
Add many test cases to cover for most happy and error paths.

Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
All other tests are calling the fixer.main() function (or other internal
functions), but none is really testing the fixer works as a script, and
its arguments work as expected.

Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
The dependabot-grpc-fixer is meant to be exercised as a script, not
through a dynamically imported module. Run the remaining tests via
runpy so they cover the supported entrypoint instead of the script
internals.

Drop the replace_range() unit test and the shared module loader to
reduce coupling to implementation details while keeping the existing
script-level coverage.

Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
Keep the generic Python version-branch protection usable by non-API
repositories, and add a separate API-specific variant that also requires
the gRPC runtime-floor migration check.

Update the GitHub setup docs to make API repositories import the
API-specific ruleset instead of the generic Python one.

Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
llucax added 4 commits May 4, 2026 14:45
For API repositories the existing monolithic 'grpc' group is replaced
with three groups so the upcoming grpc-migration workflow can target
them precisely:

  - 'grpc-compatible' for patch and minor updates of grpcio,
    grpcio-tools and protobuf together.
  - 'grpcio-major' for major updates of grpcio and grpcio-tools.
  - 'protobuf-major' for major updates of protobuf.

The split is needed because grpcio releases historically lag behind
protobuf, so a single 'grpc' group for major updates would block one
on the other.

The grpc/protobuf packages are also excluded from the patch and minor
groups, as they need to be handled by the dedicated grpc-migration
workflow that syncs the runtime '>=' floors with the new build pins.

Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
The 'auto-dependabot' workflow now skips PRs from the three new grpc
groups so they are exclusively handled by the upcoming new migration
workflow.

Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
The new 'grpc-migration' workflow handles Dependabot PRs from the
three new grpc/protobuf groups by syncing the runtime '>=' floors in
'pyproject.toml' with the new build-time pins.

The workflow is only rendered for API projects (the only template
type that pins grpc/protobuf as build-time dependencies), so the
post-generation hook removes the leftover empty file for the other
project types.

Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
Add a 'migrate_grpc_workflow_setup' step that brings existing API
projects in line with the new grpc/protobuf handling. The step is a
no-op for non-API projects.

For API projects it:

* Updates '.github/dependabot.yml' to use the new
  'grpc-compatible'/'grpcio-major'/'protobuf-major' Dependabot groups,
  replacing any verbatim old 'grpc:' group and inserting the three
  groups otherwise. It also adds the grpc/protobuf packages to the
  'patch' and 'minor' groups' 'exclude-patterns'. Customized 'grpc:'
  groups are left untouched and reported as a manual step.

* Updates '.github/workflows/auto-dependabot.yaml' to skip PRs from
  the three new grpc Dependabot groups.

* Installs (or replaces) '.github/workflows/grpc-migration.yaml'
  with the latest workflow contents. Existing experimental versions
  shipped to some repositories are unconditionally replaced.

* Adds the 'Fix gRPC/protobuf runtime floors' required status check
  to the 'Protect version branches' GitHub ruleset via the 'gh' CLI.

Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
@llucax llucax force-pushed the fix-grpc-group branch from c017d6f to 2f18ecd Compare May 4, 2026 12:48
llucax added 2 commits May 4, 2026 14:53
Add a new 'gRPC migration workflow' section to the advanced usage
guide modeled after the 'Black formatting migration workflow'
section. The new section explains:

* Why the workflow exists (grpc/protobuf cross-version guarantee
  versus Dependabot's build-only updates).
* That the workflow ships with the cookiecutter template for API
  projects and uses the 'dependabot-grpc-fixer.py' script from this
  repository.
* How to recreate the caller workflow, dependabot groups and
  auto-dependabot interaction in API projects that were not
  generated from the template.

Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
Signed-off-by: Leandro Lucarella <luca-frequenz@llucax.com>
@llucax llucax force-pushed the fix-grpc-group branch from 2f18ecd to f6ea363 Compare May 4, 2026 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

part:cookiecutter Affects the generation of projects using cookiecutter part:template Affects the cookiecutter template files part:tests Affects the unit, integration and performance (benchmarks) tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants