Skip to content

Support dynamic inputs with the remote repo contents cache#27634

Closed
fmeum wants to merge 4 commits intobazelbuild:masterfrom
fmeum:all-repos-remote
Closed

Support dynamic inputs with the remote repo contents cache#27634
fmeum wants to merge 4 commits intobazelbuild:masterfrom
fmeum:all-repos-remote

Conversation

@fmeum
Copy link
Copy Markdown
Collaborator

@fmeum fmeum commented Nov 11, 2025 •
edited
Loading

With this change, all reproducible repository rules can now be cached in a disk or remote cache, including those with dependencies recorded dynamically during evaluation.

This is made possible by introducing a new intermediate type of synthetic AC entries. When looking up the predeclared inputs hash for a repo rule with dynamic dependencies, the action result for such an intermediate entry lists one or more sets of inputs (e.g. a particular file in another repo or an environment variable name). These inputs are then requested from Skyframe and their current values are hashed to obtain the key of the next AC entry, which is again either an intermediate entry or a final entry containing the contents of the repository.

RELNOTES: The remote repo contents cache now supports all reproducible repo rules.

@fmeum fmeum force-pushed the all-repos-remote branch 4 times, most recently from b4b3182 to 7b4b79f Compare November 17, 2025 11:25
@fmeum fmeum force-pushed the all-repos-remote branch 7 times, most recently from ee70885 to 94c992a Compare November 26, 2025 20:32
@fmeum fmeum mentioned this pull request Dec 13, 2025
Open
@fmeum fmeum changed the title TMP: Support all repo rules with the remote repo contents cache Support dynamic inputs with the remote repo contents cache Dec 16, 2025
@fmeum fmeum force-pushed the all-repos-remote branch 2 times, most recently from 0763010 to fbaafe6 Compare December 16, 2025 11:38
@fmeum fmeum marked this pull request as ready for review December 16, 2025 11:39
@fmeum fmeum requested review from a team, Wyverald, lberki and meteorcloudy as code owners December 16, 2025 11:39
@fmeum fmeum requested review from tjgq and removed request for a team and lberki December 16, 2025 11:39
@github-actions github-actions bot added team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. team-Remote-Exec Issues and PRs for the Execution (Remote) team awaiting-review PR is awaiting review from an assigned reviewer labels Dec 16, 2025
@fmeum
Copy link
Copy Markdown
Collaborator Author

fmeum commented Dec 16, 2025

@bazel-io fork 9.0.0

@bazel-io bazel-io mentioned this pull request Dec 16, 2025
Closed
@fmeum fmeum requested review from Wyverald and removed request for meteorcloudy January 20, 2026 16:24
@fmeum
Copy link
Copy Markdown
Collaborator Author

fmeum commented Jan 20, 2026

@tjgq @Wyverald I rebased onto master, which made the PR much smaller and also allowed me to get rid of the newly introduced Skyframe state. It's now pretty much "just" new business logic.

@aignas aignas mentioned this pull request Feb 12, 2026
Merged
@mortenmj mortenmj mentioned this pull request Feb 14, 2026
Merged
github-merge-queue bot pushed a commit to bazel-contrib/rules_python that referenced this pull request Feb 23, 2026
@aignas @rickeylev
feat(pypi): make whl_library reproducible under pipstar (#3589)
06aa36d 
This is testing the new API to make use of remote caching mechanisms.

Needs: bazelbuild/bazel#27634

---------

Co-authored-by: Richard Levasseur <rlevasseur@google.com>
rickeylev added a commit to rickeylev/rules_python that referenced this pull request Feb 23, 2026
@aignas @rickeylev
feat(pypi): make whl_library reproducible under pipstar (bazel-contri…
7f6b275 
…b#3589)

This is testing the new API to make use of remote caching mechanisms.

Needs: bazelbuild/bazel#27634

---------

Co-authored-by: Richard Levasseur <rlevasseur@google.com>
(cherry picked from commit 06aa36d)
@rickeylev rickeylev mentioned this pull request Feb 23, 2026
Merged
rickeylev added a commit to bazel-contrib/rules_python that referenced this pull request Feb 23, 2026
@rickeylev @aignas
feat(pypi): make whl_library reproducible under pipstar (#3589) (#3632)
bac5494 
This is testing the new API to make use of remote caching mechanisms.

Needs: bazelbuild/bazel#27634

---------

(cherry picked from commit 06aa36d)

Co-authored-by: Ignas Anikevicius <240938+aignas@users.noreply.github.com>
@fmeum
Copy link
Copy Markdown
Collaborator Author

fmeum commented Mar 12, 2026

@bazel-io fork 9.1.0

@bazel-io bazel-io mentioned this pull request Mar 12, 2026
Open
@iancha1992
Copy link
Copy Markdown
Member

iancha1992 commented Mar 12, 2026

@fmeum Could you please resolve the conflicts?

@fmeum fmeum force-pushed the all-repos-remote branch from e88b831 to 8e579dc Compare March 13, 2026 11:12
@fmeum
Copy link
Copy Markdown
Collaborator Author

fmeum commented Mar 13, 2026

@iancha1992 Done!

Wyverald
Wyverald approved these changes Mar 23, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@Wyverald Wyverald left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great work, as always! I think I understand most of what this PR is doing :) Thanks for pushing through!

@fmeum fmeum force-pushed the all-repos-remote branch from 8e579dc to a62cada Compare March 23, 2026 22:40
@fmeum fmeum requested a review from Wyverald March 23, 2026 22:47
Wyverald
Wyverald approved these changes Mar 24, 2026
View reviewed changes
@fmeum fmeum requested a review from Wyverald March 24, 2026 08:39
@Wyverald Wyverald added awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally and removed awaiting-review PR is awaiting review from an assigned reviewer labels Mar 24, 2026
@github-actions github-actions bot removed the awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally label Mar 26, 2026
@fmeum fmeum deleted the all-repos-remote branch March 26, 2026 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tjgq tjgq Awaiting requested review from tjgq

@Wyverald Wyverald Awaiting requested review from Wyverald Wyverald is a code owner

Assignees

No one assigned

Labels

team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. team-Remote-Exec Issues and PRs for the Execution (Remote) team

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@fmeum @iancha1992 @Wyverald