feat(bzlmod): expose platform-agnostic repo target for toolchain interpreter by chrislovecnm · Pull Request #1155 · bazel-contrib/rules_python

chrislovecnm · 2023-04-05T16:53:28Z

This exposes a new repo and target, @{name}_host_interpreter//:python, created by python.toolchain(), that points to the host OS's Python interpreter for that particular toolchain.

This solves two problems:

pip.parse() can now refer to the same interpreter used in the toolchains
There is now a canonical, public, way to refer to the host OS Python interpreter
for repository rules.

The above were sort of possible for users to do already, but it required them to write much more configuration and extension code to do so. This moves that sort of boilerplate into our code so they have a simpler configuration.

Also:

removing bzlmod support in the build_file_generation example; making examples work
with both WORKSPACE and MODULE is a pain, hence splitting them.
adding an example of bzlmod and gazelle
improved documentation in the pip arguments

Closes: #1161

CI failing

examples/build_file_generation/MODULE.bazel

examples/bzlmod/MODULE.bazel

aignas · 2023-04-06T01:20:13Z

examples/bzlmod/BUILD.bazel

+    # NOTE: we can use this flag in order to make our setup compatible with
+    # bzlmod.
+    use_pip_repository_aliases = True,


Suggested change

# NOTE: we can use this flag in order to make our setup compatible with

# bzlmod.

use_pip_repository_aliases = True,

use_pip_repository_aliases = True,

Since this is a bzlmod only example, we don't need this comment

Do we need the flag when we are not using bzlmod?

examples/bzlmod/MODULE.bazel

examples/build_file_generation/WORKSPACE

examples/bzlmod/runfiles/BUILD.bazel

chrislovecnm · 2023-04-11T21:32:28Z

We are having a few flakes, but the building on Windows OS is not working. See #1163. I have recreated the problem locally.

chrislovecnm · 2023-05-03T16:37:42Z

@aignas I added a new example that includes bzlmod and gazelle. We have some issues with the sub-module, which I will file. I am not changing the build_file_generation example, and I will back out bzlmod support out of that example in another PR.

rickeylev

Looking pretty good, mostly small stuff

examples/bzlmod_build_file_generation/BUILD.bazel

examples/bzlmod_build_file_generation/runfiles/runfiles_test.py

python/extensions.bzl

Having both bzlmod and a WORKSPACE file confuses the user, and I have bazel-contrib#1155 which adds a new example for gazelle and bzlmod.

Having both bzlmod and a WORKSPACE file confuses the user, and I have #1155 which adds a new example for gazelle and bzlmod.

python/extensions.bzl

This commit includes: - adding a new bzlmod_build_file_geration demo - adding the capabilty where python.toolchain create the symlink to set the pip python interperter target - improved documentation in the pip arguments

chrislovecnm · 2023-05-04T18:02:08Z

@linzhp, maybe I do not understand what you recommend, but this is what I did as a test:

diff --git a/python/extensions.bzl b/python/extensions.bzl
index 3bcbb50..3c3d582 100644
--- a/python/extensions.bzl
+++ b/python/extensions.bzl
@@ -151,11 +151,12 @@ def _repo_mapped_label(module_label, extension_name, apparent):
           name used by the extension named by `ext_name` (i.e. the value of the
           `name` arg the extension passes to repository rules)
     """
-    return Label("@@{module}~{extension_name}~{apparent}".format(
-        module = module_label.workspace_name,
-        extension_name = extension_name,
-        apparent = apparent,
-    ))
+    return Label("@python3_x86_64-unknown-linux-gnu")
+    #return Label("@@{module}~{extension_name}~{apparent}".format(
+    #    module = module_label.workspace_name,
+    #    extension_name = extension_name,
+    #    apparent = apparent,
+    #))

 # We are doing some bazel stuff here that could use an explanation.
 # The basis of this function is that we need to create a symlink to

I used return Label("@python3_x86_64-unknown-linux-gnu") on Linux instead of building the label in the fashion that is working.

We then get the error:

$ bazel clean --expunge && bazel build //...
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.
Starting local Bazel server and connecting to it...
DEBUG: gazelle@0.27.0/MODULE.bazel:7:6: WARNING: The bazel_gazelle Bazel module is still highly experimental and subject to change at any time. Only use it to try out bzlmod for now.
DEBUG: rules_go@0.37.0/MODULE.bazel:8:6: WARNING: The rules_go Bazel module is still highly experimental and subject to change at any time. Only use it to try out bzlmod for now.
DEBUG: rules_go@0.33.0/MODULE.bazel:7:6: WARNING: The rules_go Bazel module is still highly experimental and subject to change at any time. Only use it to try out bzlmod for now.
INFO: Repository rules_python~override~python~python3_host_interpreter instantiated at:
  callstack not available
Repository rule _host_hub defined at:
  /home/clove/.cache/bazel/_bazel_clove/e19dab5c65f1ab29c26be4ab0039d0eb/external/rules_python~override/python/extensions.bzl:209:28: in <toplevel>
ERROR: An error occurred during the fetch of repository 'rules_python~override~python~python3_host_interpreter':
   Traceback (most recent call last):
        File "/home/clove/.cache/bazel/_bazel_clove/e19dab5c65f1ab29c26be4ab0039d0eb/external/rules_python~override/python/extensions.bzl", line 205, column 21, in _host_hub_impl
                repo_ctx.symlink(label, "python")
Error in symlink: Unable to load package for @[unknown repo 'python3_x86_64-unknown-linux-gnu' requested from @rules_python~override]//:python3_x86_64-unknown-linux-gnu: The repository '@[unknown repo 'python3_x86_64-unknown-linux-gnu' requested from @rules_python~override]' could not be resolved: No repository visible as '@python3_x86_64-unknown-linux-gnu' from repository '@rules_python~override'
ERROR: <builtin>: fetching _host_hub rule @rules_python~override//python:rules_python~override~python~python3_host_interpreter: Traceback (most recent call last):
        File "/home/clove/.cache/bazel/_bazel_clove/e19dab5c65f1ab29c26be4ab0039d0eb/external/rules_python~override/python/extensions.bzl", line 205, column 21, in _host_hub_impl
                repo_ctx.symlink(label, "python")
Error in symlink: Unable to load package for @[unknown repo 'python3_x86_64-unknown-linux-gnu' requested from @rules_python~override]//:python3_x86_64-unknown-linux-gnu: The repository '@[unknown repo 'python3_x86_64-unknown-linux-gnu' requested from @rules_python~override]' could not be resolved: No repository visible as '@python3_x86_64-unknown-linux-gnu' from repository '@rules_python~override'
INFO: Repository bazel_tools~remote_coverage_tools_extension~remote_coverage_tools instantiated at:
  callstack not available
Repository rule http_archive defined at:
  /home/clove/.cache/bazel/_bazel_clove/e19dab5c65f1ab29c26be4ab0039d0eb/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
INFO: Repository rules_cc~0.0.2 instantiated at:
  callstack not available
Repository rule http_archive defined at:
  /home/clove/.cache/bazel/_bazel_clove/e19dab5c65f1ab29c26be4ab0039d0eb/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
INFO: Repository rules_java~5.3.5 instantiated at:
  callstack not available
Repository rule http_archive defined at:
  /home/clove/.cache/bazel/_bazel_clove/e19dab5c65f1ab29c26be4ab0039d0eb/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
ERROR: /home/clove/.cache/bazel/_bazel_clove/e19dab5c65f1ab29c26be4ab0039d0eb/external/rules_python~override~pip~pip/tabulate/BUILD.bazel:3:6: no such package '@rules_python~override~pip~pip_tabulate//': no such package '@rules_python~override~python~python3_host_interpreter//': Unable to load package for @[unknown repo 'python3_x86_64-unknown-linux-gnu' requested from @rules_python~override]//:python3_x86_64-unknown-linux-gnu: The repository '@[unknown repo 'python3_x86_64-unknown-linux-gnu' requested from @rules_python~override]' could not be resolved: No repository visible as '@python3_x86_64-unknown-linux-gnu' from repository '@rules_python~override' and referenced by '@rules_python~override~pip~pip//tabulate:tabulate'
ERROR: Analysis of target '//:bzlmod_build_file_generation' failed; build aborted:
INFO: Elapsed time: 3.694s
INFO: 0 processes.

Let me know if there is a different approach. We can always iterate to improve this, but we have already spent many hours of dev time on this.

…d in .bazelrc (#1207) This correctly handles the integration tests and examples that are a part of the `rules_python` workspace and should not be included in the deleted packages list. This brings the changes made to the `.bazelrc` very close to what is in `main`, but I would like to update this later once #1155 and #1205 are merged. Fixes #919.

rickeylev · 2023-05-05T22:22:13Z

I think I remember why calling Label() there doesn't work: It's being called in the repo implementation function, not as part of evaluating the code generated by the repo.

When Label() is called in the host_hub_impl implementation function in extensions.bzl, it's current context is the rules_python module. Since the rules_python module itself never called use_repo() on anything (it can't; the repo names are user selected), it's repo visibility mapping is empty.

If host_hub generated a .bzl file with e.g. Label("@python3_x86_64-linux"), that would work, because when someone loads the host hub repo, that Label call will be evaluated in the context of the host hub repo, which does have access to the other repos that the extension generated.

fmeum · 2023-05-05T22:32:32Z

@rickeylev That's correct. This situation is especially tricky since the resolved label is needed in the implementation function of the repo rule. This may require introducing a helper extension.

@Wyverald and I were discussing a module_ctx.label_in_repo function that would drastically simplify such situations. If you would like to see something like this realized, could you file an issue for it?

chrislovecnm requested review from f0rmiga and rickeylev as code owners April 5, 2023 16:53

chrislovecnm marked this pull request as draft April 5, 2023 16:54

chrislovecnm changed the title ~~Fix: Gazelle updates~~ Fix: Gazelle updates for examples Apr 5, 2023

f0rmiga previously approved these changes Apr 5, 2023

View reviewed changes

rickeylev reviewed Apr 5, 2023

View reviewed changes

examples/build_file_generation/MODULE.bazel Outdated Show resolved Hide resolved

rickeylev reviewed Apr 5, 2023

View reviewed changes

aignas reviewed Apr 6, 2023

View reviewed changes

aignas mentioned this pull request Apr 6, 2023

Gazelle plugin's modules_mapping rule still depends on WORKSPACE file when enable bzlmod #1153

Closed

chrislovecnm force-pushed the gazelle-updates branch 5 times, most recently from 67670bc to e5754bf Compare April 6, 2023 16:04

chrislovecnm commented Apr 6, 2023

View reviewed changes

examples/bzlmod/runfiles/BUILD.bazel Show resolved Hide resolved

chrislovecnm mentioned this pull request Apr 6, 2023

refactor: Updating example to only use bzlmod and updates to example #1147

Closed

chrislovecnm changed the title ~~Fix: Gazelle updates for examples~~ Refactor: Gazelle and bzlmod updates for examples Apr 6, 2023

chrislovecnm force-pushed the gazelle-updates branch 2 times, most recently from 149d730 to ba28c92 Compare April 11, 2023 20:03

chrislovecnm mentioned this pull request Apr 11, 2023

gazelle not working with windows #1163

Closed

chrislovecnm force-pushed the gazelle-updates branch 7 times, most recently from a9aec42 to 3c796e0 Compare April 17, 2023 18:35

chrislovecnm mentioned this pull request Apr 18, 2023

Implement hermetic toolchain with bzlmod - toolchain not configurable work on different platforms #1161

Closed

chrislovecnm force-pushed the gazelle-updates branch from 9679cae to 60818c4 Compare May 3, 2023 16:26

rickeylev requested changes May 3, 2023

View reviewed changes

chrislovecnm force-pushed the gazelle-updates branch 4 times, most recently from 7af481d to 3e4a140 Compare May 3, 2023 20:07

chrislovecnm added a commit to lionkube/rules_python that referenced this pull request May 3, 2023

feat: removing bzlmod from example

667978b

Having both bzlmod and a WORKSPACE file confuses the user, and I have bazel-contrib#1155 which adds a new example for gazelle and bzlmod.

This was referenced May 3, 2023

feat: removing bzlmod from example #1200

Merged

Example of pip_parse_vendored under bzlmod #1198

Closed

rickeylev pushed a commit that referenced this pull request May 3, 2023

feat: removing bzlmod from example (#1200)

96b4fa1

Having both bzlmod and a WORKSPACE file confuses the user, and I have #1155 which adds a new example for gazelle and bzlmod.

linzhp reviewed May 3, 2023

View reviewed changes

python/extensions.bzl Show resolved Hide resolved

This was referenced May 4, 2023

test(pip_parse): add integration tests for macros in requirements.bzl #1205

Closed

fix(update_deleted_packages.sh): wheels example should not be included in .bazelrc #1207

Merged

chrislovecnm force-pushed the gazelle-updates branch 4 times, most recently from 64eea9c to 4f04915 Compare May 4, 2023 17:07

chrislovecnm force-pushed the gazelle-updates branch from 4f04915 to 1ec8e5e Compare May 4, 2023 17:15

rickeylev approved these changes May 4, 2023

View reviewed changes

rickeylev changed the title ~~feat: Bzlmod refactor and allow for dynamic python interpreter target~~ feat(bzlmod): expose platform-agnostic repo target for toolchain interpreter May 4, 2023

rickeylev added this pull request to the merge queue May 4, 2023

Merged via the queue into bazel-contrib:main with commit 0912bba May 4, 2023

aaliddell mentioned this pull request May 12, 2023

asyncpg fails to import under bzlmod, works under workspace #1222

Closed

Uh oh!

Conversation

chrislovecnm commented Apr 5, 2023 • edited by rickeylev Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aignas Apr 6, 2023

Choose a reason for hiding this comment

Uh oh!

chrislovecnm Apr 6, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chrislovecnm commented Apr 11, 2023

Uh oh!

chrislovecnm commented May 3, 2023

Uh oh!

rickeylev left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chrislovecnm commented May 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rickeylev commented May 5, 2023

Uh oh!

fmeum commented May 5, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

chrislovecnm commented Apr 5, 2023 •

edited by rickeylev

Loading

chrislovecnm commented May 4, 2023 •

edited

Loading