bazel@HEAD breaks rules_python toolchain resolution when using the starlark implementation of rules_python

### Description of the bug:


It seems that our tests started failing expecting a different toolchain version than what we are getting.

### Which category does this issue belong to?


_No response_

### What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.


This can be reproduced in `rules_python` as follows:
```
USE_BAZEL_VERSION=last_green RULES_PYTHON_ENABLE_PYSTAR=1 bazelisk test //python/tests/toolchains:python_3_9_10_x86_64-unknown-linux-gnu_test
2023/10/17 09:12:13 Using unreleased version at commit 14e3d1b825dab76c5886fffb9dd1c01841c18795
2023/10/17 09:12:13 Downloading https://storage.googleapis.com/bazel-builds/artifacts/centos7/14e3d1b825dab76c5886fffb9dd1c01841c18795/bazel...
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Analyzed target //python/tests/toolchains:python_3_9_10_x86_64-unknown-linux-gnu_test (69 packages loaded, 3152 targets configured).
FAIL: //python/tests/toolchains:python_3_9_10_x86_64-unknown-linux-gnu_test (see /home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/execroot/rules_python/bazel-out/k8-fastbuild/testlogs/python/tests/toolchains/python_3_9_10_x86_64-unknown-linux-gnu_test/test.log)
INFO: From Testing //python/tests/toolchains:python_3_9_10_x86_64-unknown-linux-gnu_test:
==================== Test output for //python/tests/toolchains:python_3_9_10_x86_64-unknown-linux-gnu_test:
$TEST_TMPDIR defined: output root default is '/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/_tmp/ef9db39a86b05066e809f377dacf23f4' and max_idle_secs default is '15'.
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'run' from /home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/bazel-out/k8-fastbuild/bin/python/tests/toolchains/run_test_3.9.10.sh.runfiles/rules_python/python/tests/toolchains/3.9.10/.bazelrc:
  Inherited 'build' options: --override_repository rules_python=/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/bazel-out/k8-fastbuild/bin/python/tests/toolchains/run_test_3.9.10.sh.runfiles/rules_python --test_output=errors
Computing main repo mapping:
WARNING: --enable_bzlmod is set, but no MODULE.bazel file was found at the workspace root. Bazel will create an empty MODULE.bazel file. Please consider migrating your external dependencies from WORKSPACE to MODULE.bazel. For more details, please refer to https://github.com/bazelbuild/bazel/issues/18958.
Computing main repo mapping:
Computing main repo mapping:
Computing main repo mapping:
Computing main repo mapping:
Loading:
Loading: 0 packages loaded
Analyzing: target @python//:python3 (1 packages loaded, 0 targets configured)
Analyzing: target @python//:python3 (1 packages loaded, 0 targets configured)
[0 / 1] checking cached actions
Analyzing: target @python//:python3 (2 packages loaded, 0 targets configured)
[1 / 1] checking cached actions
Analyzing: target @python//:python3 (4 packages loaded, 14 targets configured)
[1 / 1] checking cached actions
Analyzing: target @python//:python3 (4 packages loaded, 14 targets configured)
[1 / 1] checking cached actions
Analyzing: target @python//:python3 (4 packages loaded, 14 targets configured)
[1 / 1] checking cached actions
Analyzing: target @python//:python3 (4 packages loaded, 14 targets configured)
[1 / 1] checking cached actions
Analyzing: target @python//:python3 (4 packages loaded, 14 targets configured)
[1 / 1] checking cached actions
Analyzing: target @python//:python3 (4 packages loaded, 14 targets configured)
[1 / 1] checking cached actions
Analyzing: target @python//:python3 (4 packages loaded, 14 targets configured)
[1 / 1] checking cached actions
INFO: Analyzed target @python//:python3 (5 packages loaded, 16 targets configured).
INFO: Found 1 target...
Target @python_x86_64-unknown-linux-gnu//:bin/python3 up-to-date (nothing to build)
INFO: Elapsed time: 22.738s, Critical Path: 0.07s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: /home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/_tmp/ef9db39a86b05066e809f377dacf23f4/_bazel_aignas/485f80f93d904973eb69e2d94ad629f9/external/python_x86_64-unknown-linux-gnu/bin/python3 --version
$TEST_TMPDIR defined: output root default is '/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/_tmp/ef9db39a86b05066e809f377dacf23f4' and max_idle_secs default is '15'.
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'test' from /home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/bazel-out/k8-fastbuild/bin/python/tests/toolchains/run_test_3.9.10.sh.runfiles/rules_python/python/tests/toolchains/3.9.10/.bazelrc:
  Inherited 'build' options: --override_repository rules_python=/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/bazel-out/k8-fastbuild/bin/python/tests/toolchains/run_test_3.9.10.sh.runfiles/rules_python --test_output=errors
Computing main repo mapping:
Loading:
Loading: 0 packages loaded
Analyzing: target //:python_version_test (1 packages loaded, 0 targets configured)
Analyzing: target //:python_version_test (1 packages loaded, 0 targets configured)
[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
Analyzing: target //:python_version_test (50 packages loaded, 2 targets configured)
[1 / 1] checking cached actions
Analyzing: target //:python_version_test (57 packages loaded, 169 targets configured)
[1 / 1] checking cached actions
Analyzing: target //:python_version_test (57 packages loaded, 169 targets configured)
[1 / 1] checking cached actions
Analyzing: target //:python_version_test (64 packages loaded, 507 targets configured)
[1 / 1] checking cached actions
Analyzing: target //:python_version_test (64 packages loaded, 507 targets configured)
[1 / 1] checking cached actions
INFO: Analyzed target //:python_version_test (70 packages loaded, 544 targets configured).
[6 / 7] Testing //:python_version_test; 0s linux-sandbox
FAIL: //:python_version_test (see /home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/_tmp/ef9db39a86b05066e809f377dacf23f4/_bazel_aignas/485f80f93d904973eb69e2d94ad629f9/execroot/_main/bazel-out/k8-fastbuild/testlogs/python_version_test/test.log)
INFO: From Testing //:python_version_test:
==================== Test output for //:python_version_test:
F
======================================================================
FAIL: test_match_toolchain (__main__.TestPythonVersion.test_match_toolchain)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/_tmp/ef9db39a86b05066e809f377dacf23f4/_bazel_aignas/485f80f93d904973eb69e2d94ad629f9/sandbox/linux-sandbox/1/execroot/_main/bazel-out/k8-fastbuild/bin/python_version_test.runfiles/_main/python_version_test.py", line 22, in test_match_toolchain
    self.assertEqual(platform.python_version(), os.getenv("PYTHON_VERSION"))
AssertionError: '3.11.5' != '3.9.10'
- 3.11.5
+ 3.9.10


----------------------------------------------------------------------
Ran 1 test in 0.002s

FAILED (failures=1)
================================================================================
INFO: Found 1 test target...
Target //:python_version_test up-to-date:
  bazel-bin/python_version_test
INFO: Elapsed time: 7.701s, Critical Path: 0.60s
INFO: 7 processes: 5 internal, 2 linux-sandbox.
INFO: Build completed, 1 test FAILED, 7 total actions
//:python_version_test                                                   FAILED in 0.4s
  /home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/_tmp/ef9db39a86b05066e809f377dacf23f4/_bazel_aignas/485f80f93d904973eb69e2d94ad629f9/execroot/_main/bazel-out/k8-fastbuild/testlogs/python_version_test/test.log

Executed 1 out of 1 test: 1 fails locally.
E
======================================================================
ERROR: test_match_toolchain (__main__.TestPythonVersion.test_match_toolchain)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/sandbox/linux-sandbox/1/execroot/rules_python/bazel-out/k8-fastbuild/bin/python/tests/toolchains/run_test_3.9.10.sh.runfiles/rules_python/python/tests/toolchains/3.9.10/run_acceptance_test.py", line 71, in test_match_toolchain
    subprocess.run("bazel test --announce_rc //...", shell=True, check=True)
  File "/home/aignas/.cache/bazel/_bazel_aignas/6f0de8c9128ee8d5dbf27ba6dcc48bdd/execroot/rules_python/external/python_3_11_6_x86_64-unknown-linux-gnu/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'bazel test --announce_rc //...' returned non-zero exit status 3.

----------------------------------------------------------------------
Ran 1 test in 30.865s

FAILED (errors=1)
================================================================================
```

### Which operating system are you running Bazel on?


Linux

### What is the output of `bazel info release`?


_No response_

### If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.


_No response_

### What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?


_No response_

### Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.


Yes,

I have bisected and narrowed down the issue as follows:
```console
$ RULES_PYTHON_ENABLE_PYSTAR=1 bazelisk --bisect=2c9d68e05861ac3e71cb86de2b69bc52403411bf..HEAD test //python/tests/toolchains:python_3_9_10_x86_64-unknown-linux-gnu_test
```

yielded the following known to work commits:
* https://github.com/bazelbuild/bazel/commit/75657569220178f11ec12f7ce33812e571c6c854 (passes)
* https://github.com/bazelbuild/bazel/commit/2b8219042c (passes)

Then there is a series of commits that are failing:
* https://github.com/bazelbuild/bazel/commit/659d7b223a (broken build)
* https://github.com/bazelbuild/bazel/commit/30d033cd7e (broken build)
* https://github.com/bazelbuild/bazel/commit/12bc1e6c27 (broken build)
* https://github.com/bazelbuild/bazel/commit/0e1b3e2d40 (broken build)
* https://github.com/bazelbuild/bazel/commit/bc1df91807 (broken build)
* https://github.com/bazelbuild/bazel/commit/cadbaa5626 (broken build)
* https://github.com/bazelbuild/bazel/commit/137d3f18be (broken build)
* https://github.com/bazelbuild/bazel/commit/4eada724a3 (test is failing)

### Have you found anything relevant by searching the web?

_No response_

### Any other information, logs, or outputs that you want to share?

Commented in https://github.com/bazelbuild/rules_python/issues/1069#issuecomment-1765475004

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bazel@HEAD breaks rules_python toolchain resolution when using the starlark implementation of rules_python #19838

Description of the bug:

Which category does this issue belong to?

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Which operating system are you running Bazel on?

What is the output of `bazel info release`?

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

Have you found anything relevant by searching the web?

Any other information, logs, or outputs that you want to share?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

bazel@HEAD breaks rules_python toolchain resolution when using the starlark implementation of rules_python #19838

Description

Description of the bug:

Which category does this issue belong to?

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Which operating system are you running Bazel on?

What is the output of bazel info release?

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

Have you found anything relevant by searching the web?

Any other information, logs, or outputs that you want to share?

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

What is the output of `bazel info release`?

If `bazel info release` returns `development version` or `(@non-git)`, tell us how you built Bazel.

What's the output of `git remote get-url origin; git rev-parse master; git rev-parse HEAD` ?