This reverts commit bfa3601.

This fails for "http://gopkg.in/natefinch/lumberjack.v2" for example.

This reverts commit bfa3601.

This fails for "http://gopkg.in/natefinch/lumberjack.v2" for example.

…epo (bazel-contrib#1837)

With this change we add support for platform-specific wheel registration
and
doing the selection of which wheel is used at build time.

This supports:
- Different package versions for different platforms.
- Use string_flags to configure what to fetch/select:
    - only whls, only sdist or auto mode.
    - libc version and `musl` vs `glibc` selection.
    - universal2 vs arch wheels for mac.
    - target osx version selection.

Summary of changes:
- The `uv pip install` would only warn the user of yanked packages but
  would not refuse to install them. Update our implementation to better
  match the same behaviour.
- A previous PR has added the support for passing it in the
`requirements_by_platform` and this just add the necessary code to make
  sure that we can also do the dependency management when parsing the
  `whl` `METADATA` files.
- Only configure `dev_pip` deps for `linux` and `osx` platforms to not
raise
  issues later.
- Add a function for generating a `whl_library` name from a `filename`.
- Add a macro for generating all config settings for a particular set of
parameters.
- Update `render_pkg_aliases` to also use those config settings.
- Update the toolchain selection `target_settings` to use the
`py_linux_libc`
config setting. With this the user can register a `musl` linux toolchain
if
  needed. We can also use similar `flag_values` to resolve bazel-contrib#1876.
- Integrate everything in the `pip` extension and setup cross-platform
  select statements.

Work towards bazel-contrib#1357, bazel-contrib#260

Stacked on bazel-contrib#1937

…el-contrib#1974)

Remove the `python_version` from `flag_values` and reduces the number of
targets we create by a lot because we no longer need to create all of
the micro
version combinations to also match when we want to match e.g. 3.10 on a
particular platform by using a trick I've learned in bazel-contrib#1968:
```starlark
select({
    "is_python_3.x": "my_config_setting"
    "//conditions:default": "is_python_3.x"
})
```

Then further optimization was done on how we create the aliases and
config
settings and optimizing the usage of `pip_whl` flag finally reduced the
internal number targets. The number config settings targets in
`//tests/private/pip_config_settings/...` where we have multiple python
versions has changed:

* `:is_*` - from **995** to **996** (the extra addition being
is_python_default).
* `:_.*is_*` - from **28272** to **2480** (just python version) and then
to **496** (final).

Work towards bazel-contrib#260

Remove dead code only used in tests.

Work towards bazel-contrib#260

…ib#2002)

Before this change `isolated`, `quiet` and `timeout` would get dropped
if they were specified to some non-default value. This fixes it.

Work towards bazel-contrib#260.

…ts (bazel-contrib#1955)

With this change the default `gazelle` instructions still work and users
do not need to worry about which package is present on which platform.

Work towards bazel-contrib#260

…around (bazel-contrib#2069)

This is extra preparation needed for bazel-contrib#2059.

Summary:
- Create `pypi_repo_utils` for more ergonomic handling of Python in repo
context.
- Split the resolution of requirements files to platforms into a
separate function
to make the testing easier. This also allows more validation that was
realized
  that there is a need for in the WIP feature PR.
- Make the code more robust about the assumption of the target platform
label.

Work towards bazel-contrib#260, bazel-contrib#1105, bazel-contrib#1868.

…azel-contrib#2068)

This is just a small PR to reduce the scope of bazel-contrib#2059.

This just moves some code from one python file to a separate one.

Work towards bazel-contrib#260, bazel-contrib#1105, bazel-contrib#1868.

…ontrib#2075)

This also changes the local_runtime_repo to explicitly check for
supported platforms instead of relying on a `None` value returned by the
helper method. This makes the behaviour exactly the same to the
behaviour before this PR and we can potentially drop the need for the
validation in the future if our local_runtime detection is more robust.

This also makes the platform detectino in `pypi_repo_utils` not depend
on `uname` and only use the `repository_ctx`. Apparently the
`module_ctx.watch` throws an error if one attempts to watch files on the
system (this is left for `repository_rule` it seems and one can only do
`module_ctx.watch` on files within the current workspace. This was
surprising, but could have been worked around by just unifying code.

This splits out things from bazel-contrib#2059 and makes the code more succinct.

Work towards bazel-contrib#260, bazel-contrib#1105, bazel-contrib#1868.

…azel-contrib#2078)

It seems that a few things broke in recent commits:
- We are not using the `MODULE.bazel.lock` file and it seems that it is
easy to
  miss when the components in the PyPI extension stop integrating well
together. This happened during the switch to `{abi}_{os}_{plat}` target
  platform passing within the code.
- The logger code stopped working in the extension after the recent
additions
  to add the `rule_name`.
- `repo_utils.getenv` was always getting `PATH` env var on bazel `6.x`.

This PR fixes both cases and updates docs to serve as a better reminder.
By
fixing the `select_whls` code and we can just rely on target platform
triples
(ABI, OS, CPU). This gets one step closer to maybe supporting optional
`python_version` which would address bazel-contrib#1708. Whilst at it we are also
adding
different status messages for building the wheel from `sdist` vs just
extracting or downloading the wheel.

Tests:
- Added more unit tests and brought them in line with the rest of the
code.
- Checked manually for differences between the `MODULE.bazel.lock` files
in our
`rules_python` extension before bazel-contrib#2069 and after this PR and there are no
  differences except in the `experimental_target_platforms` attribute in
`whl_library`. Before this PR you would see that we do not select any
wheels
  for e.g. `MarkupSafe` and we are always building from `sdist`.

Work towards bazel-contrib#260.

…2059)

Before this change the `all_requirements` and related constants will
have
packages that need to be installed only on specific platforms and will
mean
that users relying on those constants (e.g. `gazelle`) will need to do
extra
work to exclude platform-specific packages. The package managers that
that
support outputting such files now include `uv` and `pdm`. This might be
also
useful in cases where we attempt to handle non-requirements lock files.

Note, that the best way to handle this would be to move all of the
requirements
parsing code to Python, but that may cause regressions as it is a much
bigger
change. This is only changing the code so that we are doing extra
processing
for the requirement lines that have env markers. The lines that have no
markers
will not see any change in the code execution paths and the python
interpreter
will not be downloaded.

We also use the `*_ctx.watch` API where available to correctly
re-evaluate the
markers if the `packaging` Python sources for this change.

Extra changes that are included in this PR:
- Extend the `repo_utils` to have a method for `arch` getting from the
`ctx`.
- Change the `local_runtime_repo` to perform the validation not relying
on the
  implementation detail of the `get_platforms_os_name`.
- Add `$(UV)` make variable for the `uv:current_toolchain` so that we
can
  generate the requirements for `sphinx` using `uv`.
- Swap the requirement generation using `genrule` and `uv` for `sphinx`
and co
so that we can test the `requirement` marker code. Note, the
`requirement`
  markers are not working well with the `requirement_cycles`.

Fixes bazel-contrib#1105.
Fixes bazel-contrib#1868.
Work towards bazel-contrib#260, bazel-contrib#1975.
Related bazel-contrib#1663.

---------

Co-authored-by: Richard Levasseur <rlevasseur@google.com>

…when build sdist (bazel-contrib#2126)

Building sdist results in `Could not find a version that satisfies the
requirement setuptool` this regressed when a fix in parameter handling
got introduced in bazel-contrib#2091.

Before this change the building from sdist when using
`experimental_index_url`
would break because `--no-index` is passed to `pip`. This means that
`pip`
would fail to locate build time dependencies needed for the packages and
would
just not work. In `whl_library` we setup `PYTHONPATH` to have some build
dependencies available (like `setuptools`) and we could use them during
building from `sdist` and to do so we need to add `--no-build-isolation`
flag.
However, for some cases we need to also add other build-time
dependencies (e.g.
`flit_core`) so that the building of the wheel in the `repository_rule`
context
is successfuly. Removing `--no-index` allows `pip` to silently fetch the
needed
build dependencies from PyPI if they are missing and continue with the
build.

This is not a perfect solution, but it does unblock users to use the
`sdist`
distributions with the experimental feature enabled by using
`experimental_index_url` (see bazel-contrib#260 for tracking of the completion).

Fixes bazel-contrib#2118
Fixes bazel-contrib#2152

---------

Co-authored-by: aignas <240938+aignas@users.noreply.github.com>

…l-contrib#2253)

With this change we set the default value of `--python_version` when
the `python.toolchain` is used in `bzlmod` and we generate the
appropriate config settings based on the registered toolchains and
given overrides by the root module.

This means that we expect the `--python_version` to be always set to
a non-empty value under `bzlmod` and we can cleanup code which was
handling `//conditions:default` case. This also means that we can
in theory drop the requirement for `python_version` in `pip.parse`
and just setup dependencies for all packages that we find in the
`requirements.txt` file and move on. This is left as future work
by myself or anyone willing to contribute. We can also start reusing
the same `whl_library` instance for multi-platform packages because
the python version flag is always set - this will simplify the layout
and makes the feature non-experimental anymore under bzlmod.

Summary:
* Add `@pythons_hub` to the `WORKSPACE` with dummy data to make
  pythons_hub work.
* Add `MINOR_MAPPING` and `PYTHON_VERSIONS` to pythons_hub to
  generate the config settings.
* Remove handling of the default version in `pypi` code under bzlmod.

Work towards bazel-contrib#2081, bazel-contrib#260, bazel-contrib#1708

---------

Co-authored-by: Richard Levasseur <rlevasseur@google.com>

…contrib#2290)

With this change we can support ibazel for building our docs again
because we will just not have any sdists that are causing issues. This
limits the scope to only supporting whls at this time, but for the
time being it is the best solution.

Fixes bazel-contrib#2223
Work towards bazel-contrib#260

…l-contrib#2368)

Before this change the `extra_pip_args` would not be passed down the
chain if `experimental_index_url` is set. This adds a test and fixes the
bug.

Work towards bazel-contrib#260

…repos (bazel-contrib#2369)

Before this change, it was impossible for users to use the targets
created with `additive_build_content` whl annotation unless they relied
on the implementation detail of the naming of the spoke repositories and
had `use_repo` statements in their `MODULE.bazel` files importing the
spoke repos.

With bazel-contrib#2325 in the works, users will have to change their `use_repo`
statements, which is going to be disruptive. In order to offer them an
alternative for not relying on the names of the spokes, there has to be
a way to expose the extra targets created and this PR implements a
method. Incidentally, the same would have happened if we wanted to
stabilize the bazel-contrib#260 work and mark `experimental_index_url` as
non-experimental anymore.

I was hoping we could autodetect them by parsing the build content
ourselves in the `pip` extension, but it turned out to be extremely
tricky and I figured that it was better to have an API rather than not
have it.

Whilst at it, also relax the naming requirements for the
`whl_modifications` attribute.

Fixes bazel-contrib#2187

…azel-contrib#2325)

With this change we finally generate the same lock file within the
legacy code `pip.parse` code path and it allows to slowly transition to
using the new code path as much as possible without user doing anything.

This moves the selection of the host-compatible lock file from the
extension evaluation to the build phase - note, we will generate extra
repositories here which might not work on the host platform, however, if
the users are consuming the `whl_library` repos through the hub repo
only, then everything should work. A known issue is that it may break
`bazel query` and in these usecases it is advisable to use `cquery`
until we have `sdist` cross-building from source fully working.

Summary:
- feat: reuse the `render_pkg_aliases` for when filename is not known
  but platform is known
- feat: support generating the extra config settings required
- feat: `get_whl_flag_versions` now generates extra args for the rules
- feat: make lock file generation the same irrespective of the host
  platform
- test: add an extra test with multiple requirements files
- feat: support cross-platform builds using `download_only = True` in
  legacy setups

Note, that users depending on the naming of the whl libraries will need
to start using `extra_hub_aliases` attribute instead to keep their
setups not relying on this implementation detail.

Fixes bazel-contrib#2268
Work towards bazel-contrib#260

---------

Co-authored-by: Richard Levasseur <richardlev@gmail.com>

…-contrib#2391)

This just cleans up the code and moves more logic from the
repository_rule
(i.e. generation of `BUILD.bazel` files) to loading time (macro
evaluation).
This makes the unit testing easier and I plan to also move the code that
is
generating config setting names from filenames to this new macro, but
wanted to
submit this PR to reduce the review chunks.

Summary:
- Add a new `pkg_aliases` macro.
- Move logic and tests for creating WORKSPACE aliases.
- Move logic and tests bzlmod aliases.
- Move logic and tests bzlmod aliases with groups.
- Add a test for extra alias creation.
- Use `whl_alias` in `pypi` extension integration tests.
- Improve the serialization of `whl_alias` for passing to the pypi hub
repo.

Related to bazel-contrib#260, bazel-contrib#2386, bazel-contrib#2337, bazel-contrib#2319 - hopefully cleaning the code up
will make
it easier to address those feature requests later.

---------

Co-authored-by: Richard Levasseur <richardlev@gmail.com>

…-contrib#2532)

The warning is somewhat non-actionable and the sources can be
inspected via the MODULE.bazel.lock file if needed. This makes it
easier to make this option a default at some point.

At the same time cleanup the code since we are not using the
`get_index_urls` to print the warning.

Work towards bazel-contrib#260

Summary:
- Move the `whl_library` creation into a separate function and remove
  the `TODO` note.
- Move the creation of the `get_index_urls` functions into outer
  `parse_modules` function and simplify the reproducible extension
  setting logic.
- Remove the `prefix` parameter from the `*repo_name` functions.
- Add an extra error message, for ensuring that invariants are met.

Work towards bazel-contrib#260

…b#2556)

Before the PR the `config_setting` names where following an internal
logic and those names would be leaking into the error messages when no
match is found. I thought that the names thus should be improved and
maybe made more similar to the `whl` filename parts that they are
derived from.

As part of this change I have also added more docs and added them to
sphinxdocs in the hopes that this documentation helps maintainers and
users looking at error messages alike.

Summary:
* Make names more similar to the whl filenames.
* Instead of having `osx_<cpu>_universal2` config settings for each
  `cpu` value, have a single `osx_universal2` config setting.
* Stop creating redundant/unused config settings
* Refactor the `_dist_config_setting` code to be simpler and create
  fewer targets by using a clever trick for the `whl` config setting
  flag value usage.

The stats:
```
$ bazel query //tests/pypi/config_settings/... | rg ":(|_)is" | wc -l
2223
$ bazel query @dev_pip//_config/... | wc -l
1982

$ bazel query //tests/pypi/config_settings/... | rg ":(|_)is" | wc -l
1780
$ bazel query @dev_pip//_config/... | wc -l
1066
```

Work towards bazel-contrib#260

…ontrib#2692)

This implements the PEP508 compliant marker evaluation in starlark and
removes the need for the Python interpreter when evaluating requirements
files passed to `pip.parse`. This makes the evaluation faster and allows
us to fix a few known issues (bazel-contrib#2690).

In the future the intent is to move the `METADATA` parsing to pure
starlark so that the `RequiresDist` could be parsed in starlark at the
macro evaluation or analysis phases. This should make it possible to
more easily solve the design problem that more and more things need to
be passed to `whl_library` as args to have a robust dependency parsing:
* bazel-contrib#2319 needs the full Python version to have correct cross-platform
compatible `METADATA` parsing and passing it to `Python` and back makes
  it difficult/annoying to implement.
* Parsing the `METADATA` file requires the precise list of target
  platform or the list of available packages in the `requirements.txt`.
  This means that without it we cannot trim the dependency tree in the
  `whl_library`. Doing this at macro loading phase allows us to depend
  on `.bzl` files in the `hub_repository` and more effectively pass
  information.

I can remotely see that this could become useful in `py_wheel` or an
building
wheels from sdists as the environment markers may be present in various
source
metadata as well. What is more `uv.lock` file has the env markers as
part of
the lock file information, so this might be useful there.

Work towards bazel-contrib#2423
Work towards bazel-contrib#260
Split from bazel-contrib#2629

…ied (bazel-contrib#2695)

This pull request modifies the SimpleAPI HTML parsing to add a new
field where we can get the `sha256` values by package version. This
allows us to very easily fallback to all packages of a particular
version when using `experimental_index_url` if the hashes are not
specified.

The code deciding which packages to query the SimpleAPI for has been
also modified to only omit queries for packages that are included via
direct URL references.

If we fail to get the data from the SimpleAPI, we will fallback to
`pip` and try to install it via the legacy behaviour.

Fixes bazel-contrib#2023
Work towards bazel-contrib#260
Work towards bazel-contrib#1357
Work towards bazel-contrib#2363

---------

Co-authored-by: Ignas Anikevicius <240938+aignas@users.noreply.github.com>

…ontrib#2629)

This PR starts using the newly introduced (bazel-contrib#2692) PEP508 compliant
requirement marker parser in starlark and moves the dependency
generation from the Python language (`whl_installer`) to the Starlark
in the `whl_library` repository rule.

This PR is (almost) a pure refactor where no bugs are fixed, but this is
foundational work that also adds notes on how things will be moved
to macros (i.e. analysis phase) so that we can fix a few long standing
bugs and prepare for stabilizing the `experimental_index_url` (bazel-contrib#260).

Refactor:
* I have migrated all of the unit tests from Python to starlark for deps
  generation from METADATA `Requires-Dist` fields.
* Read the `METADATA` file itself in Starlark.

Work towards bazel-contrib#260, bazel-contrib#2319, bazel-contrib#2241
Fixes bazel-contrib#2423

This PR moves the parsing of `Requires-Dist` to the loading phase
within the `whl_library_targets_from_requires` macro. The original
`whl_library_targets` macro has been left unchanged so that I don't have
to reinvent the unit tests - it is well covered under tests.

Before this PR we had to wire the `target_platforms` via the
`experimental_target_platforms` attr in the `whl_library`, which means
that whenever this would change (e.g. the minor Python version changes),
the wheel would be re-extracted even though the final result may be the
same.

This refactor uncovered that the dependency graph creation was incorrect
if we had multiple target Python versions due to various heuristics that
this had. In hindsight I had them to make the generated `BUILD.bazel`
files more readable when the unit test coverage was not great. Now this
is unnecessary and since everything is happening in Starlark I thought
that having a simpler algorithm that does the right thing always is the
best way.

This also cleans up the code by removing left over TODO notes or code
that no longer make sense.

Work towards bazel-contrib#260, bazel-contrib#2319

This is a flag to start leveraging of the new code paths. The Starlark
implementation has been added in 1.4 and has been reverted in the latest
release candidates. The `env` variable will be a good way to roll it out
more
gradually and get more testing.

For now we are switching only the `whl_library` internals as the
`requirements.txt` files from `uv` may use `*` in `python_full_version`
and
`platform_version` that are not yet fully supported (bazel-contrib#2826).

Main goals for this is to start using Starlark implementation so that we
don't
have any hidden variables. What is more, having this in Starlark is the
most
maintainable long-term solution for supporting cross-platform builds.

Work towards bazel-contrib#260

---------

Co-authored-by: Richard Levasseur <richardlev@gmail.com>

Summary:
- `pep508_deps` is now much simpler, because the hard work is done in
  analysis phase
- `whl_library` BUILD.bazel tests now also have a test for the legacy
  flow.

One thing that I noticed is that now we have an implicit dependency on
the python toolchain when getting all of the `whl` target tree. This is
a filegroup target that includes dependent wheels. However, we fallback
to the flag values if we don't have the toolchain, so we should be good
in general.

Overall I like how this is turning out because we don't need to pipe the
`target_platforms` anymore when we enable `PIPSTAR` feature. This means
that we can start creating fewer whl_library instances - e.g. a
`py3-none-any` wheel can be fetched once instead of once per python
interpreter version. I'll leave this optimization for a later time.

Work towards bazel-contrib#260

---------

Co-authored-by: Richard Levasseur <richardlev@gmail.com>