Skip to content

Exclude autogenerated BUILD file from wheel library runfiles#441

Closed
person142 wants to merge 1 commit intobazel-contrib:masterfrom
person142:pip-parsed-no-build
Closed

Exclude autogenerated BUILD file from wheel library runfiles#441
person142 wants to merge 1 commit intobazel-contrib:masterfrom
person142:pip-parsed-no-build

Conversation

@person142
Copy link
Copy Markdown
Contributor

@person142 person142 commented Mar 27, 2021

PR Checklist

Please check if your PR fulfills the following requirements:

  • Does not include precompiled binaries, eg. .par files. See CONTRIBUTING.md for info
  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

  • Bugfix
  • Feature (please, look at the "Scope of the project" section in the README.md file)
  • Code style update (formatting, local variables)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • CI related changes
  • Documentation content changes
  • Other... Please describe:

What is the current behavior?

Bazel appears to inject a BUILD file into the root of an external repository. For the non-incremental pip_install the external repository root ends up looking something like this:

BUILD            WORKSPACE        pypi__attrs      requirements.bzl

Since pip_parse doesn't have the extra layers of directories you end up with the external repository looking something like this:

BUILD                  WORKSPACE              attrs-20.3.0.dist-info
BUILD.bazel            attr

The build file is then picked up in the data glob and injected into the runfiles.

Issue Number: #440

What is the new behavior?

This adds the autogenerated BUILD to the exclude so that it doesn't end up in the runfiles. It still won't fix the issue identified in #427 however, but AFAICT that will require changes to Bazel itself to not autogenerate the BUILD files?

Does this PR introduce a breaking change?

  • Yes
  • No

Other information

None

@person142 person142 force-pushed the pip-parsed-no-build branch from 86ffdba to 6b492b7 Compare April 9, 2021 02:25
@person142
Copy link
Copy Markdown
Contributor Author

@hrfuller a polite ping (don't have permission to request you as a reviewer).


data_exclude = ["*.whl", "**/*.py", "**/* *", "BUILD.bazel", "WORKSPACE"] + pip_data_exclude
data_exclude = (
["*.whl", "**/*.py", "**/* *", "BUILD", "BUILD.bazel", "WORKSPACE"]
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should WORKSPACE.Bazel be a part of this list too?

Copy link
Copy Markdown
Contributor

@hrfuller hrfuller Apr 15, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://docs.bazel.build/versions/master/external.html#working-with-external-dependencies

Looks like WORKSPACE.bazel is an allowed value. We don't use it in our repository rules though.

@hrfuller
Copy link
Copy Markdown
Contributor

hrfuller commented Apr 15, 2021

I believe this would cause a regression to #427 in pip_parse. I think the correct way to do this is probably to extract the wheels into a subdirectory instead of at the root of the individual repos.

@person142
Copy link
Copy Markdown
Contributor Author

person142 commented Apr 17, 2021

I believe this would cause a regression to #427 in pip_parse

Hasn’t the regression already happened? An empty BUILD file is already present when using pip_parse.

@person142
Copy link
Copy Markdown
Contributor Author

person142 commented Apr 17, 2021

Ah, or maybe “this” -> “the current state of affairs” and not “the changes in this PR”, in which case yes I think so.

@hrfuller
Copy link
Copy Markdown
Contributor

My concern is that excluding "BUILD" would exclude a valid build directory on case insensitive file systems. There wont be a regression ala #427 because we write build file contents into BUILD.bazel now. Bazel "glob" ignores directories by default so if you tried to install the keyring wheel used as example in $427 the build directory would be excluded.

As an aside if you need an immediate workaround you can use the pip_data_excludes

As I mention above, I think the correct way to fix this issue is to create the wheel package in a sub directory.

@person142
Copy link
Copy Markdown
Contributor Author

There wont be a regression ala #427 because we write build file contents into BUILD.bazel now

Right, but when using pip_parse there is also an autogenerated BUILD file in the directory too (which is what I was trying to convey in #440). i.e. right now there is both BUILD.bazel and BUILD in the generated repos, which should already cause the regression.

Certainly this PR isn't the right fix, and I'm happy to close it-but I believe the regression is already present. Using a sub-directory would be another work around-but at the end of the day this appears to be a Bazel bug-it seems to be adding an automatic empty BUILD file even when a BUILD.bazel file is already present.

@person142
Copy link
Copy Markdown
Contributor Author

A concrete example of the regression (rules_python is at master):

$ cat keyring_requirements.txt
#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile keyring_requirements.in
#
keyring==22.3.0
    # via -r keyring_requirements.in
$ cat WORKSPACE
local_repository(
    name = "rules_python",
    path = "../rules_python",
)

load("@rules_python//python:pip.bzl", "pip_parse")

pip_parse(
    name = "keyring",
    requirements_lock = "//:keyring_requirements.txt",
)

load("@keyring//:requirements.bzl", "install_deps")
install_deps()
$ bazelisk build @keyring_pypi__keyring//...
[snip]
NotADirectoryError: [Errno 20] Not a directory: 'build/lib'
)
INFO: Elapsed time: 2.335s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)

@person142
Copy link
Copy Markdown
Contributor Author

person142 commented Apr 17, 2021

Ah, I was mistaken; it's not a Bazel thing; looks like it's this line that inserts the empty build file:

https://github.com/bazelbuild/rules_python/blob/master/python/pip_install/pip_repository.bzl#L15

I'll open a new PR with the proper fix...

@person142
Copy link
Copy Markdown
Contributor Author

#457 should be a better fix.

@person142 person142 closed this Apr 17, 2021
@person142 person142 deleted the pip-parsed-no-build branch April 17, 2021 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants