Bazel by stgpetrovic · Pull Request #4528 · pytorch/xla

stgpetrovic · 2023-01-27T23:01:41Z

Bazel integration for PyTorch/XLA's libxla_computation_client.so.

This is a large change but should have no effect on the build.

Work done:

bazel WORKSPACE created
tensorflow moved to a external dependency, patched and pinned via WORKSPACE
bazelrc ported to ensure equivalent build
BUILD file reworked to include files form external tesorflow dependency
includes reworked to match new file locations
includes in torch_xla/crsc rewored too
setup modified to find files in new locations and build new targets
integration for bazel and clangd for development support
integration for bazel for coverage for calculating and rendering coverage
sample tests written to demonstrate coverage
fixes to code to remove warnings (and warning suppressions)

This enables for:

simpler programming since the code is not copied in tensorflow tree
visual studio code references
visual studio autocompletion
visual studio refactoring and smart navigation
faster rebuild times
deterministic builds with simple bazel build, no shell or the like

Future work:

migrate torch_xla to bazel too to get all perks named above for the whole codebase
support for remote build execution so that build times can be much faster
untangling certain parts of code, splitting into smaller compilation units, for incremental build speed
and more

will-cromar

This is awesome! Thanks so much for your hard work on this

JackCaoG · 2023-01-30T22:34:54Z

  bazel clean
-  popd
-else
-  # Overlay llvm-raw secondary cache. The remote cache should be updated


@yeounoh regarding removing this secondary cahce, I think we added to make upstream CI more stable.

Not sure what to do here. We can patch the TF repo with this change.

I think that llvm tensorflow external repository already names both of these URLs so bazel will fetch from both and there's no need to do this anymore.

Not 100% sure but it looks like it.

wdyt?

I've patched in TF with this 2 sed equivalents so we have it as a patch, not as a in-place replacement.

miladm · 2023-01-31T19:50:40Z

CC @ManfeiBai

JackCaoG · 2023-01-31T23:13:09Z

@stgpetrovic The branch cut for the pytorch 2.0 is scheduled to be this Friday, I am a bit hesitant to merge this one right before the branch cut. Do you have any opinion?

stgpetrovic · 2023-02-01T09:20:29Z

I think that we should first cut and then we have a large merge window for this large change; something's bound to explode here.

This reverts commit 3a90f5a.

* Fix HLO dumping (#4619) * Update TF pin to 2/13 (#4615) * Update TF pin to 2/13 * Fix pinned commit * Add patch to revert TF 3e24055 * Add comment to new patch * Fix patch command in TPU CI (#4623) * Skip execution for extract_compiled_graph (#4612) * Only warm up cache for dynamo extract_graph step * Add missing config * Make sure warm up run does not cause place holder to be created * Fix tests * Disable failing `test_operations.py` tests on TPU (#4622) * Disable `test_operations.py` tests failing on TPU * Add to TPU CI * Bazel (#4528) * Replace tensorflow with a bazel external repository * Basic migration to bazel for xla_client. * Revert to blob * Add vscode config. * Update newlines * Merge with pjrt client test build changes. * Migrate tests to new build * Format test and plugin * Order imports * Conditionally apply tf patches; apply pt patches always. * Format python * configure formatters * Mirror TF pin update an fixes in bazel. * Support local and sandboxed build based on flags * Add cloud cache URLs for llvm. * Merge with upstream * Update TF pin * Fix patching regression * Revert "Bazel (#4528)" (#4631) This reverts commit 3a90f5a. --------- Co-authored-by: JackCaoG <59073027+JackCaoG@users.noreply.github.com> Co-authored-by: Will Cromar <wcromar@google.com> Co-authored-by: stgpetrovic <stgpetrovic@gmail.com>

* Replace tensorflow with a bazel external repository * Basic migration to bazel for xla_client. * Revert to blob * Add vscode config. * Update newlines * Merge with pjrt client test build changes. * Migrate tests to new build * Format test and plugin * Order imports * Conditionally apply tf patches; apply pt patches always. * Format python * configure formatters * Mirror TF pin update an fixes in bazel. * Support local and sandboxed build based on flags * Add cloud cache URLs for llvm. * Merge with upstream * Update TF pin * Fix patching regression

This reverts commit 3a90f5a.

* Replace tensorflow with a bazel external repository * Basic migration to bazel for xla_client. * Revert to blob * Add vscode config. * Update newlines * Merge with pjrt client test build changes. * Migrate tests to new build * Format test and plugin * Order imports * Conditionally apply tf patches; apply pt patches always. * Format python * configure formatters * Mirror TF pin update an fixes in bazel. * Support local and sandboxed build based on flags * Add cloud cache URLs for llvm. * Merge with upstream * Update TF pin * Fix patching regression

This reverts commit 3a90f5a.

* Replace tensorflow with a bazel external repository * Basic migration to bazel for xla_client. * Revert to blob * Add vscode config. * Update newlines * Merge with pjrt client test build changes. * Migrate tests to new build * Format test and plugin * Order imports * Conditionally apply tf patches; apply pt patches always. * Format python * configure formatters * Mirror TF pin update an fixes in bazel. * Support local and sandboxed build based on flags * Add cloud cache URLs for llvm. * Merge with upstream * Update TF pin * Fix patching regression

This reverts commit 3a90f5a.

stgpetrovic requested a review from will-cromar January 27, 2023 23:01

miladm self-requested a review January 30, 2023 18:49

miladm assigned stgpetrovic Jan 30, 2023

will-cromar reviewed Jan 30, 2023

View reviewed changes

Comment thread .bazelversion

Comment thread WORKSPACE

Comment thread build_torch_xla_libs.sh

Comment thread WORKSPACE

will-cromar requested a review from JackCaoG January 30, 2023 21:05

JackCaoG reviewed Jan 30, 2023

View reviewed changes

Comment thread test/cpp/cpp_test_util.h

stgpetrovic force-pushed the bazel branch 2 times, most recently from 5a48f3c to 81912ed Compare January 31, 2023 10:49

miladm reviewed Jan 31, 2023

View reviewed changes

Comment thread .bazelrc

miladm reviewed Jan 31, 2023

View reviewed changes

Comment thread third_party/xla_client/pjrt_computation_client_test.cc

JackCaoG mentioned this pull request Jan 31, 2023

[OpenXLA]Migrate all codes to use XLA from OpenXLA #4543

Closed

stgpetrovic force-pushed the bazel branch 3 times, most recently from f606826 to 6438e58 Compare February 7, 2023 12:49

stgpetrovic force-pushed the bazel branch 4 times, most recently from 7e2d557 to 7aa34be Compare February 14, 2023 17:44

stgpetrovic added 8 commits February 14, 2023 19:05

Replace tensorflow with a bazel external repository

7eeb625

Basic migration to bazel for xla_client.

713db2f

Revert to blob

95fddbb

Add vscode config.

452da42

Update newlines

8c010a4

Merge with pjrt client test build changes.

037dc5b

Migrate tests to new build

113edaa

Format test and plugin

d9c53d2

stgpetrovic added 9 commits February 14, 2023 19:05

Order imports

287be86

Conditionally apply tf patches; apply pt patches always.

28be564

Format python

cbfde7f

configure formatters

89a56c4

Mirror TF pin update an fixes in bazel.

a843fc8

Support local and sandboxed build based on flags

b1d71f2

Add cloud cache URLs for llvm.

b6924cd

Merge with upstream

f795d30

Update TF pin

ca26955

stgpetrovic force-pushed the bazel branch from 7aa34be to ca26955 Compare February 14, 2023 19:09

Fix patching regression

6165a84

stgpetrovic merged commit 3a90f5a into master Feb 15, 2023

stgpetrovic deleted the bazel branch February 15, 2023 22:34

JackCaoG added a commit that referenced this pull request Feb 15, 2023

Revert "Bazel (#4528)"

5d6ac12

This reverts commit 3a90f5a.

JackCaoG mentioned this pull request Feb 15, 2023

Revert "Bazel" #4631

Merged

JackCaoG added a commit that referenced this pull request Feb 15, 2023

Revert "Bazel (#4528)" (#4631)

286ef02

This reverts commit 3a90f5a.

chandrasekhard2 pushed a commit that referenced this pull request Feb 22, 2023

Revert "Bazel (#4528)" (#4631)

17d80ff

This reverts commit 3a90f5a.

chandrasekhard2 pushed a commit that referenced this pull request Feb 22, 2023

Revert "Bazel (#4528)" (#4631)

c7ae2e5

This reverts commit 3a90f5a.

mateuszlewko pushed a commit that referenced this pull request Mar 15, 2023

Revert "Bazel (#4528)" (#4631)

91a93c7

This reverts commit 3a90f5a.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bazel#4528

Bazel#4528
stgpetrovic merged 18 commits intomasterfrom
bazel

stgpetrovic commented Jan 27, 2023

Uh oh!

will-cromar left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JackCaoG Jan 30, 2023

Uh oh!

stgpetrovic Feb 1, 2023

Uh oh!

stgpetrovic Feb 8, 2023 •

edited

Loading

Uh oh!

stgpetrovic Feb 9, 2023

Uh oh!

Uh oh!

miladm commented Jan 31, 2023

Uh oh!

Uh oh!

Uh oh!

JackCaoG commented Jan 31, 2023

Uh oh!

stgpetrovic commented Feb 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

stgpetrovic commented Jan 27, 2023

Uh oh!

will-cromar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JackCaoG Jan 30, 2023

Choose a reason for hiding this comment

Uh oh!

stgpetrovic Feb 1, 2023

Choose a reason for hiding this comment

Uh oh!

stgpetrovic Feb 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stgpetrovic Feb 9, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miladm commented Jan 31, 2023

Uh oh!

Uh oh!

Uh oh!

JackCaoG commented Jan 31, 2023

Uh oh!

stgpetrovic commented Feb 1, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

stgpetrovic Feb 8, 2023 •

edited

Loading