Bazel build of pytorch by werkt · Pull Request #35220 · pytorch/pytorch

werkt · 2020-03-23T18:36:07Z

No description provided.

dr-ci · 2020-03-23T18:38:24Z

💊 CircleCI build failures summary and remediations

As of commit 1e84fb8 (more details on the Dr. CI page):

✅ None of the build failures appear to be your fault 💚

2/2 broken upstream at merge base 2b068d1 on Apr 01 from 9:47am to 12:34pm PDT (9 commits; a736b99 - 990b541)
Please rebase on the viable/strict branch (expand for instructions)

If your commit is newer than viable/strict, you can try basing on an older, stable commit:
```
git fetch https://github.com/pytorch/pytorch viable/strict
git rebase --onto FETCH_HEAD $(git merge-base origin/master HEAD)
```
If your commit is older than viable/strict:
```
git fetch https://github.com/pytorch/pytorch viable/strict
git rebase FETCH_HEAD
```
Check out the recency history of this "viable master" tracking branch.

🚧 2 upstream failures:

These were probably caused by upstream breakages:

pytorch_xla_linux_xenial_py3_6_clang7_test since Mar 31
pytorch_windows_vs2019_py36_cuda10.1_test2 on Apr 01 from 9:47am to 12:34pm PDT (9 commits; a736b99 - 990b541)

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker.

See how this bot performed.

This comment has been revised 118 times.

albanD · 2020-03-24T17:44:20Z

Just adding reviewers to make sure it is looked at by someone. Feel free to remove yourself if someone else is already looking into it.

malfet · 2020-03-25T04:51:29Z

Can the list of srcs be generated by glob?

Suggested change

"src/FbgemmFP16UKernelsAvx2.cc",

"src/FbgemmFloat16ConvertAvx2.cc",

"src/FbgemmI8Depthwise3DAvx2.cc",

"src/FbgemmI8Depthwise3x3Avx2.cc",

"src/FbgemmI8DepthwiseAvx2.cc",

"src/FbgemmI8DepthwisePerChannelQuantAvx2.cc",

"src/OptimizedKernelsAvx2.cc",

"src/PackDepthwiseConvMatrixAvx2.cc",

"src/QuantUtilsAvx2.cc",

"src/UtilsAvx2.cc",

# Private headers

"src/FbgemmFP16Common.h",

"src/FbgemmFP16UKernelsAvx2.h",

"src/FbgemmI8Depthwise2DAvx2-inl.h",

"src/FbgemmI8DepthwiseAvx2-inl.h",

"src/MaskAvx2.h",

"src/OptimizedKernelsAvx2.h",

"src/TransposeUtils.h",

"src/TransposeUtilsAvx2.h",

glob("src/*Avx2.*") + [

"src/TransposeUtils.h",

],

They can, but I was seeking to emulate what fbgemm does as a part of its build.

nicolov · 2020-03-29T05:43:34Z

Thanks a lot for your efforts upstreaming this. Are you interested in patches/feedback in the case where pytorch is included as a third-party repository, or are you planning to first merge a working standalone build, and then make changes?

werkt · 2020-03-29T16:11:22Z

Thanks a lot for your efforts upstreaming this. Are you interested in patches/feedback in the case where pytorch is included as a third-party repository, or are you planning to first merge a working standalone build, and then make changes?

Merge a working copy that prevents breakages. We (Uber) are doing this because we have the same use case, and will be working towards novel third party inclusion utilities after it has landed. This is just to get a foot in the door.

nicolov · 2020-03-29T17:37:04Z

Merge a working copy that prevents breakages.

Perfect, makes sense. Are you planning to include CUDA support in the first round?

werkt · 2020-03-29T18:40:36Z

No, cuda is coming later. We use this internally for it, but the FB torch team was onboard for cpu first. Hermetic cuda (the goal) requires a toolchain, which we haven't formalized for release. You can see some snippets of our cuda definitions in these BUILD files for now

nicolov · 2020-03-29T21:26:35Z

No, cuda is coming later.

Cool, thanks. I'm migrating our internal BUILD files for pytorch to this, and was wondering whether to wait or not. I'm guessing you can import the CUDA-on-clang toolchain from TensorFlow?

werkt · 2020-03-30T03:17:29Z

No, cuda is coming later.

Cool, thanks. I'm migrating our internal BUILD files for pytorch to this, and was wondering whether to wait or not. I'm guessing you can import the CUDA-on-clang toolchain from TensorFlow?

Negative. The clang-primary cuda compilation does not work, nvcc must be used as a frontend for cuda compilations with pytorch. We investigated that early on and could not make it work.

nicolov · 2020-03-30T03:30:51Z

We investigated that early on and could not make it work.

What types of errors were you getting? I do have it working with clang 9 and CUDA 10 (modulo a few hacks required to get rid of compilation errors due to the different host/device compilation model).

werkt · 2020-03-30T04:05:06Z

We investigated that early on and could not make it work.

What types of errors were you getting? I do have it working with clang 9 and CUDA 10.

"have it working" can you elaborate? You are currently building all pytorch cuda sources with a device target with clang as the exec'd compiler frontend, not with clang as the host cc and nvcc as the frontend?

nicolov · 2020-03-30T04:36:19Z

"have it working" can you elaborate?

We don't use nvcc at all, just the clang PTX backend directly. IIRC there's a TensorFlow configuration that does the same.

werkt · 2020-03-30T14:22:15Z

We don't use nvcc at all, just the clang PTX backend directly. IIRC there's a TensorFlow configuration that does the same.

Taking this conversation offline.

nicolov · 2020-03-31T04:07:11Z

For reference, these are the (small) changes needed to make this work as an external repo (also supports remote execution - that's why I had to add a bunch of Python files to the gen script). https://gist.github.com/nicolov/09af3c1a8c7dcf971822daf8f2dc9caf

nicolov · 2020-03-31T23:07:00Z

+        "c10/cuda/impl/cuda_cmake_macros.h",
+    ],
+    deps = [
+        "@com_github_gflags_gflags//:gflags",


I wonder how you can get by with adding the complete gflags and glog libraries here. c10_headers in linked in one of the .sos that have linkstatic, and my build complains about duplicate flags definitions.

facebook-github-bot

@seemethere has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

seemethere · 2020-04-01T16:32:53Z

@werkt sorry to do this to you but it appears as though we need a rebase.

werkt · 2020-04-01T17:40:08Z

@werkt sorry to do this to you but it appears as though we need a rebase.

Not a problem. The end of that test clause changed twice in the last hour :(

facebook-github-bot

@malfet has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Preliminary step to merge pytorch#35220 Test Plan: CI

Summary: Preliminary step to merge #35220 Pull Request resolved: #35924 Test Plan: CI Differential Revision: D20832159 Pulled By: malfet fbshipit-source-id: 29ff2e3c04c08c39c49f35414f94b76f0651859a

…pytorch#35951) Summary: Pull Request resolved: pytorch#35951 Change generate_code to keep folder structure the same regardless of whether install path is provide Amend build_variables.bzl accordingly Another preliminary step to merge pytorch#35220 Test Plan: CI Differential Revision: D20839410 fbshipit-source-id: 923ce994f2847421c23ded59ac315deadd708b4e

…#35951) Summary: Pull Request resolved: #35951 Change generate_code to keep folder structure the same regardless of whether install path is provide Amend build_variables.bzl accordingly Another preliminary step to merge #35220 Test Plan: CI Reviewed By: EscapeZero, seemethere Differential Revision: D20839410 fbshipit-source-id: 02297560a7e48aa7c6271f7a8517fc4a1ab35271

facebook-github-bot · 2020-04-04T06:13:26Z

@malfet merged this pull request in 585f153.

Summary: Pull Request resolved: pytorch#35220 Reviewed By: seemethere Differential Revision: D20783179 Pulled By: malfet fbshipit-source-id: b160908a3e107790fa06057a77de9d6d23493bbc

Summary: Preliminary step to merge pytorch#35220 Pull Request resolved: pytorch#35924 Test Plan: CI Differential Revision: D20832159 Pulled By: malfet fbshipit-source-id: 29ff2e3c04c08c39c49f35414f94b76f0651859a

…pytorch#35951) Summary: Pull Request resolved: pytorch#35951 Change generate_code to keep folder structure the same regardless of whether install path is provide Amend build_variables.bzl accordingly Another preliminary step to merge pytorch#35220 Test Plan: CI Reviewed By: EscapeZero, seemethere Differential Revision: D20839410 fbshipit-source-id: 02297560a7e48aa7c6271f7a8517fc4a1ab35271

Summary: Pull Request resolved: pytorch#35220 Reviewed By: seemethere Differential Revision: D20783179 Pulled By: malfet fbshipit-source-id: b160908a3e107790fa06057a77de9d6d23493bbc

pytorchbot added the open source label Mar 23, 2020

werkt force-pushed the bazel branch from aa0ef13 to 8111acf Compare March 23, 2020 20:59

albanD requested review from kostmo and malfet March 24, 2020 17:43

albanD added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Mar 24, 2020

albanD removed the request for review from kostmo March 24, 2020 17:51

malfet reviewed Mar 25, 2020

View reviewed changes

dzhulgakov mentioned this pull request Mar 26, 2020

Share build file lists between CMake, Bazel and Buck builds #35456

Closed

malfet reviewed Mar 26, 2020

View reviewed changes

Comment thread third_party/cpuinfo.BUILD Outdated

malfet approved these changes Mar 26, 2020

View reviewed changes

driazati reviewed Mar 26, 2020

View reviewed changes

Comment thread WORKSPACE Outdated

seemethere reviewed Mar 27, 2020

View reviewed changes

Comment thread tools/setup_helpers/generate_code.py Outdated

Comment thread .circleci/config.yml Outdated

werkt requested a review from apaszke as a code owner March 28, 2020 00:09

werkt force-pushed the bazel branch from cfe6eee to 1e9698a Compare March 28, 2020 17:21

Bazel build of pytorch with gating CI

108c33c

werkt force-pushed the bazel branch from b0d5166 to 108c33c Compare March 30, 2020 16:12

nicolov reviewed Mar 31, 2020

View reviewed changes

seemethere approved these changes Mar 31, 2020

View reviewed changes

facebook-github-bot reviewed Mar 31, 2020

View reviewed changes

George Gensure added 4 commits April 1, 2020 13:18

Merge branch 'master' into bazel

48645de

Remove non-ppc pip_install tests from bazel test

94469e8

Remove ineffectual (for gcc) -Wno-everything flag

cec2ee8

Merge branch 'master' into bazel

1e84fb8

facebook-github-bot reviewed Apr 3, 2020

View reviewed changes

malfet mentioned this pull request Apr 3, 2020

Use full include path in autogenerated Functions.cpp #35924

Closed

malfet added a commit to malfet/pytorch that referenced this pull request Apr 3, 2020

Use full include path in autogenerated Functions.cpp

a8141b8

Preliminary step to merge pytorch#35220 Test Plan: CI

malfet mentioned this pull request Apr 3, 2020

Keep same autogenerated files structure between fbcode and OSS builds #35951

Closed

facebook-github-bot closed this in 585f153 Apr 4, 2020

facebook-github-bot added the merged label Apr 4, 2020

malfet mentioned this pull request Apr 6, 2020

Bazel build of pytorch with gating CI #36011

Closed

mruberry added the Merged label Oct 28, 2020

-        "src/FbgemmFP16UKernelsAvx2.cc",
-        "src/FbgemmFloat16ConvertAvx2.cc",
-        "src/FbgemmI8Depthwise3DAvx2.cc",
-        "src/FbgemmI8Depthwise3x3Avx2.cc",
-        "src/FbgemmI8DepthwiseAvx2.cc",
-        "src/FbgemmI8DepthwisePerChannelQuantAvx2.cc",
-        "src/OptimizedKernelsAvx2.cc",
-        "src/PackDepthwiseConvMatrixAvx2.cc",
-        "src/QuantUtilsAvx2.cc",
-        "src/UtilsAvx2.cc",
-        # Private headers
-        "src/FbgemmFP16Common.h",
-        "src/FbgemmFP16UKernelsAvx2.h",
-        "src/FbgemmI8Depthwise2DAvx2-inl.h",
-        "src/FbgemmI8DepthwiseAvx2-inl.h",
-        "src/MaskAvx2.h",
-        "src/OptimizedKernelsAvx2.h",
-        "src/TransposeUtils.h",
-        "src/TransposeUtilsAvx2.h",
+        glob("src/*Avx2.*") + [
+          "src/TransposeUtils.h",
+        ],

Conversation

werkt commented Mar 23, 2020

Uh oh!

dr-ci Bot commented Mar 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💊 CircleCI build failures summary and remediations

🚧 2 upstream failures:

Uh oh!

albanD commented Mar 24, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

malfet Mar 25, 2020

Choose a reason for hiding this comment

Uh oh!

werkt Mar 25, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nicolov commented Mar 29, 2020

Uh oh!

werkt commented Mar 29, 2020

Uh oh!

nicolov commented Mar 29, 2020

Uh oh!

werkt commented Mar 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nicolov commented Mar 29, 2020

Uh oh!

werkt commented Mar 30, 2020

Uh oh!

nicolov commented Mar 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

werkt commented Mar 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nicolov commented Mar 30, 2020

Uh oh!

werkt commented Mar 30, 2020

Uh oh!

nicolov commented Mar 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nicolov Mar 31, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

seemethere commented Apr 1, 2020

Uh oh!

werkt commented Apr 1, 2020

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Apr 4, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

dr-ci Bot commented Mar 23, 2020 •

edited

Loading

werkt commented Mar 29, 2020 •

edited

Loading

nicolov commented Mar 30, 2020 •

edited

Loading

werkt commented Mar 30, 2020 •

edited

Loading

nicolov commented Mar 31, 2020 •

edited

Loading

nicolov Mar 31, 2020 •

edited

Loading