Completely remove build_aten and use_aten by mingzhe09088 · Pull Request #10469 · pytorch/pytorch

mingzhe09088 · 2018-08-13T16:56:31Z

Breaking out of #8338 to completely remove build_aten and use_aten.

orionr

One thing I'm debating is whether we want all of the ANDROID and IOS checks (if (NOT ANDROID and NOT IOS), etc.), but I think we are likely to incrementally remove these checks over time, so it's okay. I'm guessing all of these are needed for iOS and Android builds to pass?

mingzhe09088 · 2018-08-13T17:09:56Z

Yes. The checks for mobile is needed for Android and iOS builds to pass.

ezyang · 2018-08-15T03:24:46Z

It seems like an un-improvement to inline BUILD_ATEN with NOT ANDROID AND NOT IOS. Maybe it would be better to introduce something like MOBILE.

bddppq · 2018-08-18T05:26:05Z

@pytorchbot retest this please

bddppq · 2018-08-18T07:25:12Z

https://ci.pytorch.org/jenkins/job/caffe2-builds/job/py2-clang3.8-rocm1.7.1-ubuntu16.04-build/10441/console

06:45:13 [ 91%] Linking HIP shared library ../lib/libcaffe2_hip.so
06:45:14 objdump: CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_CUDAByteType.h.o: File format not recognized
06:45:14 objdump: CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_CUDACharType.h.o: File format not recognized
06:45:14 objdump: CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_CUDADoubleType.h.o: File format not recognized
06:45:14 objdump: CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_CUDAFloatType.h.o: File format not recognized
06:45:14 objdump: CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_CUDAGenerator.h.o: File format not recognized
06:45:14 objdump: CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_CUDAHalfType.h.o: File format not recognized
06:45:14 objdump: CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_CUDAIntType.h.o: File format not recognized
06:45:14 objdump: CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_CUDALongType.h.o: File format not recognized
06:45:14 objdump: CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_CUDAShortType.h.o: File format not recognized
06:45:14 objdump: CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_RegisterCUDA.h.o: File format not recognized
06:45:14 objdump: CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_SparseCUDAByteType.h.o: File format not recognized
06:45:14 objdump: CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_SparseCUDACharType.h.o: File format not recognized
06:45:14 objdump: CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_SparseCUDADoubleType.h.o: File format not recognized
06:45:14 objdump: CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_SparseCUDAFloatType.h.o: File format not recognized
06:45:14 objdump: CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_SparseCUDAIntType.h.o: File format not recognized
06:45:14 objdump: CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_SparseCUDALongType.h.o: File format not recognized
06:45:14 objdump: CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_SparseCUDAShortType.h.o: File format not recognized
07:18:20 CMakeFiles/caffe2_hip.dir/__/build_caffe2/aten/src/ATen/caffe2_hip_generated_CUDAByteType.h.o: file not recognized: File format not recognized
07:18:20 clang-7.0: error: linker command failed with exit code 1 (use -v to see invocation)
07:18:20 Died at /opt/rocm/hip/bin/hipcc line 565.

@iotamudelta @Jorghi12 Could you help taking a look?

bddppq · 2018-08-18T09:31:02Z

After investigation this is because ATen_CUDA_SRCS contains header files, and hip_add_library determines whether a file is header only based on property HEADER_FILE_ONLY and not look into the filename. Filtering out the headers from ATen_CUDA_SRCS has fixed the issue.

aten/src/ATen/CMakeLists.txt

.jenkins/pytorch/build.sh

cmake/Dependencies.cmake

mingzhe09088 · 2018-08-20T22:05:39Z

@pytorchbot retest this please

…ding

This reverts commit 8f988b6a9225502e9b9c22338ad473aac18e66f8.

This reverts commit 9b4a4446eeaa09e453a7be5f6294b0c733df86fc.

This reverts commit 9eef9fc8fd1a5497aec7236872d925443b6a47f0.

facebook-github-bot

mingzhe09088 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

bddppq · 2018-08-21T04:44:01Z

Summary: Breaking out of #8338 to completely remove build_aten and use_aten. Pull Request resolved: pytorch/pytorch#10469 Reviewed By: orionr Differential Revision: D9413639 Pulled By: mingzhe09088 fbshipit-source-id: b7203aa4f5f2bb95c504c8dc187a3167f2570183

* upstream/master: (147 commits) Support Loading to GPU (pytorch#10710) More changes for hidden visibility (pytorch#10692) Add arguments __repr__ in Distribution base class Add util function from core type to dtype (pytorch#10716) Make ONNX_ATEN_FALLBACK as internal default option Set the BUILD_ENVIRONMENT variable before installing sccache. (pytorch#10640) Avoid shadowing i, j vars in GeneralProposals test (pytorch#10721) Move THNN Reduction to ATen/core. (pytorch#10703) Completely remove build_aten and use_aten (pytorch#10469) Make empty list literals construct empty Tensor[] (pytorch#10705) Soumith's last few patches to v0.4.1 Fix issues link in Caffe2 readme (pytorch#10711) Update the onnx Gemm op to FC/FCTransposed logic in caffe2 onnx backend (pytorch#10108) Adding ATEN_NO_TEST option to root level cmake for propogation to aten Allow method-style casts on tensors (pytorch#10641) Fix pytorch#10698 build failure (pytorch#10704) Add support for Log() Add a bisect percentile operator (pytorch#10563) Fix EnsureCPUOutputOp (pytorch#10651) Nomnigraph - rename some APIs that invole Subtree to Subgraph (pytorch#10551) ...

Summary: Breaking out of pytorch#8338 to completely remove build_aten and use_aten. Pull Request resolved: pytorch#10469 Reviewed By: orionr Differential Revision: D9413639 Pulled By: mingzhe09088 fbshipit-source-id: b7203aa4f5f2bb95c504c8dc187a3167f2570183

mingzhe09088 requested a review from orionr August 13, 2018 16:56

mingzhe09088 requested review from apaszke, colesbury, ezyang, gchanan, soumith and zdevito as code owners August 13, 2018 16:56

orionr approved these changes Aug 13, 2018

View reviewed changes

bddppq reviewed Aug 19, 2018

View reviewed changes

aten/src/ATen/CMakeLists.txt Outdated

This comment was marked as off-topic.

Sign in to view

bddppq reviewed Aug 20, 2018

View reviewed changes

.jenkins/pytorch/build.sh Outdated

This comment was marked as off-topic.

Sign in to view

bddppq reviewed Aug 20, 2018

View reviewed changes

cmake/Dependencies.cmake Outdated

This comment was marked as off-topic.

Sign in to view

mingzhe09088 and others added 13 commits August 20, 2018 15:52

Completely remove build_aten and use_aten

8f359ce

also hipify pytorch files in caffe2 CI

6b3d20f

Fix aten issues with windows and rocm

d5e1eb0

test windows fix

a917f9d

limit build threads for rocm to avoid OOM issue

8548c04

Bump MAX_JOBS to 4 in rocm builds after swithcing to bigcpu CI machines

f66a0cb

Try removing MAX_JOBS restriction in rocm builds

2af0ad7

Filter out headers from ATen_CUDA_SRCS

6e6feb2

.

a5746b0

.

c00844e

Should add ATen_CUDA_SRCS to Caffe2_HIP_SRCS instead of simply overri…

11f5249

…ding

Remove MAX_JOBS restriction for pytorch rocm builds as well

9c02497

.

379a72d

bddppq and others added 15 commits August 20, 2018 15:52

.

7071ecf

This reverts commit 8f988b6a9225502e9b9c22338ad473aac18e66f8.

export USE_ROCM=1

19c7208

-DUSE_ROCM

c3f1747

Revert "export USE_ROCM=1"

745f99e

This reverts commit 9b4a4446eeaa09e453a7be5f6294b0c733df86fc.

Revert "-DUSE_ROCM"

d3cf8a1

This reverts commit 9eef9fc8fd1a5497aec7236872d925443b6a47f0.

.

b173b7f

.

90eb09f

ignore unresolved symbols for rocm builds

48d5149

.

33549aa

.

71e3cf6

.

93ac886

.

6dd3a4d

.

fa28636

introduce BUILD_ATEN_MOBILE

667697e

cleanup

2e93568

facebook-github-bot reviewed Aug 20, 2018

View reviewed changes

facebook-github-bot closed this in f0d8a36 Aug 21, 2018

ezyang added the merged label Jun 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Completely remove build_aten and use_aten#10469

Completely remove build_aten and use_aten#10469
mingzhe09088 wants to merge 28 commits intopytorch:masterfrom
mingzhe09088:remove-build-aten

mingzhe09088 commented Aug 13, 2018

Uh oh!

orionr left a comment

Uh oh!

mingzhe09088 commented Aug 13, 2018

Uh oh!

ezyang commented Aug 15, 2018

Uh oh!

bddppq commented Aug 18, 2018

Uh oh!

bddppq commented Aug 18, 2018

Uh oh!

bddppq commented Aug 18, 2018

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

mingzhe09088 commented Aug 20, 2018

Uh oh!

facebook-github-bot left a comment

Uh oh!

bddppq commented Aug 21, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

mingzhe09088 commented Aug 13, 2018

Uh oh!

orionr left a comment

Choose a reason for hiding this comment

Uh oh!

mingzhe09088 commented Aug 13, 2018

Uh oh!

ezyang commented Aug 15, 2018

Uh oh!

bddppq commented Aug 18, 2018

Uh oh!

bddppq commented Aug 18, 2018

Uh oh!

bddppq commented Aug 18, 2018

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

mingzhe09088 commented Aug 20, 2018

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

bddppq commented Aug 21, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants