Set up sccache inside the DockerFile. by Jorghi12 · Pull Request #9697 · pytorch/pytorch

Jorghi12 · 2018-07-23T02:09:27Z

Start setting up sccache inside of the DockerFile, instead of in the jenkins build script. Also created a fork of sccache that supports HCC, uploaded to AWS.
Allowing maximum swap space usage for ROCm builds to alleviate the LLVM out of memory issue we've been facing.

yf225 · 2018-07-23T02:28:49Z

Is there a way to call sccache --zero-stats before the build, and then call sccache --show-stats after the build so we can manually check and make sure cache hits are happening? I also noticed that there is report_compile_cache_stats function, but not sure if it is used by all the builds.

yf225

See comment

yf225 · 2018-07-23T03:00:32Z

I will start a Caffe2 docker image build based on the changes here.

By default the swap memory limit is set to 2 x the memory limit. Testing build using unlimited swap for ROCm.

facebook-github-bot

@Jorghi12 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Jorghi12 · 2018-07-24T00:39:16Z

Since this change is updating the DockerFile & the build script at once, the caffe2-docker trigger https://ci.pytorch.org/jenkins/job/caffe2-docker-trigger/148/ will be the source of truth.

Jorghi12 · 2018-07-24T02:32:06Z

The Caffe2 Docker Trigger passes all but one: https://ci.pytorch.org/jenkins/job/caffe2-docker-trigger/149/

Due to:
denied: The repository with name 'caffe2/py2-mkl-ubuntu16.04' in registry with id '308535385114' already has the maximum allowed number of images which is '10000'

@ezyang @yf225

E: Held packages were changed and -y was used without --allow-change-held-packages.

Trying without memory swap to test an issue.

Keep hip thrust.

docker/caffe2/jenkins/common/install_ccache.sh

+for compiler in "${compilers[@]}"; do
+  (
+    echo "#!/bin/sh"
+    echo "exec $SCCACHE $(which $compiler) \"\$@\""


docker/caffe2/jenkins/common/install_rocm.sh

 }

+# This will be removed after merging an upcoming PR.
 install_hcrng() {


docker/caffe2/jenkins/common/install_rocm.sh

-fi
-
 install_hip_thrust
 install_hcrng


docker/caffe2/jenkins/build.sh


 # Build image
-docker build \
+docker build $(if [[ "$image" == *rocm* ]]; then echo "--memory-swap -1"; fi) \


facebook-github-bot

@Jorghi12 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

ezyang · 2018-07-30T13:17:06Z

@pytorchbot retest this please

ezyang · 2018-07-30T13:17:16Z

Can we merge this?

ezyang · 2018-08-01T15:18:09Z

This must be landed with an update to the Docker images.

yf225 · 2018-08-21T16:53:42Z

@Jorghi12 Is there a new status on this one?

ezyang · 2018-10-11T06:56:55Z

This diff is abandoned.

Set up sccache inside the DockerFile

6723502

Jorghi12 requested a review from yf225 July 23, 2018 02:09

yf225 requested changes Jul 23, 2018

View reviewed changes

yf225 approved these changes Jul 23, 2018

View reviewed changes

Jorghi12 added 3 commits July 23, 2018 01:51

Keeping sccache in build script.

cd163e8

Remove extra var.

192b31b

Testing usage of swap space for ROCm builds.

4c6db7d

By default the swap memory limit is set to 2 x the memory limit. Testing build using unlimited swap for ROCm.

facebook-github-bot reviewed Jul 24, 2018

View reviewed changes

Going to use caffe2 docker trigger as source of truth.

caf502c

Remove unnecessary patches.

79b38c2

Jorghi12 added 3 commits July 23, 2018 22:36

Merge branch 'master' into caffe2_sccache

c9a540f

revert hcsparse

dcb964d

Adding patch for HIP bug.

658851b

Jorghi12 mentioned this pull request Jul 25, 2018

Correctly set up sscache inside of the Caffe2 Docker Images. ROCm/pytorch#65

Closed

Handling held package error.

7f23140

E: Held packages were changed and -y was used without --allow-change-held-packages.

Jorghi12 requested a review from bddppq July 25, 2018 02:50

Jorghi12 added 4 commits July 24, 2018 22:59

Update build.sh

3bdcddb

Trying without memory swap to test an issue.

Update install_rocm.sh

1a2dbb7

Keep hip thrust.

Enabling memory swap.

b8d06b5

Fixing typo..

fecafa3

bddppq reviewed Jul 25, 2018

View reviewed changes

ezyang reviewed Jul 25, 2018

View reviewed changes

docker/caffe2/jenkins/build.sh Outdated

# Build image

docker build \

docker build $(if [[ "$image" == *rocm* ]]; then echo "--memory-swap -1"; fi) \

This comment was marked as off-topic.

Sign in to view

Jorghi12 added 4 commits July 25, 2018 11:34

Making sure to install sccache after compilers are installed.

d103e61

Installing hcsparse

e99f4c6

Update build.sh

065cc2b

Merge branch 'master' into caffe2_sccache

ad644f9

The package "hcsparse" wasn't found in packages.

76b6b72

facebook-github-bot reviewed Jul 26, 2018

View reviewed changes

ailzhang added the ready for review (this tag is deprecated) All PRs are ready for review unless they are draft, WIP, or have undismissed requested changes label Jul 31, 2018

ezyang closed this Oct 11, 2018

ezyang added the open source label Jun 24, 2019

Conversation

Jorghi12 commented Jul 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yf225 commented Jul 23, 2018

Uh oh!

yf225 left a comment

Choose a reason for hiding this comment

Uh oh!

yf225 commented Jul 23, 2018

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

Jorghi12 commented Jul 24, 2018

Uh oh!

Jorghi12 commented Jul 24, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

ezyang commented Jul 30, 2018

Uh oh!

ezyang commented Jul 30, 2018

Uh oh!

ezyang commented Aug 1, 2018

Uh oh!

yf225 commented Aug 21, 2018

Uh oh!

ezyang commented Oct 11, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Jorghi12 commented Jul 23, 2018 •

edited

Loading

Jorghi12 commented Jul 24, 2018 •

edited

Loading