Support NVCC '-ccbin' argument by colesbury · Pull Request #145 · ccache/ccache

colesbury · 2016-11-01T18:29:34Z

Fixes #144

jrosdahl · 2017-02-08T20:59:48Z

Thanks, looks reasonable. However, I see an edge case: If -ccbin specifies a directory and compiler_check is content then the content of the directory node is hashed. I guess it works, but is it the wanted behavior?

How does NVCC find the compiler to run if -ccbin specifies a directory?

jeremad · 2017-05-12T09:01:41Z

If this help in any way, it works like a charm on my big complicated CUDA project 👍

lukeyeager · 2017-06-09T00:02:46Z

I'm using this branch for Caffe2 builds, both locally and on TravisCI: facebookarchive/caffe2#735

I have requested CMake support for ccache+nvcc here, in anticipation of this being merged.
https://gitlab.kitware.com/cmake/cmake/issues/16953

Summary: Uncached build: https://travis-ci.org/lukeyeager/caffe2/builds/239677224 Cached build: https://travis-ci.org/lukeyeager/caffe2/builds/239686725 * Parallel builds everywhere * All builds use CCache for quick build times (help from pytorch/pytorch#614, ccache/ccache#145) * Run ctests when available (continuation of #550) * Upgraded from cuDNN v5 to v6 * Fixed MKL build (by updating pkg version) * Fixed android builds (b6f905a#commitcomment-22404119) * ~~Building NNPACK fails with no discernible error message (currently disabled entirely)~~ * ~~Android builds continue to fail with existing error:~~ * ~~OSX builds time-out:~~ | Before | After | Changes | | --- | --- | --- | | COMPILER=g++ | linux | without CUDA | | COMPILER=g++-5 | linux-gcc5 | without CUDA | | COMPILER=g++ | linux-cuda | updated to cuDNN v6 | | BLAS=MKL | linux-mkl | updated pkg version | | BUILD_TARGET=android | linux-android | | | COMPILER=clang++ | osx | | | BUILD_TARGET=ios | osx-ios | | | BUILD_TARGET=android | osx-android | | | QUICKTEST | **GONE** | | | COMPILER=g++-4.8 | **GONE** | | | COMPILER=g++-4.9 | **GONE** | | Closes #735 Reviewed By: Yangqing Differential Revision: D5228966 Pulled By: bwasti fbshipit-source-id: 6cfa6f5ff05fbd5c2078beea79564f1f3b9812fe

lukeyeager · 2017-06-16T22:04:32Z

CMake now supports -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache. It will be in v3.9.
https://gitlab.kitware.com/cmake/cmake/merge_requests/949

Thanks @bradking!

bradking · 2017-06-19T15:29:22Z

It will be in v3.9.

Actually it will be in CMake 3.10 (and is already available in nightly binaries). We're already in the release candidate cycle for CMake 3.9 so it's too late for new features in general.

colesbury · 2017-06-30T16:49:39Z

@lukeyeager, do you know how nvcc chooses the compiler if -ccbin is a directory?

lukeyeager · 2017-07-05T16:23:47Z

Here's the --help information for that flag with the CUDA 8.0 compiler:

--compiler-bindir <path>                   (-ccbin)                          
        Specify the directory in which the host compiler executable resides.  The
        host compiler executable name can be also specified to ensure that the correct
        host compiler is selected.  In addition, driver prefix options ('--input-drive-prefix',
        '--dependency-drive-prefix', or '--drive-prefix') may need to be specified,
        if nvcc is executed in a Cygwin shell or a MinGW shell on Windows.

I'm not sure what that means, exactly. Let me go find one of the compiler guys ...

lukeyeager · 2017-07-05T16:36:12Z

When you pass a directory to -ccbin, nvcc is going to look for a binary matching the name of the standard compiler on your platform in that directory (gcc on linux, probably clang on osx, and whatever the name of the binary is for the msvc compiler on windows). I believe that without the -ccbin flag, nvcc looks for the same binary, but looks for it in $PATH.

But, as @bas-aarts pointed out, the compiler is often dynamically linked to libraries outside of that directory, and may even call out to other backend executables, etc. So just getting a checksum of the binary isn't strictly safe. How does ccache typically handle this? It might be a good enough solution to just use the last modified date of the directory.

bas-aarts · 2017-07-05T17:54:04Z

Assuming no-one intentionally messes with their compiler installation, a checksum on the version string should suffice.

afbjorklund · 2017-07-05T20:00:43Z

Sounds like this flag could be related to the $CCACHE_PATH, then ?

You can use a custom command to $CCACHE_COMPILERCHECK,
if you feel that mtime (or content, if paranoid) is not enough a check.

lukeyeager · 2017-07-10T21:45:22Z

Seems like this strategy should be sufficient:

if [ -f "$ccbin" ]; then
    # File: md5sum
    ccache_hash="$(md5sum "$ccbin" | awk '{print $1}')"
elif [ -d "$ccbin" ]; then
    # Directory: mtime
    ccache_hash="$(stat -c %Y "$ccbin")"
else
    echo "'$ccbin' is not a file or directory"
    exit 1
fi

colesbury · 2017-08-18T16:28:23Z

@jrosdahl, what do you think the best strategy is? Should we:

keep the behavior in the PR (hash the directory node)
guess the which compiler NVCC will use (gcc/clang/msvc)
bail out if -ccbin refers to a directory

jrosdahl · 2018-01-11T14:15:46Z

@jrosdahl, what do you think the best strategy is? Should we:

keep the behavior in the PR (hash the directory node)

guess the which compiler NVCC will use (gcc/clang/msvc)

bail out if -ccbin refers to a directory

I'd say 2 or 3. Would it be hard to guess the compiler?

jrosdahl · 2018-01-31T20:26:06Z

This PR says "Fixes #144" which is now closed since #193 also fixed #144, but the hashing part of this PR was not included in #193. So I guess this PR is still partly relevant? @colesbury + @seeraven: Correct?

seeraven · 2018-02-01T04:43:52Z

@jrosdahl: Yes that's right.

See #145.

jrosdahl · 2018-02-03T20:19:28Z

Applied the remaining parts now, thanks!

Also, I took a shot at option 2 discussed previously ("guess the which compiler NVCC will use (gcc/clang/msvc)") in 9c051cc. @colesbury and/or @seeraven: I would appreciate it if you could review my commit.

jrosdahl · 2018-02-08T21:09:44Z

Closing this now. (A review of my commit would still be appreciated...)

Summary: Uncached build: https://travis-ci.org/lukeyeager/caffe2/builds/239677224 Cached build: https://travis-ci.org/lukeyeager/caffe2/builds/239686725 * Parallel builds everywhere * All builds use CCache for quick build times (help from pytorch/pytorch#614, ccache/ccache#145) * Run ctests when available (continuation of facebookarchive/caffe2#550) * Upgraded from cuDNN v5 to v6 * Fixed MKL build (by updating pkg version) * Fixed android builds (facebookarchive/caffe2@b6f905a#commitcomment-22404119) * ~~Building NNPACK fails with no discernible error message (currently disabled entirely)~~ * ~~Android builds continue to fail with existing error:~~ * ~~OSX builds time-out:~~ | Before | After | Changes | | --- | --- | --- | | COMPILER=g++ | linux | without CUDA | | COMPILER=g++-5 | linux-gcc5 | without CUDA | | COMPILER=g++ | linux-cuda | updated to cuDNN v6 | | BLAS=MKL | linux-mkl | updated pkg version | | BUILD_TARGET=android | linux-android | | | COMPILER=clang++ | osx | | | BUILD_TARGET=ios | osx-ios | | | BUILD_TARGET=android | osx-android | | | QUICKTEST | **GONE** | | | COMPILER=g++-4.8 | **GONE** | | | COMPILER=g++-4.9 | **GONE** | | Closes facebookarchive/caffe2#735 Reviewed By: Yangqing Differential Revision: D5228966 Pulled By: bwasti fbshipit-source-id: 6cfa6f5ff05fbd5c2078beea79564f1f3b9812fe

Support NVCC '-ccbin' argument

f367e43

jrosdahl added the issue: feature New feature label Feb 8, 2017

lukeyeager mentioned this pull request Jun 5, 2017

TravisCI Overhaul facebookarchive/caffe2#735

Closed

seeraven mentioned this pull request Aug 27, 2017

Bugfixes and Tests for nvcc #193

Closed

jrosdahl added this to the 3.4 milestone Feb 3, 2018

jrosdahl added a commit that referenced this pull request Feb 3, 2018

Hash host compilers invoked by nvcc

9c051cc

See #145.

jrosdahl closed this Feb 8, 2018

ezyang mentioned this pull request Oct 31, 2018

Disable CCACHE while building NCCL pytorch/pytorch#13340

Closed

Conversation

colesbury commented Nov 1, 2016

Uh oh!

jrosdahl commented Feb 8, 2017

Uh oh!

jeremad commented May 12, 2017

Uh oh!

lukeyeager commented Jun 9, 2017

Uh oh!

lukeyeager commented Jun 16, 2017

Uh oh!

bradking commented Jun 19, 2017

Uh oh!

colesbury commented Jun 30, 2017

Uh oh!

lukeyeager commented Jul 5, 2017

Uh oh!

lukeyeager commented Jul 5, 2017

Uh oh!

bas-aarts commented Jul 5, 2017

Uh oh!

afbjorklund commented Jul 5, 2017

Uh oh!

lukeyeager commented Jul 10, 2017

Uh oh!

colesbury commented Aug 18, 2017

Uh oh!

jrosdahl commented Jan 11, 2018

Uh oh!

jrosdahl commented Jan 31, 2018

Uh oh!

seeraven commented Feb 1, 2018

Uh oh!

jrosdahl commented Feb 3, 2018

Uh oh!

jrosdahl commented Feb 8, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants