💊 CI failures summary and remediations

As of commit d068b7a (more details on the Dr. CI page):

✅ None of the CI failures appear to be your fault 💚

• 2/2 tentatively recognized as flaky ❄️
  • Click here to rerun these jobs

❄️ 2 failures tentatively classified as flaky

but reruns have not yet been triggered to confirm:

caffe2_onnx_main_py3_6_clang7_ubuntu16_04_build (1/2)

Step: "Build" (full log | diagnosis details | 🔁 rerun) ❄️

sys	0m0.060s 
Jun 29 23:01:19 ++ export BUILD_ENVIRONMENT=caffe2-onnx-main-py3.6-clang7-ubuntu16.04-build 
Jun 29 23:01:19 ++ BUILD_ENVIRONMENT=caffe2-onnx-main-py3.6-clang7-ubuntu16.04-build 
Jun 29 23:01:19 ++ git submodule sync 
Jun 29 23:01:19 ++ git submodule update -q --init --recursive 
Jun 29 23:01:44 error: RPC failed; curl 56 GnuTLS recv error (-54): Error in the pull function. 
Jun 29 23:01:44 fatal: The remote end hung up unexpectedly 
Jun 29 23:01:44 fatal: early EOF 
Jun 29 23:01:44 fatal: index-pack failed 
Jun 29 23:01:44 fatal: clone of 'https://github.com/intel/mkl-dnn.git' into submodule path 'mkl-dnn' failed 
Jun 29 23:01:44 Failed to recurse into submodule path 'third_party/ideep' 

binary_windows_libtorch_3_7_cpu_debug_build (2/2)

Step: "Build" (full log | diagnosis details | 🔁 rerun) ❄️

The system cannot find the file specified. 
Could Not Find C:\w\b\windows\miniconda.exe 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current 
                                 Dload  Upload   Total   Spent    Left  Speed 
   0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0 100 54.6M  100 54.6M    0     0  54.6M      0  0:00:01 --:--:--  0:00:01  205M 
Collecting package metadata (current_repodata.json): ...working... done 
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source. 
Collecting package metadata (repodata.json): ...working... done 
Solving environment: ...working... done 
 
CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://repo.anaconda.com/pkgs/main/win-64/mkl-2020.1-216.conda> 
Elapsed: - 
 
An HTTP error occurred when trying to retrieve this URL. 
HTTP errors are often intermittent, and a simple retry will get you on your way. 
 
 
 
## Package Plan ## 
 
  environment location: C:\w\b\windows\conda\envs\py37 

CI is failing because of merge conflicts when it tries to fast forward the branch. Looking into it now.

One thing to consider for CPU benchmarks (but may be hard to do properly): controlling for CPU throttling https://lemire.me/blog/2018/01/16/microbenchmarking-calls-for-idealized-conditions/ and maybe for CPU thread affinity

One thing to consider for CPU benchmarks (but may be hard to do properly): controlling for CPU throttling and maybe for CPU thread affinity

Yeah, it's a tough problem to be sure. I'm using the standard set of tricks to mitigate this:

1. Conduct a number of trials rather than one long run. (Generally tens to hundreds)
2. Use Median rather than mean for robustness to outliers / systematic throttling / etc.
3. Discard trials where too much variation is observed.
4. Trim visualization based on estimated significant figures.

I've found that measurements are fairly stable (at least on my machine), but I agree that it's definitely something to watch out for and this is not a panacea. The hope is that later versions will have proper runtime integration so we can get counts, allocations, etc.

This is pretty cool, I like nice APIs!

For CPU control it's hard to do so from within process. But we could provide a standard wrapper script that would turn off turbo (not sure whether there's a standard way to do it), pin to a single cpu and set thread scheduler priority to performance.

Thanks!

I agree, there doesn't seem to be any way to control the environment without spawning a subprocess. One concern would be overhead; given that most measurements tend to be short (< 1s), I worry that the vast majority of time would be spent creating and destroying these controlled envs. One could imagine keeping a pool of subprocesses and having Timers "submit" work to it, but you start to get into non-trivial engineering complexity vs just spacing out replicates.

I've updated the end-to-end example to test 3 PRs, as well as including a script to build the expected environments. The following were run on a 56 core, 8 P100 machine:

#39850 (cc @xwang233)

• summary
• Full results (GPU)

#39967 (cc @ShawnZhong)

• summary
• full results (CPU)
• full results (GPU)

#39744 (cc @nikitaved)

• summary
• full results (CPU)
• full results (GPU)

@nikitaved Thanks for pointing out that the GPU variation for your PR is quite high. (I can't seem to find the comment)
I re-ran it using the same environment for both before and after and saw similarly high levels of variation. After some experimentation, I found two things:

1. Running CPU and GPU at the same time significantly increases the variance (~ +/- 70% with vs. ~ +/- 30% without). I had limited the number of workers in the CPU pool, but htop indicates that I was still saturating the CPU at times so it was probably contending on the CPU portion. (e.g. stream management and op launch.)

2. Replicates are essential. I had this notion that GPU measurements were more stable than CPU, but empirically I had to go to 5 replicates to get down to < 5% variability. (The Timer does internal replicates, but those are clearly correlated either by device or temporally.)

Even then, you have to throw out almost 50% of the test cases because the variation is too high. These are basically all <75 us cases; there's just too much overhead jitter. Which is a shame. I think I'm going to experiment with having a worker do more measurements if it detects that a result is noisy to see if that lets us save some of those results.

It looks like the underlying issue is that the GPUs are not identical, so the variance was due to master and branch being scheduled on different devices. Running them on the same GPU significantly reduces GPU run-to-run variation. And for CPU, 5 replicates is sufficient to clamp down on the variance. (Though a non-trivial number still get culled due to excessive variation.) In order to make this easier to validate, I added a --test_variance option that runs both the before and after on the same environment, so any difference is noise.

#39744 Variance test
#39744 summary (new methodology)

The variance test shows that we're now within ~5%, and rerunning the actual PR shows no meaningful change for GPU (Yay!) and CPU change which is beyond noise, but much more muted than before indicating that the +/- 50% swings were noise rather than real changes, and when we clamp down on that noise the real signal is ~5-10% relative difference.

I'll rerun all of the PR benchmarks tomorrow.

Updated runs: (This is on an 8xV100 machine since I lost the 8xP100 machine and assignment is random.)
#39850

• Variance test (acceptable)
• Summary
• Full results (GPU)

#39967

• Variance test (acceptable)
• Summary
• Full results (CPU)
• Full results (GPU)

#39744

• Variance test (acceptable)
• Summary
• Full results (CPU)
• Full results (GPU)

@robieta , thanks for the update! When it comes to sorting, I observed that the TensorIterator might give a significant boost for large multidimensional contiguous tensors with the last dimensions sorted. Maybe it could be of any use as it might affect the performance of any dim-apply type of algorithm.

@robieta merged this pull request in f394979.