Timings in the current situation (after some improvements were made)

taken from https://ci.docker.com/public/job/moby/job/master/53/flowGraphTable/ (master) and https://ci.docker.com/public/job/moby/7 (#39747) for Windows.

Duration in the "New" column is the total time to run the Job (which may include some setting up, printing docker info / docker version)

Currently, our moby/moby PR checks take about 2 hours. I think we can cut that in half with a bit of parallelization. The Jenkinsfile PR #38565 and build nodes now hooked up, I think we can do this. Breaking out the unit test into a separate step in PR #39638 is an example of setting up a parallel step.

I've analyzed the current things we run in a PR check to see what else we can do:

The PR checks run in parallel:

• janky ~115 minutes
  

  moby/hack/ci/janky

  Lines 5 to 14 in 4fb5e9e

  	hack/validate/default
  	hack/test/unit
  	hack/make.sh \
  	binary-daemon \
  	dynbinary \
  	test-docker-py \
  	test-integration-flaky \
  	test-integration \
  	cross

• experimental ~95 minutes
  

  moby/hack/ci/experimental

  Lines 7 to 9 in 4fb5e9e

  	hack/make.sh \
  	binary-daemon \
  	test-integration

• powerpc ~120 minutes
  

  moby/hack/ci/powerpc

  Lines 5 to 6 in 4fb5e9e

  	hack/test/unit
  	hack/make.sh dynbinary test-integration

• z ~115 minutes
  

  moby/hack/ci/z

  Lines 5 to 6 in 4fb5e9e

  	hack/test/unit
  	hack/make.sh dynbinary test-integration

• windowsRS1 ~75 minutes
  

  moby/hack/ci/windows.ps1

  Lines 810 to 816 in 4fb5e9e

  	$Duration= $(Measure-Command { & docker run `
  	--rm `
  	-e c=$c `
  	--workdir "c`:\go\src\github.com\docker\docker\integration-cli" `
  	-v "$env:TEMP\binary`:c:\target" `
  	docker `
  	"`$env`:PATH`='c`:\target;'+`$env:PATH`; `$env:DOCKER_HOST`='tcp`://'+(ipconfig | select -last 1).Substring(39)+'`:2357'; c:\target\runIntegrationCLI.ps1" | Out-Host } )

• windowsRS5-process ~90 minutes
  

  moby/hack/ci/windows.ps1

  Lines 810 to 816 in 4fb5e9e

  	$Duration= $(Measure-Command { & docker run `
  	--rm `
  	-e c=$c `
  	--workdir "c`:\go\src\github.com\docker\docker\integration-cli" `
  	-v "$env:TEMP\binary`:c:\target" `
  	docker `
  	"`$env`:PATH`='c`:\target;'+`$env:PATH`; `$env:DOCKER_HOST`='tcp`://'+(ipconfig | select -last 1).Substring(39)+'`:2357'; c:\target\runIntegrationCLI.ps1" | Out-Host } )

The individual parts of the PR check have timings of approx:

• hack/validate/default ~1 minute
• hack/test/unit ~7 minutes
• hack/make.sh binary-daemon ~1 minute
• hack/make.sh dynbinary ~1 minute
• hack/make.sh test-docker-py ~7 minute
• hack/make.sh test-integration-flaky ~1 minute
• hack/make.sh test-integration ~95 minutes
  • integration ~10 minutes
  • integration-cli ~85 minutes
• hack/make.sh cross ~2 minutes

I think the big one to parallelize into its own step would be the legacy integration-cli tests that take ~85 minutes. If we can also break that down further by test suite, we could then get to sub-1-hr tests.