The moby/moby CI is far from being optimized :

• one big Dockerfile that hits really bad when something changes in it handled by multi-stage Updates the Dockerfile to use multi-stage #35100
• there is at least 5 jobs on each PR, each of them doing a lot of steps in common (some of them are not really required to be repeated), and they all took at least 1h30 to 2h.
• the integration tests are still quite flaky, and when they are, it's another 2h round of build, "just" to get green light !
• the CI definition is not on the repository, and needs to be taken care for each job.
• Only building the dev image (used for any make target) takes ~14min on the CI.

The idea of this issue is to track enhancement required for the moby ci to be better, and help us instead of scaring us each time we wanna do rebase or a small change. Here is a brain-dump list of work we should do to enable that.

• Use multi-stage build to lower cache burn when we update a dependency
• Split target to use different stages (see Dockerfile: split CI parts to separate Dockerfile / stage #36415)
• Migrate integration-cli to integration and make those tests as parallelizable as possible (see Replace integration-cli suite with API test suite #32866)
• Try publishing stage(s) build as image and use --cache-from to remove the need to build the base image for each and every CI run (on each jobs)
• Switch to Jenkinsfile to be able to make the builds evolve directly from the source code (in a secure manner)
  • Migrate current jobs into a Jenkinsfile as is
  • Define a pipeline to optimize time spent for a run
  • Split integration tests suite (integration-cli and integration) into several parallel run (probably on multiple node)
  • Refactor Docker Master jobs to have "one" build job and X targeted test jobs (for different distribution/graphdriver/setup)
  • Make sure we can restart steps independently from each other
• Remove experimental build/runner : experimental is (for quite a long time now) a runtime option (build before). We should have a experimental test suite instead of a whole build/runner dedicated for that. wip in Migrate test-integration-cli experimental macvlan test to integration #36697 Migrate test-integration-cli experimental build tests to integration #36746 Migrate test-integration-cli experimental ipvlan test to integration  #36722
• Explore having a bot/merge queue workflow.
  Once the required LGTM are in place, a bot takes over to merge in a merge queue — rebuilds jobs against master, merge it if everything is good, or labels it and comments on it if something went wrong.
  We could also have some build run only if asked from maintainers/curator, and have a lighter CI runs when PRs are in certain states (like design-review) …
  See Bottylicious - automate all the things! #35490 for that.
• Measure the CI
  • How much time each step takes (and their evolution in time)
  • What are the slowest test
  • What are the most flaky test (report each test failure to be able to track which tests need some love)
  • Create a dashboard for maintainers with all those information