[Build] Split tasks by arch for increased parallelism by Ikuni17 · Pull Request #223471 · elastic/kibana

Ikuni17 · 2025-06-11T21:56:11Z

Summary

Closes elastic/kibana-operations#110

This is a continuation of #217929. Previously a Task created two artifacts of a specific flavor, one for x64 and one for ARM64. These were still running serially, even though the Task itself was parallelized. This PR splits each architecture into its own Task so we can increase parallelism. The result is a 14% further decrease in time taken for the build step; 53 min down to 46 min. Between the two PRs, the artifact build time is reduced 57%!

Testing

https://buildkite.com/elastic/kibana-artifacts-snapshot/builds/6555

Next steps

Moving the ARM artifact builds to an ARM native machine would likely see the next biggest reduction because they take 2-3 times longer than their x86 counterparts. Though this would likely be a decent amount of work.
Some of the global tasks could be grouped into parallel tasks, but there is some dependency between those tasks and the gains would be minimal.

Ikuni17 · 2025-06-11T23:42:58Z

buildkite test this

This reverts commit 0e14df7.

This reverts commit 6b6784f.

Ikuni17 · 2025-06-13T19:56:56Z

buildkite test this

elasticmachine · 2025-06-13T20:28:02Z

Pinging @elastic/kibana-operations (Team:Operations)

Ikuni17 · 2025-06-13T20:34:28Z

src/dev/build/tasks/os_packages/docker_generator/run.ts

  for (const [, dockerTemplate] of Object.entries(dockerTemplates)) {
-    await write(resolve(dockerBuildDir, dockerTemplate.name), dockerTemplate.generator(scope));
+    let filename: string;
+    if (!dockerTemplate.name.includes('kibana.yml') && flags.architecture === 'aarch64') {


When running in parallel, an image flavor will have a race condition since both architectures are writing to the same build_docker.sh and Dockerfile. The result is two of the same architectures artifact. So now we're creating the files for each architecture.

delanni · 2025-06-24T15:25:53Z

src/dev/build/build_distributables.ts

    if (options.createDebPackage) {
      // control w/ --deb or --skip-os-packages
-      artifactTasks.push(Tasks.CreateDebPackage);
+      artifactTasks.push(Tasks.CreateDebPackageX64);


wouldn't this mean, that we're always building both platforms (i.e.: --all-platforms even without adding this flag)?

The --all-platforms flag is only related to building the archives and in some instances we're forcing it to true depending on other flags. All of the Tasks are building both platforms already, without any control via flags. Currently, one task creates two artifacts serially. So we're splitting that task up by platform to let them run in parallel.

delanni · 2025-06-24T15:28:26Z

src/dev/build/tasks/os_packages/docker_generator/templates/build_docker_sh.template.ts

    (dockerTag ? dockerTag : version) + (dockerTagQualifier ? '-' + dockerTagQualifier : '');
  const dockerTargetName = `${imageTag}${imageFlavor}:${tag}`;
  const dockerArchitecture = architecture === 'aarch64' ? 'linux/arm64' : 'linux/amd64';
+  const dockerfileName = architecture === 'aarch64' ? 'Dockerfile.arm64' : 'Dockerfile';


shouldn't these be Dockerfile.arm64 and Dockerfile.x64 for symmetry?

Ikuni17 · 2025-07-07T20:08:15Z

💚 All backports created successfully

Status	Branch	Result
✅	9.0
✅	8.19
✅	8.18
✅	8.17

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…#226889) # Backport This will backport the following commits from `main` to `9.1`: - [[Build] Split tasks by arch for increased parallelism (#223471)](#223471)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)  Co-authored-by: Brad White <Ikuni17@users.noreply.github.com>

kibanamachine · 2025-07-08T22:49:00Z