Skip to content

Add ability to generate builds.json in parallel #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
LaurentGoderre wants to merge 1 commit into from
Closed

Add ability to generate builds.json in parallel #4

LaurentGoderre wants to merge 1 commit into from

Conversation

@LaurentGoderre
Copy link
Member

@LaurentGoderre LaurentGoderre commented Nov 15, 2023

No description provided.

@tianon
Copy link
Member

tianon commented Nov 15, 2023

While this will go faster, I don't think the end result will be correct, especially in the case of dependencies between images (because the order is actually important/relevant to the way the entries are processed and relate to each other).

@LaurentGoderre
Copy link
Member Author

LaurentGoderre commented Nov 15, 2023

@tianon I can probably modify it to preserve the order then

@tianon
Copy link
Member

tianon commented Nov 15, 2023

See

meta-scripts/builds.sh

Lines 59 to 61 in 21fdbb2

# for each sourceId, try to calculate a buildId, then do a registry lookup to get an imageId
# then, anything that has a calculated buildId but *not* an imageId is something that needs a build
# (and each buildId needs to include the imageIds of all the parent images -- without those, the buildId is invalid / impossible to calculate, which forces us to build everything in order)
- what I mean is that the calculation itself depends on the order and on calculation of builds vs deps in the same process.

(So we can't split it unless the split is dep aware, which means that in pure shell, we'll be limited on how much we can actually split this as our list grows.)

@LaurentGoderre
Copy link
Member Author

LaurentGoderre commented Nov 15, 2023

@tianon this iteration of the script generates the same output when I tested it but there might be scenario where it would be different?

@whalelines
Copy link

whalelines commented Nov 16, 2023

@tianon , can you please provide more explanation for the order dependency you are describing? It is not clear to me how within a single run of the script the various image dependencies interact.

@yosifkit
Copy link
Member

yosifkit commented Nov 16, 2023 •
edited
Loading

Currently none of the subset images have a dependency to other subset images outside their repo (i.e., the docker:dind image only depends on another docker image, like :cli), so this parallelization will currently work. The problem arises when there are cross repo image dependencies. Those lookups are put in sourceArchResolved. Right now it's fine because they are all registry lookups via image name, but later they would be a different reference directly to oisupport/staging-* instead (in case the parent image hadn't migrated to library/).

@tianon
Copy link
Member

tianon commented Nov 16, 2023

Exactly that. Here's a minimal example from my personal images with an inter-repo dependency -- the thing to note (the conclusion of this test) is that when builds.sh is run on tianon/mojo:latest alone, it returns nothing because it doesn't know that tianon/perl:latest (the parent image) does have a build ID because they have to be run together for it to cache/know that information, so it's not as simple as just running more things in parallel (or I likely would've done so originally):

$ jq 'with_entries(select(.value | (.arches[].parents | has("tianon/perl:latest")) or (.allTags | index("tianon/perl:latest"))))' .../tianon-sources.json | tee both.json
{
  "15141c3ed840530aecc507b6c2ad82963814433ded2a7ec6dc3ed16a292ae109": {
    "sourceId": "15141c3ed840530aecc507b6c2ad82963814433ded2a7ec6dc3ed16a292ae109",
    "reproducibleGitChecksum": "227a460a276760fcd668cb69ec0b736ba2bf05c0a12a9472084bb40b1b314f05",
    "allTags": [
      "tianon/perl:latest"
    ],
    "entry": {
      "GitRepo": "https://github.com/tianon/dockerfiles.git",
      "GitFetch": "refs/heads/master",
      "GitCommit": "a0516ebb34296e9cffa91e94bf5d32104c7d54c2",
      "Directory": "perl",
      "File": "Dockerfile",
      "Builder": "",
      "SOURCE_DATE_EPOCH": 1625693927
    },
    "arches": {
      "amd64": {
        "tags": [
          "tianon/perl:latest"
        ],
        "archTags": [],
        "froms": [
          "perl:5.32"
        ],
        "platformString": "linux/amd64",
        "platform": {
          "architecture": "amd64",
          "os": "linux"
        },
        "parents": {
          "perl:5.32": {
            "sourceId": null,
            "pin": null
          }
        }
      }
    }
  },
  "236d56cd470c3d456845c86b060ba92ee02003ed8256c833507c00bee6fc8a7e": {
    "sourceId": "236d56cd470c3d456845c86b060ba92ee02003ed8256c833507c00bee6fc8a7e",
    "reproducibleGitChecksum": "729bd3f0f189bca23c83a057ee61a4f19f8d7d444968d9aea998c069cb6ca7e7",
    "allTags": [
      "tianon/mojo:latest"
    ],
    "entry": {
      "GitRepo": "https://github.com/tianon/dockerfiles.git",
      "GitFetch": "refs/heads/master",
      "GitCommit": "1461dbdd7bda6d51c0219e8a7cf7eae03669de93",
      "Directory": "mojo",
      "File": "Dockerfile",
      "Builder": "",
      "SOURCE_DATE_EPOCH": 1675269649
    },
    "arches": {
      "amd64": {
        "tags": [
          "tianon/mojo:latest"
        ],
        "archTags": [],
        "froms": [
          "tianon/perl:latest"
        ],
        "platformString": "linux/amd64",
        "platform": {
          "architecture": "amd64",
          "os": "linux"
        },
        "parents": {
          "tianon/perl:latest": {
            "sourceId": "15141c3ed840530aecc507b6c2ad82963814433ded2a7ec6dc3ed16a292ae109",
            "pin": null
          }
        }
      }
    }
  }
}
$ jq 'to_entries | [ nth(0) ] | from_entries' both.json > perl.json
$ jq 'to_entries | [ nth(1) ] | from_entries' both.json > mojo.json
$ export BASHBREW_STAGING_TEMPLATE='tianon/zz-staging:ARCH-BUILD'
$ diff -u <(./builds.sh both.json) <({ ./builds.sh perl.json && ./builds.sh mojo.json; } | jq -s 'add')
15141c3ed840530aecc507b6c2ad82963814433ded2a7ec6dc3ed16a292ae109 (tianon/perl:latest):
 -> amd64: 15141c3ed840530aecc507b6c2ad82963814433ded2a7ec6dc3ed16a292ae109 (tianon/perl:latest):
 -> amd64: 8577d4d62fee648220c7ae5e375c400805990f1f24b6467feabaf0b6cea8dc37
8577d4d62fee648220c7ae5e375c400805990f1f24b6467feabaf0b6cea8dc37
236d56cd470c3d456845c86b060ba92ee02003ed8256c833507c00bee6fc8a7e (tianon/mojo:latest):
 -> amd64: da5a5cda7aee32c4a3bb1fac6942565d6a08bd372173f126eee1e5a37d411925
236d56cd470c3d456845c86b060ba92ee02003ed8256c833507c00bee6fc8a7e (tianon/mojo:latest):
 -> amd64: not yet!
--- /dev/fd/63	2023-11-16 14:06:50.114951005 -0800
+++ /dev/fd/62	2023-11-16 14:06:50.114951005 -0800
@@ -85,84 +85,5 @@
         }
       }
     }
-  },
-  "da5a5cda7aee32c4a3bb1fac6942565d6a08bd372173f126eee1e5a37d411925": {
-    "buildId": "da5a5cda7aee32c4a3bb1fac6942565d6a08bd372173f126eee1e5a37d411925",
-    "build": {
-      "img": "tianon/zz-staging:amd64-da5a5cda7aee32c4a3bb1fac6942565d6a08bd372173f126eee1e5a37d411925",
-      "resolved": {
-        "manifest": {
-          "ref": "tianon/zz-staging:amd64-da5a5cda7aee32c4a3bb1fac6942565d6a08bd372173f126eee1e5a37d411925@sha256:feb5c0eb2e7b2a5f124a384c80ba1374c4ec6e85a07ee687912eeedda0737d6f",
-          "desc": {
-            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
-            "digest": "sha256:feb5c0eb2e7b2a5f124a384c80ba1374c4ec6e85a07ee687912eeedda0737d6f",
-            "size": 2421,
-            "platform": {
-              "architecture": "amd64",
-              "os": "linux"
-            }
-          }
-        }
-      },
-      "sourceId": "236d56cd470c3d456845c86b060ba92ee02003ed8256c833507c00bee6fc8a7e",
-      "arch": "amd64",
-      "parents": {
-        "tianon/perl:latest": "sha256:c6eb152171d370f45894fcc9b51f954782eb4acaa119fd1fe0be2878f40c48d8"
-      },
-      "resolvedParents": {
-        "tianon/perl:latest": {
-          "manifest": {
-            "ref": "tianon/zz-staging:amd64-8577d4d62fee648220c7ae5e375c400805990f1f24b6467feabaf0b6cea8dc37@sha256:c6eb152171d370f45894fcc9b51f954782eb4acaa119fd1fe0be2878f40c48d8",
-            "desc": {
-              "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
-              "digest": "sha256:c6eb152171d370f45894fcc9b51f954782eb4acaa119fd1fe0be2878f40c48d8",
-              "size": 1585,
-              "platform": {
-                "architecture": "amd64",
-                "os": "linux"
-              }
-            }
-          }
-        }
-      }
-    },
-    "source": {
-      "sourceId": "236d56cd470c3d456845c86b060ba92ee02003ed8256c833507c00bee6fc8a7e",
-      "reproducibleGitChecksum": "729bd3f0f189bca23c83a057ee61a4f19f8d7d444968d9aea998c069cb6ca7e7",
-      "allTags": [
-        "tianon/mojo:latest"
-      ],
-      "entry": {
-        "GitRepo": "https://github.com/tianon/dockerfiles.git",
-        "GitFetch": "refs/heads/master",
-        "GitCommit": "1461dbdd7bda6d51c0219e8a7cf7eae03669de93",
-        "Directory": "mojo",
-        "File": "Dockerfile",
-        "Builder": "",
-        "SOURCE_DATE_EPOCH": 1675269649
-      },
-      "arches": {
-        "amd64": {
-          "tags": [
-            "tianon/mojo:latest"
-          ],
-          "archTags": [],
-          "froms": [
-            "tianon/perl:latest"
-          ],
-          "platformString": "linux/amd64",
-          "platform": {
-            "architecture": "amd64",
-            "os": "linux"
-          },
-          "parents": {
-            "tianon/perl:latest": {
-              "sourceId": "15141c3ed840530aecc507b6c2ad82963814433ded2a7ec6dc3ed16a292ae109",
-              "pin": null
-            }
-          }
-        }
-      }
-    }
   }
 }

@tianon tianon mentioned this pull request Dec 8, 2023
Merged
@tianon tianon deleted the parallel-builds branch December 15, 2023 00:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@LaurentGoderre @tianon @whalelines @yosifkit