.circleci: Improve docker image build workflow by seemethere · Pull Request #37976 · pytorch/pytorch

seemethere · 2020-05-06T23:11:08Z

.circleci: Improve docker image build workflow

Improves the docker image build workflow from many steps to basically
transparent from a user's perspective.

To update docker images now all one has to do is edit the
.circleci/docker folder and it will update automatically and also
dynamically add the tags to the list of tags to keep from the garbage
collector.

Adding a new image will currently stay the same but we can explore doing
that dynamically as well.

How the build workflow works:

Docker tags are determined by the hash defined from git for the
.circleci/docker sub-directory (extracted using git rev-parse)
Images are only built if the computed hash is not found in ecr and
the hash is different than the previously computed hash. The
previously computed hash is found using the same process as before
but subbing out HEAD for the merge base between HEAD and the base
git revision
That tag is then passed through the jobs using a shared workspace
which is added to downstream jobs using the circleci ${BASH_ENV}

How the new garbage collection works:

Tags to keep are generated by stepping through all of the commits in
in the .circleci/docker subdirectory

Signed-off-by: Eli Uriegas eliuriegas@fb.com

dr-ci · 2020-05-06T23:16:57Z

💊 CI failures summary and remediations

As of commit cf20d93 (more details on the Dr. CI page):

💚 💚 Looks good so far! There are no failures yet. 💚 💚

1 failure not recognized by patterns:

Job	Step	Action
^{docker-pytorch-linux-xenial-py3-clang5-android-ndk-r19c}	^{build_docker_image_pytorch-linux-xenial-py3-clang5-android-ndk-r19c}	🔁 rerun

This comment was automatically generated by Dr. CI (expand for details).

Follow this link to opt-out of these comments for your Pull Requests.

Please report bugs/suggestions on the GitHub issue tracker.

See how this bot performed.

This comment has been revised 71 times.

Improves the docker image build workflow from many steps to basically transparent from a user's perspective. To update docker images now all one has to do is edit the .circleci/docker folder and it will update automatically and also dynamically add the tags to the list of tags to keep from the garbage collector. Adding a new image will currently stay the same but we can explore doing that dynamically as well. How the build workflow works: - Docker tags are determined by the hash defined from git for the .circleci/docker sub-directory (extracted using git rev-parse) - Images are only built if the computed hash is not found in ecr and the hash is different than the previously computed hash. The previously computed hash is found using the same process as before but subbing out HEAD for the merge base between HEAD and the base git revision - That tag is then passed through the jobs using a shared workspace which is added to downstream jobs using the circleci ${BASH_ENV} How the new garbage collection works: - Tags to keep are generated by stepping through all of the commits in in the .circleci/docker subdirectory Signed-off-by: Eli Uriegas <eliuriegas@fb.com>

facebook-github-bot

@seemethere is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2020-05-12T00:19:08Z

@seemethere merged this pull request in def9f15.

ezyang · 2020-05-12T01:48:46Z

+              GENERATED_IMAGE_TAG=$(\
+                git log --oneline --pretty='%H' .circleci/docker \
+                  | xargs -I '{}' git rev-parse '{}:.circleci/docker' \
+                  | paste -sd "," -)


It'd probably be good to have a little comment explaining what this is doing

malfet · 2020-05-12T01:55:17Z

This PR was reverted, as it caused some regressions on master.

Summary: Previous attempts to get this right: * #38335 * #38279 * #37976 This tag kept getting deleted before the docker image ci workflow could be merged causing it to have upstream breakages. It'd be best to make sure the garbage collector just doesnt garbage collect it. This is a pre-step to merge #38484 Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: #38483 Differential Revision: D21577359 Pulled By: seemethere fbshipit-source-id: c4e0709bd8fff8f24a988b60eaa9f8c01576ef2f

Summary: closes #37855 Relies on #38483 Previous attempts to get this right: * #38335 * #38279 * #37976 This reverts commit 8063960. Improves the docker image build workflow from many steps to basically transparent from a user's perspective. To update docker images now all one has to do is edit the .circleci/docker folder and it will update automatically and also dynamically add the tags to the list of tags to keep from the garbage collector. Adding a new image will currently stay the same but we can explore doing that dynamically as well. How the build workflow works: - Docker tags are determined by the hash defined from git for the .circleci/docker sub-directory (extracted using git rev-parse) - Images are only built if the computed hash is not found in ecr and the hash is different than the previously computed hash. The previously computed hash is found using the same process as before but subbing out HEAD for the merge base between HEAD and the base git revision - That tag is then passed through the jobs using a shared workspace which is added to downstream jobs using the circleci ${BASH_ENV} How the new garbage collection works: - Tags to keep are generated by stepping through all of the commits in in the .circleci/docker subdirectory Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: #38484 Differential Revision: D21585458 Pulled By: seemethere fbshipit-source-id: 37792a1e0f5e5531438c4ae61507639c133aa76d

Summary: closes pytorch#37855 ## .circleci: Improve docker image build workflow Improves the docker image build workflow from many steps to basically transparent from a user's perspective. To update docker images now all one has to do is edit the .circleci/docker folder and it will update automatically and also dynamically add the tags to the list of tags to keep from the garbage collector. Adding a new image will currently stay the same but we can explore doing that dynamically as well. ### How the build workflow works: - Docker tags are determined by the hash defined from git for the .circleci/docker sub-directory (extracted using git rev-parse) - Images are only built if the computed hash is not found in ecr and the hash is different than the previously computed hash. The previously computed hash is found using the same process as before but subbing out HEAD for the merge base between HEAD and the base git revision - That tag is then passed through the jobs using a shared workspace which is added to downstream jobs using the circleci ${BASH_ENV} ### How the new garbage collection works: - Tags to keep are generated by stepping through all of the commits in in the .circleci/docker subdirectory Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: pytorch#37976 Differential Revision: D21511048 Pulled By: seemethere fbshipit-source-id: e4b153a6078e3875f6cfa03a903b2e951d803cce

Summary: Previous attempts to get this right: * pytorch#38335 * pytorch#38279 * pytorch#37976 This tag kept getting deleted before the docker image ci workflow could be merged causing it to have upstream breakages. It'd be best to make sure the garbage collector just doesnt garbage collect it. This is a pre-step to merge pytorch#38484 Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: pytorch#38483 Differential Revision: D21577359 Pulled By: seemethere fbshipit-source-id: c4e0709bd8fff8f24a988b60eaa9f8c01576ef2f

Summary: closes pytorch#37855 Relies on pytorch#38483 Previous attempts to get this right: * pytorch#38335 * pytorch#38279 * pytorch#37976 This reverts commit 46bc26c. Improves the docker image build workflow from many steps to basically transparent from a user's perspective. To update docker images now all one has to do is edit the .circleci/docker folder and it will update automatically and also dynamically add the tags to the list of tags to keep from the garbage collector. Adding a new image will currently stay the same but we can explore doing that dynamically as well. How the build workflow works: - Docker tags are determined by the hash defined from git for the .circleci/docker sub-directory (extracted using git rev-parse) - Images are only built if the computed hash is not found in ecr and the hash is different than the previously computed hash. The previously computed hash is found using the same process as before but subbing out HEAD for the merge base between HEAD and the base git revision - That tag is then passed through the jobs using a shared workspace which is added to downstream jobs using the circleci ${BASH_ENV} How the new garbage collection works: - Tags to keep are generated by stepping through all of the commits in in the .circleci/docker subdirectory Signed-off-by: Eli Uriegas <eliuriegas@fb.com> Pull Request resolved: pytorch#38484 Differential Revision: D21585458 Pulled By: seemethere fbshipit-source-id: 37792a1e0f5e5531438c4ae61507639c133aa76d

seemethere force-pushed the rework_docker_ci_build_flow branch 2 times, most recently from f81725f to 99b8528 Compare May 6, 2020 23:16

seemethere force-pushed the rework_docker_ci_build_flow branch 15 times, most recently from 771125d to 5e3e689 Compare May 7, 2020 20:55

seemethere commented May 7, 2020

View reviewed changes

Comment thread .circleci/scripts/calculate_directory_hash.sh Outdated

seemethere force-pushed the rework_docker_ci_build_flow branch 5 times, most recently from 6251269 to f42022b Compare May 8, 2020 18:19

seemethere force-pushed the rework_docker_ci_build_flow branch from f42022b to cf20d93 Compare May 8, 2020 21:27

seemethere changed the title ~~wip, rework docker ci build flow~~ .circleci: Improve docker image build workflow May 8, 2020

seemethere requested review from ezyang, malfet and suo May 8, 2020 21:29

seemethere marked this pull request as ready for review May 8, 2020 21:30

seemethere requested a review from kostmo May 10, 2020 07:15

malfet approved these changes May 11, 2020

View reviewed changes

facebook-github-bot reviewed May 11, 2020

View reviewed changes

facebook-github-bot closed this in def9f15 May 11, 2020

seemethere deleted the rework_docker_ci_build_flow branch May 11, 2020 23:04

facebook-github-bot added the merged label May 12, 2020

ezyang reviewed May 12, 2020

View reviewed changes

malfet mentioned this pull request May 12, 2020

[RELAND] .circleci: Improve docker image build workflow #38279

Closed

This was referenced May 14, 2020

.cirlceci: Keep tags that look like a sha1 #38483

Closed

[RELAND] .circleci: Improve docker image build workflow #38484

Closed

This was referenced May 18, 2020

[RELAND] .circleci: Improve docker image build workflow (electric boogaloo) #38532

Closed

.circleci: Improve docker image build workflow (electric boogaloo) #38686

Closed

mruberry added the Merged label Oct 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

.circleci: Improve docker image build workflow#37976

.circleci: Improve docker image build workflow#37976
seemethere wants to merge 1 commit intopytorch:masterfrom
seemethere:rework_docker_ci_build_flow

seemethere commented May 6, 2020 •

edited

Loading

Uh oh!

dr-ci Bot commented May 6, 2020 •

edited

Loading

Uh oh!

Uh oh!

facebook-github-bot left a comment

Uh oh!

facebook-github-bot commented May 12, 2020

Uh oh!

ezyang May 12, 2020

Uh oh!

malfet commented May 12, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

seemethere commented May 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!