Skip to content

[v0.6] remotecache: only visit each item once when walking results#1598

Merged
AkihiroSuda merged 1 commit intomoby:docker-19.03from
tonistiigi:1903-picks
Jul 26, 2020
Merged

[v0.6] remotecache: only visit each item once when walking results#1598
AkihiroSuda merged 1 commit intomoby:docker-19.03from
tonistiigi:1903-picks

Conversation

@tonistiigi
Copy link
Copy Markdown
Member

#1577

Missed this in the latest batch.

Before this change, walkAllResults did not skip an item if it had already been
visited, so walking the graph of results actually followed every single possible
path in the graph instead of just visiting each item once.

On small graphs this wasn't noticeable, but on sufficiently large remote cache
imports this rapidly scaled out of control. For example, I first encountered
this when importing a max-cache from a build of a full linux rootfs from source
(which takes about 30 minutes to build w/out cache on an 18-core machine) and
after 30 minutes the cache import was still running with all CPUs pegged at 100%

To fix this, walkAllResults now keeps a map of already visited items (keyed by
their pointer) and skips visiting an item if it's already been visited, making
it usable on large remote cache imports.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
(cherry picked from commit 806fddd)
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
@AkihiroSuda AkihiroSuda merged commit a1e4f48 into moby:docker-19.03 Jul 26, 2020
thaJeztah added a commit to thaJeztah/docker that referenced this pull request Jul 27, 2020
full diff: moby/buildkit@4cb720e...a1e4f48

Brings in the cherry-picks from moby/buildkit#1596 and moby/buildkit#1598 :

- Add --force flag in git fetch command
- Fix socket handling during copy (Treat unix sockets as regular files)
- Remotecache: Only visit each item once when walking results.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
docker-jenkins pushed a commit to docker-archive/docker-ce that referenced this pull request Jul 28, 2020
full diff: moby/buildkit@4cb720e...a1e4f48

Brings in the cherry-picks from moby/buildkit#1596 and moby/buildkit#1598 :

- Add --force flag in git fetch command
- Fix socket handling during copy (Treat unix sockets as regular files)
- Remotecache: Only visit each item once when walking results.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 765245d54b0aad21795e01f504bd1078552bef4a
Component: engine
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants