perf: eliminate slow chown operations in Docker builds (~41min → seconds)#12256
Merged
raymyers merged 1 commit intoOpenHands:mainfrom Jan 8, 2026
Merged
Conversation
Problem: Docker image builds for SWE-bench take ~41 minutes at step #17 due to recursive chown operations on /openhands/poetry and /openhands/micromamba. These directories contain millions of files (conda packages, virtualenvs). Root Cause: - micromamba create/install ran as root, creating root-owned files - Later, chown -R changed ownership to openhands user - overlay2 filesystem is slow for metadata-heavy operations Solution: - Run micromamba create/install as openhands user from the start - Files are created with correct ownership, no chown needed - Remove redundant chown -R on poetry and micromamba dirs Expected improvement: ~2500s → <10s (100x+ faster builds) Fixes #12043
Collaborator
|
@tofarr @chuckbutkus would you like to review this PR? |
raymyers
approved these changes
Jan 8, 2026
Contributor
|
Great find, @pdaxt! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Eliminates ~41-minute Docker build delay caused by recursive
chownoperations on large directories. Files are now created with correct ownership from the start.Problem
Docker image builds for SWE-bench evaluation take ~41 minutes at step #17 (
Dockerfile.j2lines 225-231) due to:These directories contain millions of inodes (conda packages, Python virtualenvs). Docker's overlay2 filesystem is notoriously slow for metadata-heavy operations like recursive ownership changes.
Observed build time:
12:27:1113:08:34Solution
Run
micromamba createandmicromamba installas theopenhandsuser instead of root. This creates all conda environment files with correct ownership from the start, eliminating the need for post-installationchown -R.Changes
Dockerfile.j2:293-297USER openhandsbefore conda operations,USER rootafterDockerfile.j2:226-231chown -Ron poetry and micromamba dirsExpected Improvement
Breaking Changes
None. The directories end up with the same ownership and permissions. Only the method of achieving that ownership changes.
Testing
Related
chownOperations During SWE-bench Evaluation #12043Note: I posted a validation comment on the issue before submitting this PR. Happy to adjust the approach based on feedback.