Fix copy large files#8409
Merged
Merged
Conversation
16693a6 to
3051f1f
Compare
Member
|
Hi @joschi, thanks for the PR. What I had in mind for the issue is checking if copy file is supported to do an async operation and then implement it in docker-java and use it in Testcontainers. |
This is an attempt to fix `OutOfMemoryError` (out of heap memory) when copying large files into a container via `GenericContainer#withCopyFileToContainer()`. The previous implementation relied on an in-memory array backing a `ByteArrayInputStream` which represents the tar archive sent to the Docker daemon which is limited by the available heap memory of the JVM at the moment of execution. By using the local disk (temporary directory), we can copy files into the container which are larger than the available heap memory of the JVM. The caveat being that we are now limited by the available disk space in the temporary directory of the machine running Testcontainers. Fixes testcontainers#4203
Member
|
Thanks for your contribution, @joschi ! PR has been updated to stream the files instead of use local disk. |
This was referenced Jul 4, 2024
|
Fixing #7239 is amazing. Thank you so much! |
lizhou1111
pushed a commit
to lizhou1111/testcontainers-java
that referenced
this pull request
Jul 13, 2024
Fix `OutOfMemoryError` (out of heap memory) when copying large files into a container via `GenericContainer#withCopyFileToContainer()`. Fixes testcontainers#4203 --------- Co-authored-by: Eddú Meléndez <eddu.melendez@gmail.com>
fokion
pushed a commit
to fokion/testcontainers-java
that referenced
this pull request
Jan 18, 2025
Fix `OutOfMemoryError` (out of heap memory) when copying large files into a container via `GenericContainer#withCopyFileToContainer()`. Fixes testcontainers#4203 --------- Co-authored-by: Eddú Meléndez <eddu.melendez@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is an attempt to fix
OutOfMemoryError(out of heap memory) when copying large files into a container viaGenericContainer#withCopyFileToContainer().The previous implementation relied on an in-memory array backing a
ByteArrayInputStreamwhich represents the tar archive sent to the Docker daemon which is limited by the available heap memory of the JVM at the moment of execution.By using the local disk (temporary directory), we can copy files into the container which are larger than the available heap memory of the JVM. The caveat being that we are now limited by the available disk space in the temporary directory of the machine running Testcontainers.
Maybe it could be useful to perform this conditionally depending on the size of the transferables or configurable via runtime property.
What do you think?
Fixes #4203