Implement option to write to a custom buffer during S3 writes#547
Implement option to write to a custom buffer during S3 writes#547
Conversation
Currently, we buffer in-memory. This works well in most cases, but can be a problem when performing many uploads in parallel. Each upload needs to buffer at least 5MB of data (that is the minimum part size for AWS S3). When performing hundreds or thousands of uploads, memory can become a bottleneck. This PR relieves that bottleneck by buffering to temporary files instead. Each part gets saved to a separate temporary file and discarded after upload. This reduces memory usage at the cost of increased IO with the local disk.
piskvorky
left a comment
There was a problem hiding this comment.
What's the motivation? Who needs thousands of parallel uploads and why?
Seems like a strange obfuscation so -1 on including, unless clearly motivated.
|
|
||
| FILE | ||
| /home/misha/git/smart_open/smart_open/__init__.py | ||
| /Users/misha/git/smart_open/smart_open/__init__.py |
There was a problem hiding this comment.
osx migration, again :)
There was a problem hiding this comment.
Migration between home and office :)
re-use the same buffer in the same writer instance
I'm partitioning a large (100GB, hundreds of millions of lines) dataset in preparation for some data processing. I need each partition to be relatively small, so I'm using a large number of partitions (hundreds/thousands). If I have all the partitions open for writing simultaneously, the partitioning is very simple:
I updated the implementation. It's not really an obfuscation anymore - the custom buffer is now an optional parameter. If people don't pass it in, the implementation behaves identically to the current version in PyPI. |
Currently, we buffer in-memory. This works well in most cases, but can
be a problem when performing many uploads in parallel. Each upload
needs to buffer at least 5MB of data (that is the minimum part size for
AWS S3). When performing hundreds or thousands of uploads, memory can
become a bottleneck.
This PR relieves that bottleneck by buffering to temporary files
instead. Each part gets saved to a separate temporary file and discarded
after upload. This reduces memory usage at the cost of increased IO with
the local disk.