Skip to content

Avoid buffering into memory during book downloads #429

@imnotjames

Description

@imnotjames

What would you like to improve?

The BookDownloadService operates in such a way that for zip & kobo book downloads we will buffer the files into memory and hold them together until fully written to to the output stream and only then flushed.

What's the current friction?

If the zip is large enough, this leads to memory pressure issues and takes up memory in the heap which we don't need to be taking up. I tested this with an audiobook with multiple flac files.

In one case we write the ZIP to a ByteArrayOutputSource

https://github.com/grimmory-tools/grimmory/blob/develop/booklore-api/src/main/java/org/booklore/service/book/BookDownloadService.java#L326-L328

In another we write the ZIP to the HttpServletResponse's OutputStream which buffers to memory until flushed.

https://github.com/grimmory-tools/grimmory/blob/develop/booklore-api/src/main/java/org/booklore/service/book/BookDownloadService.java#L187-L189

In yet another we copy a local filesystem file to the HttpServletResponse OutputStream

https://github.com/grimmory-tools/grimmory/blob/develop/booklore-api/src/main/java/org/booklore/service/book/BookDownloadService.java#L282-L283

What would make this better?

Either:

  • Find a way to use FileSystemResource by writing the converted kobo fliles / ZIP to a temp file (known issue: FileSystemResource is async so we would need to retain the zip file until the end of the request.)
  • Use StreamingBodyResponse to asynchronously write to the output buffer via chunked responses (may not be acceptable for Kobo reader, this will need testing.)

Anything else? (Optional)

No response

Before Submitting

  • I've searched existing issues and confirmed this enhancement hasn't been requested yet

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels
    No fields configured for Enhancement.

    Projects

    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions