Download archived items as single WACZ file when possible via new download endpoint query parameter #2850
Conversation
For crawls and all-crawls endpoints, this commit adds an optional preferSingleWACZ query param which will download an archived item as a single WACZ file when possible instead of repackaging single WACZs into multiWACZs.
Download Files -> Download All update help text to indicate all files downloaded as a single WACZ
ikreymer
left a comment
There was a problem hiding this comment.
Looks good! Made some tweaks to the button and label, eg. 'Download All' instead of 'Download Files' and changed the label to 'Download all as a single WACZ' so that it covers all cases.
|
This works well, but wonder if it would be worth it to just implement this as a redirect to the S3 endpoint instead? |
Since redirect would have same API, can revisit it in a future optimization, and add this for now in interest of solving the issue. |
Fixes #2648
Replaces #2805
This PR introduces a
preferSingleWACZquery parameter to the/all-crawls/<crawl_id>/downloadand/crawls/<crawl_id>/downloadendpoints. When set to true, these endpoints will only create multi-WACZs when a crawl has more than one WACZ file, and otherwise will stream the original crawl WACZ.This flag is not enabled by default to prevent introducing breaking changes to the API, but the frontend is updated to use it in all places where it seemed appropriate.
A new backend test is also added to account for the change.
Comments and suggestions on other ways to implement this behavior are very welcome!