azure: use PutBlob API for uploads instead of PutBlock API + PutBlock List API#5544
Conversation
MichaelEischer
left a comment
There was a problem hiding this comment.
I don't see a reason to keep any method other than auto. So let's get rid of old upload path and just use the auto variant. More variants just mean more code that can cause problems.
|
Hi @konidev20 , Thanks again for your time. This new method is working. After some analysis, please consider this: So maybe you just need to make some changes in
As you can see, values are matching the service versions table:
Also, it could be great to let the end user to choose the size (under the limits of course) with 2 options: (seems Rclone do it like that):
This will ensure that anyone can set the values considering there needs and/or limits (seems that old Service Version of AzureStorage are still running. Also limits are not the same for the Emulator: |
|
The Windows test failing in unrelated.
@R23-Git |
|
What would be the actual benefit of adding these options? Users already have the option to set the pack file size target via According to https://learn.microsoft.com/en-us/rest/api/storageservices/versioning-for-the-azure-storage-services all regions use a 2025 version by now. Keeping support for the emulator sounds reasonable though, so what about only cutting files into smaller parts that are larger than 2000MB?
I'm against adding this feature unless someone can provide a reason why it's actually necessary. |
6275024 to
eb72ca5
Compare
MichaelEischer
left a comment
There was a problem hiding this comment.
Let's keep the old blob size of 100MB for large uploads. Other than that the change looks fine.
I will need to update the docs and add a changelog. Will do that and mark this PR ready for review. Thanks for the review! |
No complaints in the past.
d29e15f to
f9ff230
Compare

What does this PR change? What problem does it solve?
The Azure Storage backend will now support uploading pack files in a single API call using the
PutBlobAPI. This will reduce the cost implications of the current approach by 50% where a combination ofPutBlockandPut Block ListAPI is used. (Azure charges per operation)Was the change previously discussed in an issue or on the forum?
Closes #5531
Checklist
[ ] I have added tests for all code changes.Existing test cases cover the changes.[ ] I have added documentation for relevant changes (in the manual).The changes are not user facing. It's an internal optimization.changelog/unreleased/that describes the changes for our users (see template).