br: add content length for multi part upload#19358
br: add content length for multi part upload#19358ti-chi-bot[bot] merged 7 commits intotikv:masterfrom
Conversation
Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
📝 WalkthroughWalkthroughAdds request customization to set explicit Content-Length headers for S3 multipart operations (create, upload part, complete, abort) and extends tests to assert those headers and add an abort-after-part multipart test with histogram checks. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Poem
🚥 Pre-merge checks | ✅ 6✅ Passed checks (6 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
No actionable comments were generated in the recent review. 🎉 Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
Adds explicit Content-Length headers to S3 multipart-upload related requests in TiKV’s AWS cloud storage implementation to improve compatibility with GCP’s S3-compatible service (which can return HTTP 411 when Content-Length is missing).
Changes:
- Customize
create_multipart_upload,complete_multipart_upload, andabort_multipart_uploadrequests to addContent-Length. - Update multipart upload tests to include
Content-Lengthexpectations. - Add a new test covering abort behavior on multipart upload failure.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Fix all issues with AI agents
In `@components/cloud/aws/src/s3.rs`:
- Around line 1069-1089: The mocked abort multipart upload response currently
uses a CompleteMultipartUploadResult XML which is semantically incorrect; locate
the ReplayEvent::new that builds the Request with query
x-id=AbortMultipartUpload (the URI containing
"x-id=AbortMultipartUpload&uploadId=1") and replace the response body in its
http::Response::builder() call to be an empty body (or a minimal empty XML) to
accurately represent AbortMultipartUpload semantics instead of the
CompleteMultipartUploadResult payload.
- Around line 1018-1019: The inline comment above the multipart test referencing
"split magic_contents into 3 parts, so we mock 5 requests here(1 begin + 3 part
+ 1 complete)" is a copy-paste artifact and should be corrected to reflect the
abort path exercised by the test: update the comment to mention "1 abort" (or "1
abort request") instead of "1 complete" and optionally note that the mocked
requests are 1 begin + 3 part + 1 abort; locate the comment near the
magic_contents split in the multipart upload test in s3.rs and change the
wording accordingly.
- Around line 1107-1114: The histogram assertion reads a process-global metric
CLOUD_REQUEST_HISTOGRAM_VEC via
get_metric_with_label_values(...).get_sample_count() and compares to an absolute
value, which is flaky because other tests (e.g., test_s3_storage_multi_part)
also observe "upload_part"; change the test to snapshot the prior count by
calling get_sample_count() before the operation, run the upload/abort steps,
then re-read get_sample_count() and assert the difference (new - old) equals the
expected number of observations ((magic_contents.len() / multi_part_size) as
u64); apply the same delta-assertion pattern to test_s3_storage_multi_part as
well so both tests compare deltas instead of absolute counts.
🧹 Nitpick comments (2)
components/cloud/aws/src/s3.rs (2)
947-955: Minor inconsistency: the 3rdupload_partmock is missing thecontent-lengthheader, while parts 1 and 2 include it.Parts 1 and 2 explicitly assert
header("content-length", "2"), but part 3 (also a 2-byte body"90") omits it. This doesn't break the test (the SDK sets the header via.content_length()on the builder anyway), but for consistency and completeness:Proposed fix
ReplayEvent::new( http::Request::builder() + .header("content-length", "2") .uri(Uri::from_static( "https://s3.cn-north-1.amazonaws.com.cn/mybucket/mykey?x-id=UploadPart&partNumber=3&uploadId=1" ))
1060-1066: Same inconsistency as in the success test: 3rdupload_partmock is missingcontent-lengthheader.Same issue as noted in
test_s3_storage_multi_part— part 3's mock should include.header("content-length", "2")for consistency with parts 1 and 2.
Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: v01dstar, YuJuncen The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
In response to a cherrypick label: new pull request created to branch |
|
In response to a cherrypick label: new pull request created to branch |
|
/cherry-pick release-8.5-20260215-v8.5.5 |
|
@Leavrth: new pull request created to branch DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository. |
* draft Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> * add unit tests Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> * fix typos Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> * make clippy Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> * fix unit test Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> --------- Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> Co-authored-by: Jianjun Liao <jianjun.liao@outlook.com>
close tikv#19352 add Content-Length into HTTP Request Header to make it compatible with GCP S3 compatible service. Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com>
close tikv#19352 add Content-Length into HTTP Request Header to make it compatible with GCP S3 compatible service. Signed-off-by: Jianjun Liao <jianjun.liao@outlook.com> Signed-off-by: rishabh mittal <mittalrishabh@gmail.com>
What is changed and how it works?
Issue Number: Close #19352
What's Changed:
Related changes
pingcap/docs/pingcap/docs-cn:Check List
Tests
Side effects
Release note
Summary by CodeRabbit
Bug Fixes
Tests