Skip to content

rgw: implement S3 additional checksum support#55076

Merged
mattbenjamin merged 13 commits intoceph:mainfrom
linuxbox2:wip-rgw-cksum
Jul 3, 2024
Merged

rgw: implement S3 additional checksum support#55076
mattbenjamin merged 13 commits intoceph:mainfrom
linuxbox2:wip-rgw-cksum

Conversation

@mattbenjamin
Copy link
Copy Markdown
Contributor

@mattbenjamin mattbenjamin commented Jan 7, 2024

Adds new Blake3 digest format (native), a concrete type to
represent digests, and static_visitor machinery to unify varying
checksum computations.

This framework, together with new trailing checksum header support,
is used to implement S3 additional checksum verification. Parts of
the AWS content checksum API work build on a prior contribution from
imtzw tongzhiwei_yewu@cmss.chinamobile.com.
Thank you!

Fixes: https://tracker.ceph.com/issues/42080
Fixes: https://tracker.ceph.com/issues/63951

squashed commits:

  • rgw_cksum: add trival test vectors for sha-format digests
    Computed digests match those produced by sha1sum, sha256sum,
    and sha512sum utilities.
  • rgw_cksum: add test vectors for blake3
    Tests the same input strings with digests validated by
    b3sum (https://crates.io/crates/b3sum).
  • rgw_ckum: switch to accel crc32c
    The internal Ceph convention appears to be to omit a final
    xor where ceph_crc32c is used, but it's required for compatibility
    with AWS implementations.
  • rgw_cksum: add XXH3 digest
  • rgw_cksum: write class encoder for rgw::digest::Cksum
  • rgw_cksum: also reverse crc32c (REBASEME)
    Mark noticed that the crc32c output was being tested against a
    byteswapped value (crc32c also needs byteswap on LE).
  • rgw_cksum: add digest::Cksum serde tests
  • rgw_cksum: fix main(...) linkage
    (so we run our main unit and not the one in gmock
  • rgw_cksum: convenience extensions for integration with RGW/S3
  • introduce rgw_cksum unique_ptr factory
  • rgw_cksum: mark string transform accessors const
  • rgw_cksum: fixup unittest_rgw_chksum compilation--all existing tests pass
  • rgw_cksum: hook up put-object checksum workflow
  • tweaks to report on content checksum mismatch
  • rgw_cksum: match SDK as well as general checksum header
  • make it more efficient
  • initialize RGWPutObj_Cksum::digest
  • rgw_cksum: write parse_cksum_type w/const char* arg
  • initialize _type correctly; doing armored wrong
  • fix expected checksum header name, clean up verify
  • fix output on checksum verify fail, cleanup
  • introduce Cksum::to_armor(); all AWS cases pass
  • oops, extra 0-byte at end of to_armor() result

Signed-off-by: Matt Benjamin mbenjamin@redhat.com

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard cephadm
  • jenkins test api
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox
  • jenkins test windows
  • jenkins test rook e2e

@mattbenjamin mattbenjamin requested a review from cbodley January 8, 2024 23:51
@cbodley cbodley added this to the squid milestone Feb 7, 2024
@github-actions
Copy link
Copy Markdown

github-actions bot commented Feb 8, 2024

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

@cbodley
Copy link
Copy Markdown
Contributor

cbodley commented Feb 13, 2024

i revisited the discussion on @imtzw's original pr and was reminded of this one: #49986 (comment)

from https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#large-object-checksums

If you've enabled additional checksum values for your multipart object, Amazon S3 calculates the checksum for each individual part by using the specified checksum algorithm. The checksum for the completed object is calculated in the same way that Amazon S3 calculates the MD5 digest for the multipart upload. You can use this checksum to verify the integrity of the object.

the existing logic for ETag/MD5 is here: https://github.com/ceph/ceph/blob/d8dcffa/src/rgw/driver/rados/rgw_sal_rados.cc#L2700-L2707

it doesn't look like this multipart behavior was implemented here either?

@cbodley
Copy link
Copy Markdown
Contributor

cbodley commented Feb 14, 2024

PutObject is missing response headers x-amz-checksum-crc32 etc

CreateMultipartUpload missing x-amz-checksum-algorithm as request header and response header

CompleteMultipartUpload is missing:

  • read the <Part> checksums out of the <CompleteMultipartUpload> document and compare against stored part checksums
  • require checksum algorithm to match CreateMultipartUpload and all part uploads
  • calculate the composite checksum of parts
  • parse request headers x-amz-checksum-crc32 etc. and, if present, compare against the composite checksum
  • store composite checksum in an object xattr for Head/GetObject (RGW_ATTR_AMZ_CKSUM or something else? needs to include the extra parts-count string at the end)
  • write composite checksum in response header ie x-amz-checksum-crc32

@cbodley
Copy link
Copy Markdown
Contributor

cbodley commented Feb 14, 2024

i pushed https://github.com/cbodley/ceph/commits/wip-rgw-cksum, which is a rebase of this branch on top of the latest in #54856, plus 3 commits which get the non-multipart test cases in ceph/s3-tests#540 passing. i still can't get xxhash to compile so i commented it out of rgw_cksum.h, but didn't include those changes

@mattbenjamin
Copy link
Copy Markdown
Contributor Author

it's possible that the blake3 and xx3 mechanisms should be skipped for now; I really wish to have them for future--but for now, there'd be no way to access them; it does seem like we could switch our xxh64 callers to xx3, almost certainly?

@cbodley
Copy link
Copy Markdown
Contributor

cbodley commented Feb 14, 2024

it's possible that the blake3 and xx3 mechanisms should be skipped for now; I really wish to have them for future--but for now, there'd be no way to access them; it does seem like we could switch our xxh64 callers to xx3, almost certainly?

agree that we should focus on the sdk-supported algorithms for now

i do think we can use our boto extension strategy to teach it about the new checksum algorithms. that should work for the low-level operations like put_object() etc. where the caller passes in their own computed checksum. we just won't be able to get the higher-level sdk operations like upload_file() to compute those checksums for us

@yuriw
Copy link
Copy Markdown
Contributor

yuriw commented Feb 15, 2024

@mattbenjamin pls rebase

@mattbenjamin
Copy link
Copy Markdown
Contributor Author

@mattbenjamin
Copy link
Copy Markdown
Contributor Author

@cbodley
Copy link
Copy Markdown
Contributor

cbodley commented Jul 2, 2024

rgw/dbstore:

FAILED s3tests_boto3/functional/test_s3.py::test_multipart_checksum_sha256 - ...
FAILED s3tests_boto3/functional/test_s3.py::test_multipart_checksum_3parts - ...

a lot of multipart test cases are filtered out using @pytest.mark.fails_on_dbstore, these tests probably need the same

rgw/lifecycle:

FAILED s3tests_boto3/functional/test_s3.py::test_lifecycle_expiration_size_gt

i can't tell why that one failed

@mattbenjamin
Copy link
Copy Markdown
Contributor Author

rgw/dbstore:

FAILED s3tests_boto3/functional/test_s3.py::test_multipart_checksum_sha256 - ...
FAILED s3tests_boto3/functional/test_s3.py::test_multipart_checksum_3parts - ...

a lot of multipart test cases are filtered out using @pytest.mark.fails_on_dbstore, these tests probably need the same

rgw/lifecycle:

FAILED s3tests_boto3/functional/test_s3.py::test_lifecycle_expiration_size_gt

i can't tell why that one failed

I've been seeing some unexpected lc failures--I saw this in a scenario where it looked like my test system went into screen lock; more recently, I saw consistent failures during specific times of the day (but not in this test).

@mattbenjamin
Copy link
Copy Markdown
Contributor Author

@mattbenjamin
Copy link
Copy Markdown
Contributor Author

rerun 3 (rerun 2 had wrong baseline):
https://pulpito.ceph.com/mbenjamin-2024-07-03_13:56:10-rgw-wip-rgw-cksum-distro-default-smithi/

@cbodley cbodley added the TESTED label Jul 3, 2024
mattbenjamin and others added 13 commits July 3, 2024 14:18
Among other things, provides XXH128 and XXH3.  Includes
compile fixes for gcc-13.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Fast, cryptographic hash functions suitable for block checksums.
BLAKE3 is the faster, more portable successor to Blake2(b,s),
now with 4x throughput.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Adds new Blake3 digest format (native), a concrete type to
represent digests, and static_visitor machinery to unify varying
checksum computations.

This framework, together with new trailing checksum header support,
is used to implement S3 additional checksum verification.  Parts of
the AWS content checksum API work build on a prior contribution from
imtzw <tongzhiwei_yewu@cmss.chinamobile.com>.
Thank you!

Fixes: https://tracker.ceph.com/issues/42080
Fixes: https://tracker.ceph.com/issues/63951

squashed commits:
* rgw_cksum: add trival test vectors for sha-format digests
      Computed digests match those produced by sha1sum, sha256sum,
      and sha512sum utilities.
* rgw_cksum: add test vectors for blake3
      Tests the same input strings with digests validated by
      b3sum (https://crates.io/crates/b3sum).
* rgw_ckum: switch to accel crc32c
      The internal Ceph convention appears to be to omit a final
      xor where ceph_crc32c is used, but it's required for compatibility
      with AWS implementations.
* rgw_cksum: add XXH3 digest
* rgw_cksum: write class encoder for rgw::digest::Cksum
* rgw_cksum: also reverse crc32c (REBASEME)
      Mark noticed that the crc32c output was being tested against a
      byteswapped value (crc32c also needs byteswap on LE).
* rgw_cksum: add digest::Cksum serde tests
* rgw_cksum: fix main(...) linkage
      (so we run our main unit and not the one in gmock
* rgw_cksum: convenience extensions for integration with RGW/S3
* introduce rgw_cksum unique_ptr factory
* rgw_cksum: mark string transform accessors const
* rgw_cksum: fixup unittest_rgw_chksum compilation--all existing tests pass
* rgw_cksum: hook up put-object checksum workflow
* tweaks to report on content checksum mismatch
* rgw_cksum: match SDK as well as general checksum header
* make it more efficient
* initialize RGWPutObj_Cksum::digest
* rgw_cksum:  write parse_cksum_type w/const char* arg
* initialize _type correctly; doing armored wrong
* fix expected checksum header name, clean up verify
* fix output on checksum verify fail, cleanup
* introduce Cksum::to_armor();  all AWS cases pass
* oops, extra 0-byte at end of to_armor() result
* use to_armor() with decoded checksums (i.e., for all S3 presentation)
* remove unnecessary finalize() in RGWPutObj_Cksum dtor
* RGWPutObj_Cksum::Factory fixes
* fixes test_object_checksum_sha256
* choose preferred checksum algorithm header if both are present
* log verified checksums in RGWPutObj::execute at 16
* checksum not needed in policy condition
* fix checksum trailing header format
* move Blake3 to rgw_digest_blake3.h

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
I.e., prove cksum == Cksum(cksum.to_armor().c_str())

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
includes commits:

* fixes init-multipart header return
* introduce checksum to SAL MultipartPart interface
* thread optional checksum through DataProcessor
* code complete multipart checksum verify
* fix formatter
* fix ckecksum format for multipart objects in GET/HEAD ops
* always return parts_count from ReadOp::prepare() if applicable
      This behavior is used when returning the checksum of a multipart
      upload object.
* tweak conditional multipart_parts_count
* add checksum output to ListMultipart
* fix nil-return from GetHeaderCksumResult
* re-arm truncated if re-entering list-parts
* complete-multipart w/list-parts
* validate supplied checksum in CompleteMultipart
* verify checksum type against initial checksum algorithm
* rgw_op: suppress more x-amz headers
* final fixes and cleanups
* remove unused t0

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
* fix to deal with parts_count == 1 asymmetry

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
parts_count now returns a value for parts_count whenever the
target object is a multipart upload.  this has no additional
overhead, since this can be read off the manifest

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
* properly transform pseudo headers in PostObj
* enable cksum verify in PostObj
* match checksum headers in match_policy_vars
* fixup add POST headers to environment

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
* remove rgw_cksum_pipe state enum, not needed [Casey review]
* remove a format that just took a single string substitution
  and passed it to an iostream [Casey review]
* use boost::to_upper* [Casey review]
* remove unused RGW_ATTR_CKSUM_ALGORITHM decl [Casey review]
* negate error code values in two places [Casey review]
* split cksum digests from base type decls
* resolve comment when checksum requested but not available
* remove redundant memset
* remove junk from rgw_blake3_digest.h
* s/ldpp_dout + fmt::format/ldpp_dout_fmt/g;
* fix conditional return of parts_count
      from RGWRados::Object::prepare().  A value for parts_count should
      be returned iff a *multipart* object manifest exists.
* remove /tmp output test
* finish moving ceph_crypto headers out of rgw_cksum.h
* consume the optional in multipart_parts_count
* target_attrs can be a reference (but not const)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Allow MultipartUpload::complete() to have previously called
Upload::get_info() without an additional round-trip to fetch
attributes.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Added EXCLUDE_FROM_ALL to prevent gristing files in
the subdir.

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
@mattbenjamin
Copy link
Copy Markdown
Contributor Author

jenkins retest this please

@mattbenjamin mattbenjamin merged commit 1a7902c into ceph:main Jul 3, 2024
Comment on lines +78 to +80
[submodule "src/boost_redis"]
path = src/boost_redis
url = https://github.com/boostorg/redis.git
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was added back when you resolved the merge conflict. i opened #58428 to remove it again

@cbodley
Copy link
Copy Markdown
Contributor

cbodley commented Jul 4, 2024

i'll think about how best to get our CMakeLists.txt to handle this for us in debug builds

@mattbenjamin this what i came up with:

# work around https://github.com/Cyan4973/xxHash/issues/943 for debug builds
target_compile_definitions(rgw_common PUBLIC
  $<$<CONFIG:Debug>:XXH_NO_INLINE_HINTS=1>
  $<$<CONFIG:RelWithDebInfo>:XXH_NO_INLINE_HINTS=1>)

would you like me to push a commit for that here? it's currently in ea9f13d as part of #58310, which goes on to add support for blake3-devel and xxhash-devel system packages when they exist

we missed this part, so main fails to build in debug mode. i pushed my commit to a new branch and opened #58429 to fix

@rkhudov
Copy link
Copy Markdown
Contributor

rkhudov commented Jul 9, 2024

@cbodley @mattbenjamin
I have multiple issues after merging this PR:

src/rgw/rgw_xxh_digest.h:29:5: error: 'XXH3_state_t' does not name a type; did you mean 'XXH32_state_t'?
XXH3_state_t s;
src/rgw/rgw_xxh_digest.h:35:23: error: 's' was not declared in this scope
XXH3_INITSTATE(&s);
src/rgw/rgw_xxh_digest.h:39:41: error: 's' was not declared in this scope
void Restart() { XXH3_64bits_reset(&s); }
src/rgw/rgw_xxh_digest.h:39:22: error: 'XXH3_64bits_reset' was not declared in this scope; did you mean 'XXH64_reset'?
 void Restart() { XXH3_64bits_reset(&s); }

@cbodley
Copy link
Copy Markdown
Contributor

cbodley commented Jul 9, 2024

@cbodley @mattbenjamin I have multiple issues after merging this PR:

thanks @rkhudov. we discussed this in slack, but for others seeing this: update your submodule with something like git submodule update --init --recursive

NitzanMordhai pushed a commit to NitzanMordhai/ceph that referenced this pull request Aug 1, 2024
rgw: implement S3 additional checksum support
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants