Skip to content

Add --checksum param to pipe, put, cp, mirror#5043

Merged
harshavardhana merged 1 commit intominio:masterfrom
klauspost:add-checksums-to-var
Sep 24, 2024
Merged

Add --checksum param to pipe, put, cp, mirror#5043
harshavardhana merged 1 commit intominio:masterfrom
klauspost:add-checksums-to-var

Conversation

@klauspost
Copy link
Contributor

@klauspost klauspost commented Sep 24, 2024 •
edited
Loading

Description

Add checksums to pipe, cp, mirror commands.

Add checksum to uploaded object. Values: MD5, CRC32, CRC32C, SHA1 or SHA256. Requires server trailing headers support like AWS, MinIO and possibly others.

--md5 is hidden, but still applied.

Uploads to buckets with object locking will still use MD5.

Bonus: mc stat shows checksum information.

How to test this PR?

Single/multipart upload (mc cp):

λ mc cp 0.txt myminio/testbucket -checksum=SHA256 -q &&mc stat myminio/testbucket/0.txt -q
`e:/gopath/src/github.com/minio/mc/0.txt` -> `myminio/testbucket/0.txt`
Total: 597.67 KiB, Transferred: 597.67 KiB, Speed: 59.64 MiB/s
Name      : 0.txt
Date      : 2024-09-24 16:30:40 CEST
Size      : 598 KiB
ETag      : 5d9003348d6762e91a212b50460b287e
VersionID : c21d90cb-d302-4707-a53d-d3c695bf8d4f
Type      : file
Checksum  : SHA256:X3pWjnxkqONadmZwlOAAF4+jRwLEZnTa/WGKc+s/M4w=
Encryption: SSE-KMS (arn:aws:kms:my-minio-key)
Metadata  :
  Content-Type: text/plain


λ mc cp ab.txt myminio/testbucket -checksum=SHA256 -q &&mc stat myminio/testbucket/ab.txt -q
`e:/gopath/src/github.com/minio/mc/ab.txt` -> `myminio/testbucket/ab.txt`
Total: 381.47 MiB, Transferred: 381.47 MiB, Speed: 239.29 MiB/s
Name      : ab.txt
Date      : 2024-09-24 16:31:10 CEST
Size      : 382 MiB
ETag      : a4831df44d6a59fff21881ccd7d5e86f-24
VersionID : d5cb7670-267a-4e9d-8a8d-a85fd60943c9
Type      : file
Checksum  : SHA256:oKTfpng60iS2rRaWKo8/Ep3wTOHmam8rQZbhzQI1tF0=-24
Encryption: SSE-KMS (arn:aws:kms:my-minio-key)
Metadata  :
  Content-Type: text/plain

(mc put)

λ mc put ab.txt myminio/testbucket -checksum=SHA256 -q &&mc stat myminio/testbucket/ab.txt -q
`e:/gopath/src/github.com/minio/mc/ab.txt` -> `myminio/testbucket/ab.txt`
Total: 381.47 MiB, Transferred: 381.47 MiB, Speed: 758.18 MiB/s
Name      : ab.txt
Date      : 2024-09-24 16:41:42 CEST
Size      : 382 MiB
ETag      : e5c129e03bc360798e6c862fc971f468-24
VersionID : 9ecadd2a-6846-4338-979a-abdbacd3fe95
Type      : file
Checksum  : SHA256:oKTfpng60iS2rRaWKo8/Ep3wTOHmam8rQZbhzQI1tF0=-24
Encryption: SSE-KMS (arn:aws:kms:my-minio-key)
Metadata  :
  Content-Type: text/plain


λ mc put 0.txt myminio/testbucket -checksum=SHA256 -q &&mc stat myminio/testbucket/0.txt -q
`e:/gopath/src/github.com/minio/mc/0.txt` -> `myminio/testbucket/0.txt`
Total: 597.67 KiB, Transferred: 597.67 KiB, Speed: 75.67 MiB/s
Name      : 0.txt
Date      : 2024-09-24 16:42:00 CEST
Size      : 598 KiB
ETag      : b99f4b15c1bd411e130e2b0c98e7df86
VersionID : c8e6b086-d794-4263-8ccd-c52e5cce1749
Type      : file
Checksum  : SHA256:X3pWjnxkqONadmZwlOAAF4+jRwLEZnTa/WGKc+s/M4w=
Encryption: SSE-KMS (arn:aws:kms:my-minio-key)
Metadata  :
  Content-Type: text/plain

(mc pipe):

λ cat 0.txt | mc pipe myminio/testbucket/0.txt -checksum=SHA256&&mc stat myminio/testbucket/0.txt -q
 552.00 KiB / ? [                                                                                                       =                               ] 2.01 MiB/s Name      : 0.txt
Date      : 2024-09-24 16:47:59 CEST
Size      : 598 KiB
ETag      : 062320249d887571850f2daea61b3e1e-1
VersionID : dfb6d042-248b-44fd-8dc7-21273c916666
Type      : file
Checksum  : SHA256:T4uVbgD8TgAJxPDPpgtktYfyVG3AkxP39l/BbWZFFHA=-1
Encryption: SSE-KMS (arn:aws:kms:my-minio-key)
Metadata  :
  Content-Type: text/plain

Types of changes

  • New feature (non-breaking change which adds functionality)

@klauspost
Add --checksum param to pipe, cp, mirror
960bf6c 
Add checksums to pipe, cp, mirror commands.

> Add checksum to uploaded object. Values: MD5, CRC32, CRC32C, SHA1 or SHA256. Requires server trailing headers support like AWS, MinIO and possibly others.

`--md5` is hidden, but still applied.

Uploads to buckets with object locking will still use MD5.

Bonus: `mc stat` shows checksum information.
@klauspost klauspost changed the title Add --checksum param to pipe, cp, mirror Add --checksum param to pipe, put, cp, mirror Sep 24, 2024
@klauspost klauspost mentioned this pull request Sep 24, 2024
Merged
8 tasks
@harshavardhana harshavardhana merged commit be87476 into minio:master Sep 24, 2024
@klauspost klauspost deleted the add-checksums-to-var branch September 24, 2024 15:52
@djwfyi djwfyi mentioned this pull request Oct 3, 2024
Closed
4 tasks
@Flowm Flowm mentioned this pull request Oct 9, 2024
Closed
klauspost added a commit to klauspost/warp that referenced this pull request Jun 3, 2025
@klauspost
Add optional checksums
7aca373 
Usage is similar to [mc](minio/mc#5043).

> To add a [checksum](https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html) to the uploaded objects, use `--checksum` parameter.
The following checksums are supported: `CRC32` (composite), `CRC32-FO` (full object), `CRC32C`, `CRC32-FO`, `CRC32C`, `SHA1`, `SHA256` and `CRC64NVME`.
Adding a checksum will always disable MD5 checksums.
@klauspost klauspost mentioned this pull request Jun 3, 2025
Merged
harshavardhana pushed a commit to minio/warp that referenced this pull request Jun 3, 2025
@klauspost
Add optional checksums (#400)
cee449a 
Usage is similar to [mc](minio/mc#5043).

> To add a [checksum](https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html) to the uploaded objects, use `--checksum` parameter.
The following checksums are supported: `CRC32` (composite), `CRC32-FO` (full object), `CRC32C`, `CRC32-FO`, `CRC32C`, `SHA1`, `SHA256` and `CRC64NVME`.
Adding a checksum will always disable MD5 checksums.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@harshavardhana harshavardhana harshavardhana approved these changes

@vadmeste vadmeste Awaiting requested review from vadmeste

@zveinn zveinn Awaiting requested review from zveinn

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@klauspost @harshavardhana