bitrot: add selftest for server startup by aead · Pull Request #11917 · minio/minio

aead · 2021-03-29T12:14:24Z

Description

This commit adds a self-test for all bitrot algorithms:

SHA-256
BLAKE2b
HighwayHash

The self-test computes an incremental checksum of pseudo-random
messages. If a bitrot algorithm implementation stops working on
some CPU architecture or with a certain Go version this self-test
will prevent the server from starting and silently corrupting data.

Motivation and Context

For additional context see: minio/highwayhash#19

How to test this PR?

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Optimization (provides speedup with no functional changes)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

Fixes a regression (If yes, please add commit-id or PR # here)
Documentation updated
Unit tests added/updated

klauspost

LGTM

klauspost · 2021-03-30T12:14:15Z

@aead Needs merge from master.

klauspost

Some print stuff.

cmd/bitrot.go

This commit adds a self-test for all bitrot algorithms: - SHA-256 - BLAKE2b - HighwayHash The self-test computes an incremental checksum of pseudo-random messages. If a bitrot algorithm implementation stops working on some CPU architecture or with a certain Go version this self-test will prevent the server from starting and silently corrupting data. For additional context see: minio/highwayhash#19

minio-trusted · 2021-03-31T22:54:55Z

Mint Automation

Test	Result
mint-large-bucket.sh	✔️
mint-fs.sh	✔️
mint-gateway-s3.sh	✔️
mint-erasure.sh	✔️
mint-dist-erasure.sh	✔️
mint-zoned.sh	✔️
mint-gateway-nas.sh	✔️
mint-compress-encrypt-dist-erasure.sh	❌ more...

11917-73be4bb/mint-compress-encrypt-dist-erasure.sh.log:

Running with
SERVER_ENDPOINT:      minio-dev7.minio.io:32600
ACCESS_KEY:           minio
SECRET_KEY:           ***REDACTED***
ENABLE_HTTPS:         0
SERVER_REGION:        us-east-1
MINT_DATA_DIR:        /mint/data
MINT_MODE:            full
ENABLE_VIRTUAL_STYLE: 0

To get logs, run 'docker cp 70d4bb39a201:/mint/log /tmp/mint-logs'

(1/15) Running aws-sdk-go tests ... done in 2 seconds
(2/15) Running aws-sdk-java tests ... done in 1 seconds
(3/15) Running aws-sdk-php tests ... done in 44 seconds
(4/15) Running aws-sdk-ruby tests ... done in 4 seconds
(5/15) Running awscli tests ... FAILED in 32 seconds
{
  "name": "awscli",
  "duration": 2516,
  "function": "aws --endpoint-url http://minio-dev7.minio.io:32600 s3api copy-object --bucket awscli-mint-test-bucket-1250 --key datafile-1-kB-copy --copy-source awscli-mint-test-bucket-1250/datafile-1-kB\n",
  "status": "FAIL",
  "error": "Hash mismatch expected 084e1383b70fb0c51acc680fef370023, got ac57de7156d7fc25ac1a65f81fa3989b"
}
(5/15) Running healthcheck tests ... done in 0 seconds
(6/15) Running mc tests ... done in 47 seconds
(7/15) Running minio-dotnet tests ... done in 37 seconds
(8/15) Running minio-go tests ... FAILED in 1 minutes and 57 seconds
{
  "args": {},
  "duration": 290,
  "error": "At least one of the pre-conditions you specified did not hold",
  "function": "CopyObjectPart(destination, source)",
  "message": "CopyObjectPart call failed",
  "name": "minio-go: testUnencryptedToSSES3CopyObjectPart",
  "status": "FAIL"
}
(8/15) Running minio-java tests ... FAILED in 1 minutes and 40 seconds
{
  "name": "minio-java",
  "function": "copyObject()",
  "args": "[match etag]",
  "duration": 390,
  "status": "FAIL",
  "error": "error occurred\nErrorResponse(code = PreconditionFailed, message = At least one of the pre-conditions you specified did not hold, bucketName = minio-java-test-325r2g7, objectName = minio-java-test-2rra85f-copy, resource = /minio-java-test-325r2g7/minio-java-test-2rra85f-copy, requestId = 16718EFBFE12E2CA, hostId = 4eeabb4e-34c9-4dfe-a3cb-41f45e21e2f7)\nrequest={method=PUT, url=http://minio-dev7.minio.io:32600/minio-java-test-325r2g7/minio-java-test-2rra85f-copy, headers=x-amz-copy-source-if-match: 71cff0a060f852067e443ad1e24ae26c-1\nx-amz-copy-source: /minio-java-test-1mmboo2/minio-java-test-2rra85f\nHost: minio-dev7.minio.io:32600\nAccept-Encoding: identity\nUser-Agent: MinIO (Linux; amd64) minio-java/8.0.3\nContent-MD5: 1B2M2Y8AsgTpgAmY7PhCfg==\nx-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\nx-amz-date: 20210331T224914Z\nAuthorization: AWS4-HMAC-SHA256 Credential=*REDACTED*/20210331/us-east-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-content-sha256;x-amz-copy-source;x-amz-copy-source-if-match;x-amz-date, Signature=*REDACTED*\n}\nresponse={code=412, headers=Accept-Ranges: bytes\nContent-Length: 418\nContent-Security-Policy: block-all-mixed-content\nContent-Type: application/xml\nETag: \"71cff0a060f852067e443ad1e24ae26c\"\nLast-Modified: Wed, 31 Mar 2021 22:49:14 GMT\nServer: MinIO\nVary: Origin\nX-Amz-Request-Id: 16718EFBFE12E2CA\nX-Xss-Protection: 1; mode=block\nDate: Wed, 31 Mar 2021 22:49:14 GMT\n}\n >>> [io.minio.MinioClient.execute(MinioClient.java:775), io.minio.MinioClient.execute(MinioClient.java:563), io.minio.MinioClient.executePut(MinioClient.java:904), io.minio.MinioClient.copyObject(MinioClient.java:1232), FunctionalTest.testCopyObjectMatchETag(FunctionalTest.java:1850), FunctionalTest.copyObject(FunctionalTest.java:2016), FunctionalTest.runObjectTests(FunctionalTest.java:3757), FunctionalTest.runTests(FunctionalTest.java:3783), FunctionalTest.main(FunctionalTest.java:3927)]"
}
(8/15) Running minio-js tests ... done in 58 seconds
(9/15) Running minio-py tests ... done in 3 minutes and 19 seconds
(10/15) Running s3cmd tests ... FAILED in 5 seconds
{
  "name": "s3cmd",
  "duration": "2838",
  "function": "test_put_object_multipart",
  "status": "FAIL",
  "error": "WARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Retrying upload of /mint/data/datafile-65-MB\nWARNING: MD5 Sums don't match!\nWARNING: Too many failures. Giving up on '/mint/data/datafile-65-MB'\nERROR: \nUpload of '/mint/data/datafile-65-MB' part 1 failed. Use\n  /usr/local/bin/s3cmd abortmp s3://s3cmd-test-bucket-31225/s3cmd-test-object-12876 cde51c67-b979-41b6-8c2e-58857c83c06a\nto abort the upload, or\n  /usr/local/bin/s3cmd --upload-id cde51c67-b979-41b6-8c2e-58857c83c06a put ...\nto continue the upload.\nERROR: Upload of '/mint/data/datafile-65-MB' failed too many times (Last reason: )"
}
(10/15) Running s3select tests ... done in 10 seconds
(11/15) Running security tests ... done in 0 seconds

Executed 11 out of 15 tests successfully.

Deleting image on docker hub
Deleting image locally

klauspost

LGTM

aead requested review from harshavardhana and klauspost March 29, 2021 12:14

aead force-pushed the self-test branch from 260099c to 7dcb6fe Compare March 29, 2021 13:43

klauspost approved these changes Mar 29, 2021

View reviewed changes

aead force-pushed the self-test branch from 7dcb6fe to 7020af2 Compare March 30, 2021 12:59

klauspost suggested changes Mar 30, 2021

View reviewed changes

cmd/bitrot.go Outdated Show resolved Hide resolved

cmd/bitrot.go Outdated Show resolved Hide resolved

aead force-pushed the self-test branch from 7020af2 to e9a9cc8 Compare March 30, 2021 19:38

harshavardhana suggested changes Mar 30, 2021

View reviewed changes

cmd/bitrot.go Outdated Show resolved Hide resolved

cmd/bitrot.go Outdated Show resolved Hide resolved

aead force-pushed the self-test branch from e9a9cc8 to 73be4bb Compare March 31, 2021 22:36

harshavardhana approved these changes Mar 31, 2021

View reviewed changes

harshavardhana requested review from klauspost and vadmeste April 2, 2021 16:18

klauspost approved these changes Apr 6, 2021

View reviewed changes

harshavardhana merged commit d5d2fc9 into minio:master Apr 6, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bitrot: add selftest for server startup#11917

bitrot: add selftest for server startup#11917
harshavardhana merged 1 commit intominio:masterfrom
aead:self-test

aead commented Mar 29, 2021

Uh oh!

klauspost left a comment

Uh oh!

klauspost commented Mar 30, 2021

Uh oh!

klauspost left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

minio-trusted commented Mar 31, 2021

Uh oh!

klauspost left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

aead commented Mar 29, 2021

Description

Motivation and Context

How to test this PR?

Types of changes

Checklist:

Uh oh!

klauspost left a comment

Choose a reason for hiding this comment

Uh oh!

klauspost commented Mar 30, 2021

Uh oh!

klauspost left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

minio-trusted commented Mar 31, 2021

Mint Automation

11917-73be4bb/mint-compress-encrypt-dist-erasure.sh.log:

Uh oh!

klauspost left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants