Skip to content

tsdb: Fixed race in chunks.Chunks method.#6515

Merged
bwplotka merged 1 commit intomasterfrom
issue-6512
Dec 24, 2019
Merged

tsdb: Fixed race in chunks.Chunks method.#6515
bwplotka merged 1 commit intomasterfrom
issue-6512

Conversation

@bwplotka
Copy link
Copy Markdown
Member

Added regression test.

Fixes #6512

Without this fix, regression test result: (not deterministic result due to concurrency):

=== RUN   TestChunkReader_ConcurrentRead
--- FAIL: TestChunkReader_ConcurrentRead (0.01s)
    db_test.go:2702: unexpected error: checksum mismatch expected:597ad276, actual:597ad276
    db_test.go:2702: unexpected error: checksum mismatch expected:dd0cdbc2, actual:dd0cdbc2
FAIL

cc @brian-brazil @roidelapluie

@roidelapluie I literally used your code with small tweaks.

Signed-off-by: Bartlomiej Plotka bwplotka@gmail.com

@bwplotka
Copy link
Copy Markdown
Member Author

bwplotka commented Dec 24, 2019

The alternative is to use sync.Pool in place of old buffer. This will pool the bytes for checksum like before and save ~20 bytes (slice header + 4 bytes for crc32) so IMO not worth it.

Copy link
Copy Markdown
Member

@roidelapluie roidelapluie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments. Thanks.

tsdb/db_test.go Outdated
@@ -2487,7 +2488,7 @@ func TestDBReadOnly_FlushWAL(t *testing.T) {

// TestChunkWriter ensures that chunk segment are cut at the set segment size and
Copy link
Copy Markdown
Member

@roidelapluie roidelapluie Dec 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// TestChunkWriter ensures that chunk segment are cut at the set segment size and
// TestChunkWriter_ReadAfterWrite ensures that chunk segment are cut at the set segment size and

@@ -570,11 +569,11 @@ func (s *Reader) Chunk(ref uint64) (chunkenc.Chunk, error) {
}

sum := sgmBytes.Range(chkEnd-crc32.Size, chkEnd)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
sum := sgmBytes.Range(chkEnd-crc32.Size, chkEnd)
sum := sgmBytes.Range(chkDataEnd, chkEnd)

@brian-brazil
Copy link
Copy Markdown
Contributor

👍

I'd focus on getting a fix in and released, we can always microoptimise later.

Copy link
Copy Markdown
Member

@roidelapluie roidelapluie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. Let's merge to master then cherry-pick to release-2.15 and release 2.15.1.

Added regression test.

Fixes #6512

Before (not deterministic result due to concurrency):
```
=== RUN   TestChunkReader_ConcurrentRead
--- FAIL: TestChunkReader_ConcurrentRead (0.01s)
    db_test.go:2702: unexpected error: checksum mismatch expected:597ad276, actual:597ad276
    db_test.go:2702: unexpected error: checksum mismatch expected:dd0cdbc2, actual:dd0cdbc2
FAIL
```

After succuess on multiple runs.


Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
@bwplotka
Copy link
Copy Markdown
Member Author

bwplotka commented Dec 24, 2019

Addressed comments. Once tests (without build - this passed previously, so it's enough) passes, let's merge.

@bwplotka bwplotka merged commit 3b8ef63 into master Dec 24, 2019
@bwplotka bwplotka deleted the issue-6512 branch December 24, 2019 21:55
bwplotka added a commit that referenced this pull request Dec 24, 2019
Added regression test.

Fixes #6512

Before (not deterministic result due to concurrency):
```
=== RUN   TestChunkReader_ConcurrentRead
--- FAIL: TestChunkReader_ConcurrentRead (0.01s)
    db_test.go:2702: unexpected error: checksum mismatch expected:597ad276, actual:597ad276
    db_test.go:2702: unexpected error: checksum mismatch expected:dd0cdbc2, actual:dd0cdbc2
FAIL
```

After succuess on multiple runs.


Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

checksum mismatch breaking Grafana queries

3 participants