Skip to content

client, roachpb: eliminate log spam when loadgens are killed#29817

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
andreimatei:rollback-spam
Sep 10, 2018
Merged

client, roachpb: eliminate log spam when loadgens are killed#29817
craig[bot] merged 1 commit intocockroachdb:masterfrom
andreimatei:rollback-spam

Conversation

@andreimatei
Copy link
Copy Markdown
Contributor

When the TPCC loadgen is CTRL-C'ed, the logs are spammed with:
I180906 22:02:41.239771 27177127 internal/client/txn.go:625 [n1] async rollback failed: TransactionStatusError: already committed (REASON_UNKNOWN): "sql txn" id=57c518aa key=/Table/61/1/1224/0 rw=true pri=0.03781170 iso=SERIALIZABLE stat=COMMITTED epo=0 ts=1536271361.068457287,0 orig=1536271361.068457287,0 max=1536271361.073978228,0 wto=false rop=false seq=11 int=5

The "async rollback" part refers to the rollback being done with a
canceled ctx (presumably a dropped connection's ctx). I believe the
error happens because there's a commit in flight when the ctx is
canceled. This patch lowers the message's level for this case.

Release note: None

When the TPCC loadgen is CTRL-C'ed, the logs are spammed with:
I180906 22:02:41.239771 27177127 internal/client/txn.go:625  [n1] async rollback failed: TransactionStatusError: already committed (REASON_UNKNOWN): "sql txn" id=57c518aa key=/Table/61/1/1224/0 rw=true pri=0.03781170 iso=SERIALIZABLE stat=COMMITTED epo=0 ts=1536271361.068457287,0 orig=1536271361.068457287,0 max=1536271361.073978228,0 wto=false rop=false seq=11 int=5

The "async rollback" part refers to the rollback being done with a
canceled ctx (presumably a dropped connection's ctx). I believe the
error happens because there's a commit in flight when the ctx is
canceled. This patch lowers the message's level for this case.

Release note: None
@andreimatei andreimatei requested a review from a team September 7, 2018 17:34
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@andreimatei
Copy link
Copy Markdown
Contributor Author

cc @jordanlewis

Copy link
Copy Markdown
Member

@jordanlewis jordanlewis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - thanks!

// with a REASON_TXN_COMMITTED.
func NewTransactionCommittedStatusError() *TransactionStatusError {
return &TransactionStatusError{
Msg: "already committed",
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this Msg seems redundant with the Reason - what if you just left it out?

Copy link
Copy Markdown
Contributor Author

@andreimatei andreimatei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bors r+

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained


pkg/roachpb/errors.go, line 417 at r1 (raw file):

Previously, jordanlewis (Jordan Lewis) wrote…

this Msg seems redundant with the Reason - what if you just left it out?

meh, I dunno. A bunch of callers still use the legacy catchall reason with a message. I'll leave it.

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Sep 10, 2018

Build failed

Copy link
Copy Markdown
Contributor Author

@andreimatei andreimatei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bors r+

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained

craig bot pushed a commit that referenced this pull request Sep 10, 2018
16354: storage: Delay initial metrics until system config is ready r=bdarnell a=bdarnell

This was previously done too soon at startup and would clutter the
logs (mainly in tests) with "failed initial metrics computation".

Fixes #13560

29817: client, roachpb: eliminate log spam when loadgens are killed r=andreimatei a=andreimatei

When the TPCC loadgen is CTRL-C'ed, the logs are spammed with:
I180906 22:02:41.239771 27177127 internal/client/txn.go:625  [n1] async rollback failed: TransactionStatusError: already committed (REASON_UNKNOWN): "sql txn" id=57c518aa key=/Table/61/1/1224/0 rw=true pri=0.03781170 iso=SERIALIZABLE stat=COMMITTED epo=0 ts=1536271361.068457287,0 orig=1536271361.068457287,0 max=1536271361.073978228,0 wto=false rop=false seq=11 int=5

The "async rollback" part refers to the rollback being done with a
canceled ctx (presumably a dropped connection's ctx). I believe the
error happens because there's a commit in flight when the ctx is
canceled. This patch lowers the message's level for this case.

Release note: None

29987: github-post: add timeout handling to the stress issue poster r=andreimatei a=andreimatei

Add explicit support to the issue poster for timeouts:
- on all runs, publish an artifacts file with a list of slow tests
- when timeouts happen, distinguish between the case where the test
currently running at the timeout point is the culprit (i.e. if it is the
longest running test) versus situations where that test is just an
innocent bystender

This patch also spruces up the github-post script in various ways. Among
them there's now better support for running it on an input that comes
directly from a test run, and not from a stress wrapper.

Release note: None

30006: opt: fix panic during SELECT MIN(NULL) r=rytaft a=arjunravinarayan

Fixes #29695.

Release note (bug fix): fix a crash when SELECT MIN(NULL) was run with
the optimizer enabled.

30008: gossip: allow receipt of "loopback infos" r=bdarnell a=petermattis

Receipt of loopback infos was disabled in #29398, but doing so had the
unfortunate effect of allowing gossip state to temporarily diverge
between nodes. Rather than disallowing loopback infos, we now ratchet
the gossip monotonic clock in order to avoid the assertion around the
gossip highwater timestamps.

Fixes #29992
Fixes #20986

Release note: None

Co-authored-by: Ben Darnell <ben@cockroachlabs.com>
Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>
Co-authored-by: Arjun Narayan <arjun@cockroachlabs.com>
Co-authored-by: Peter Mattis <petermattis@gmail.com>
@craig
Copy link
Copy Markdown
Contributor

craig bot commented Sep 10, 2018

Build succeeded

@craig craig bot merged commit c32a379 into cockroachdb:master Sep 10, 2018
@andreimatei andreimatei deleted the rollback-spam branch September 10, 2018 19:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants