Skip to content

storage: Bring back MaxSyncDuration env var, fix disk-stalled roachtest#56064

Merged
craig[bot] merged 1 commit intocockroachdb:masterfrom
itsbilal:fix-disk-stall-roachtest
Oct 28, 2020
Merged

storage: Bring back MaxSyncDuration env var, fix disk-stalled roachtest#56064
craig[bot] merged 1 commit intocockroachdb:masterfrom
itsbilal:fix-disk-stall-roachtest

Conversation

@itsbilal
Copy link
Copy Markdown
Contributor

The disk-stalled roachtest relies on the ability to control disk stall
detection / fatal intervals, as charybdefs only injects 50ms of
delay per syscall. This change adds an env variable, similar to the
one removed in #55186 to set max sync duration, except now it governs
the default of the cluster setting. The roachtest now modifies that
env variable to let disk stall detection trip on short syscall
delays.

Fixes #54332.

Release note: None.

@itsbilal itsbilal self-assigned this Oct 28, 2020
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

Copy link
Copy Markdown
Collaborator

@petermattis petermattis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @itsbilal, @jbowens, and @petermattis)


pkg/cmd/roachtest/disk_stall.go, line 100 at r1 (raw file):

	dur := 10 * time.Minute
	if !affectsDataDir && !affectsLogDir {
		dur = 65 * time.Second

Did this duration need to change?

The disk-stalled roachtest relies on the ability to control disk stall
detection / fatal intervals, as charybdefs only injects 50ms of
delay per syscall. This change adds an env variable, similar to the
one removed in cockroachdb#55186 to set max sync duration, except now it governs
the default of the cluster setting. The roachtest now modifies that
env variable to let disk stall detection trip on short syscall
delays.

Fixes cockroachdb#54332.

Release note: None.
@itsbilal itsbilal force-pushed the fix-disk-stall-roachtest branch from e656a64 to 79f1751 Compare October 28, 2020 18:42
Copy link
Copy Markdown
Contributor Author

@itsbilal itsbilal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TFTR!

Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @jbowens and @petermattis)


pkg/cmd/roachtest/disk_stall.go, line 100 at r1 (raw file):

Previously, petermattis (Peter Mattis) wrote…

Did this duration need to change?

It doesn't. Reverted.

@itsbilal
Copy link
Copy Markdown
Contributor Author

bors r+

@craig
Copy link
Copy Markdown
Contributor

craig bot commented Oct 28, 2020

Build succeeded:

@craig craig bot merged commit 67dab4c into cockroachdb:master Oct 28, 2020
itsbilal added a commit to itsbilal/cockroach that referenced this pull request Nov 2, 2020
The COCKROACH_ENGINE_MAX_SYNC_DEFAULT* env variables were overhauled
after 20.2. The roachtest was updated in cockroachdb#56064 to use the new
env vars that yield to cluster settings. However, that roachtest
was also not updated to work correctly with <=20.2 cockroach that
still looked for the older env vars, so this change makes that
fix.

Fixes cockroachdb#56174.

Release note: None
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

roachtest: disk-stalled/log=false,data=true failed

3 participants