release-22.1: test_quit.tcl consistently fails #96167
Description
I'm getting test_quit.tcl errors on the latest on the release-22.1 branch. I believe this is because of #95888.
It mainfests as https://teamcity.cockroachdb.com/viewLog.html?buildId=8506765&buildTypeId=Cockroach_UnitTests_Acceptance&tab=buildResultsDiv, but I can reproduce this quite easily.
Temporary remediation
git rebase --onto 24b7cf5e00a55e58f2b3d4525bb5ac10bb5523e5 HEAD~<#commits> <branchname>
Before the change:
Running:
git checkout d79dfd24085d8e480badee4e72f9709b3af603c9
./dev build short
expect -f ./pkg/cli/interactive_tests/test_quit.tcl ./cockroach
I get the expected output:
.230130 19:57:05.N EXPECT TEST: BEGIN START SERVER
?column?
------------
1
(1 row)
Time: 1ms
.230130 19:57:06.N EXPECT TEST: START SERVER DONE
spawn /bin/bash
PS1=':''/# '
The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
bash-3.2$ PS1=':''/# '
:/#
.230130 19:57:06.N EXPECT TEST: START TEST: Test that quit with a very short timeout still proceeds with hard shutdown
./cockroach quit --insecure --drain-wait=1ns
Command "quit" is deprecated, see 'cockroach node drain' instead to drain a
server without terminating the server process (which can in turn be done using
an orchestration layer or a process manager, or by sending a termination signal
directly).
warning: draining a node without node ID or passing --self explicitly is deprecated.
node is draining...
W230130 08:57:06.926464 1 cli/quit.go:99 [-] 1 graceful shutdown may not have completed successfully; check the node's logs for details.
W230130 08:57:06.926480 1 cli/quit.go:103 [-] 2 drain did not complete successfully; hard shutdown may cause disruption
ok
:/#
.230130 19:57:06.N EXPECT TEST: END TEST
.230130 19:57:07.N EXPECT TEST: BEGIN STOP SERVER
.230130 19:57:07.N EXPECT TEST: END STOP SERVER
After the change
Running:
git checkout d79dfd24085d8e480badee4e72f9709b3af603c9
./dev build short
expect -f ./pkg/cli/interactive_tests/test_quit.tcl ./cockroach
I consistently get:
.230130 19:59:20.N EXPECT TEST: BEGIN START SERVER
?column?
------------
1
(1 row)
Time: 0ms
.230130 19:59:23.N EXPECT TEST: START SERVER DONE
spawn /bin/bash
PS1=':''/# '
The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
bash-3.2$ PS1=':''/# '
:/#
.230130 19:59:23.N EXPECT TEST: START TEST: Test that quit with a very short timeout still proceeds with hard shutdown
./cockroach quit --insecure --drain-wait=1ns
Command "quit" is deprecated, see 'cockroach node drain' instead to drain a
server without terminating the server process (which can in turn be done using
an orchestration layer or a process manager, or by sending a termination signal
directly).
warning: draining a node without node ID or passing --self explicitly is deprecated.
node is draining...
ERROR: drain timeout, consider adjusting --drain-wait, especially under custom server.shutdown.{drain,query,connection,lease_transfer}_wait cluster settings
Failed running "quit"
cc @ecwall
Jira issue: CRDB-23978