[BUG] Frequent test TIMEOUTs in 7.2, 7.4 and 8.0 #14196
Description
I am experiencing frequent test TIMEOUTs in version lines 7.2, 7.4 and 8.0, when I run them in Gentoo test environment. I noticed that after the latest security release of versions 7.2.10, 7.4.5 and 8.0.3. I don't think that I experienced the similar timeouts when I was testing the previous security batch, when 7.2.9, 7.4.4 and 8.0.2 was released but they are now also affected. grep -C5 TIMEOUT redis-{7.2.10,7.4.5,8.0.3}/temp/build.log gives me this:
redis-7.2.10/temp/build.log-[ignore]: hash with one huge field: large memory flag not provided
redis-7.2.10/temp/build.log-[91/91 done]: violations (1 seconds)
redis-7.2.10/temp/build.log-Testing solo test
redis-7.2.10/temp/build.log-[ok]: Active defrag (46825 ms)
redis-7.2.10/temp/build.log-[skip]: Active defrag eval scripts
redis-7.2.10/temp/build.log:[TIMEOUT]: clients state report follows.
redis-7.2.10/temp/build.log-sock561b0c21d610 => (IN PROGRESS) Active defrag big keys
redis-7.2.10/temp/build.log-Killing still running Redis server 1751
redis-7.2.10/temp/build.log-Killing still running Redis server 1797
redis-7.2.10/temp/build.log-Killing still running Redis server 1794
redis-7.2.10/temp/build.log-Killing still running Redis server 1967
--
redis-7.2.10/temp/build.log- 0 seconds - bitops-large-memory
redis-7.2.10/temp/build.log- 1 seconds - violations
redis-7.2.10/temp/build.log-
redis-7.2.10/temp/build.log-!!! WARNING The following tests failed:
redis-7.2.10/temp/build.log-
redis-7.2.10/temp/build.log:*** [TIMEOUT]: clients state report follows.
redis-7.2.10/temp/build.log-Cleanup: may take some time... OK
redis-7.2.10/temp/build.log- * ERROR: dev-db/redis-7.2.10::gentoo failed (test phase):
redis-7.2.10/temp/build.log- * Failed to run command: ./runtest
redis-7.2.10/temp/build.log- *
redis-7.2.10/temp/build.log- * Call stack:
--
redis-7.4.5/temp/build.log-[ok]: Active defrag big keys: cluster (28088 ms)
redis-7.4.5/temp/build.log-[ok]: Active defrag pubsub: cluster (35031 ms)
redis-7.4.5/temp/build.log-[ok]: Active Defrag HFE: cluster (8165 ms)
redis-7.4.5/temp/build.log-[ok]: Active defrag main dictionary: standalone (48346 ms)
redis-7.4.5/temp/build.log-[ok]: Active defrag eval scripts: standalone (5361 ms)
redis-7.4.5/temp/build.log:[TIMEOUT]: clients state report follows.
redis-7.4.5/temp/build.log-sock5555f20636e0 => (IN PROGRESS) Active defrag big keys: standalone
redis-7.4.5/temp/build.log-Killing still running Redis server 20333
redis-7.4.5/temp/build.log-
redis-7.4.5/temp/build.log- The End
redis-7.4.5/temp/build.log-
--
redis-7.4.5/temp/build.log- 162 seconds - integration/replication-psync
redis-7.4.5/temp/build.log- 1 seconds - bitops-large-memory
redis-7.4.5/temp/build.log-
redis-7.4.5/temp/build.log-!!! WARNING The following tests failed:
redis-7.4.5/temp/build.log-
redis-7.4.5/temp/build.log:*** [TIMEOUT]: clients state report follows.
redis-7.4.5/temp/build.log-Cleanup: may take some time... OK
redis-7.4.5/temp/build.log- * ERROR: dev-db/redis-7.4.5::gentoo failed (test phase):
redis-7.4.5/temp/build.log- * Failed to run command: ./runtest
redis-7.4.5/temp/build.log- *
redis-7.4.5/temp/build.log- * Call stack:
--
redis-8.0.3/temp/build.log-[ignore]: SETBIT values larger than UINT32_MAX and lzf_compress/lzf_decompress correctly: large memory flag not provided
redis-8.0.3/temp/build.log-[95/95 done]: bitops-large-memory (0 seconds)
redis-8.0.3/temp/build.log-Testing solo test
redis-8.0.3/temp/build.log-[ok]: Active defrag main dictionary: cluster (53920 ms)
redis-8.0.3/temp/build.log-[ok]: Active defrag eval scripts: cluster (7182 ms)
redis-8.0.3/temp/build.log:[TIMEOUT]: clients state report follows.
redis-8.0.3/temp/build.log-sock55bd580abee0 => (IN PROGRESS) Active defrag big keys: cluster
redis-8.0.3/temp/build.log-Killing still running Redis server 23878
redis-8.0.3/temp/build.log-
redis-8.0.3/temp/build.log- The End
redis-8.0.3/temp/build.log-
--
redis-8.0.3/temp/build.log- 240 seconds - integration/replication-psync
redis-8.0.3/temp/build.log- 0 seconds - bitops-large-memory
redis-8.0.3/temp/build.log-
redis-8.0.3/temp/build.log-!!! WARNING The following tests failed:
redis-8.0.3/temp/build.log-
redis-8.0.3/temp/build.log:*** [TIMEOUT]: clients state report follows.
redis-8.0.3/temp/build.log-Cleanup: may take some time... OK
redis-8.0.3/temp/build.log- * ERROR: dev-db/redis-8.0.3::gentoo failed (test phase):
redis-8.0.3/temp/build.log- * Failed to run command: ./runtest
redis-8.0.3/temp/build.log- *
redis-8.0.3/temp/build.log- * Call stack:
Interestingly, 6.2.19 versions is not affected, therefore, I bisected the code between 8.0.3 and 6.2.19. The git bisect pointed to the commit 98b3f52 - add test suite infra to test RESP3 attributes (#10247) and tests pass if I revert this change in 8.0.3. The tests are executed with following parameters in gentoo ebuild:
./runtest --clients 16 --skiptest '/Active defrag for argv retained by the main thread from IO thread.*' --skipunit unit/oom-score-adj --skiptest 'CONFIG SET rollback on apply error' --tlsand it is related to --tls parameter, because tests pass without it.