[CI] IndexAuditUpgradeIT.testAuditLogs fails #33867
Description
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.4+bwc-tests/8/console
Reproduction line
does not reproduce locally
./gradlew :x-pack:qa:rolling-upgrade:with-system-key:v6.3.1#upgradedClusterTestRunner \
-Dtests.seed=57D936B12E792F96 \
-Dtests.class=org.elasticsearch.upgrades.IndexAuditUpgradeIT \
-Dtests.method="testAuditLogs" \
-Dtests.security.manager=true \
-Dtests.locale=bg \
-Dtests.timezone=Pacific/Marquesas
Example relevant log:
00:00:46 > Throwable #1: java.lang.AssertionError: Found node buckets [{doc_count=2431, key=node-2}, {doc_count=2343, key=upgraded-node-0}, {doc_count=2175, key=node-1}, {doc_count=2106, key=upgraded-node-1}, {doc_count=891, key=node-0}]
00:00:46 > Expected: a collection with size <6>
00:00:46 > but: collection size was <5>
00:00:46 > at __randomizedtesting.SeedInfo.seed([57D936B12E792F96:300A75E395A08B0D]:0)
00:00:46 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testAuditLogs$0(IndexAuditUpgradeIT.java:70)
00:00:46 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:830)
00:00:46 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:804)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.testAuditLogs(IndexAuditUpgradeIT.java:68)
00:00:46 > at java.lang.Thread.run(Thread.java:748)
00:00:46 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=2431, key=node-2}, {doc_count=2175, key=node-1}, {doc_count=1983, key=upgraded-node-0}, {doc_count=1794, key=upgraded-node-1}, {doc_count=891, key=node-0}]
00:00:46 > Expected: a collection with size <6>
00:00:46 > but: collection size was <5>
00:00:46 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testAuditLogs$0(IndexAuditUpgradeIT.java:70)
00:00:46 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
00:00:46 > ... 39 more
00:00:46 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=2431, key=node-2}, {doc_count=2175, key=node-1}, {doc_count=1983, key=upgraded-node-0}, {doc_count=1794, key=upgraded-node-1}, {doc_count=891, key=node-0}]
00:00:46 > Expected: a collection with size <6>
00:00:46 > but: collection size was <5>
00:00:46 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testAuditLogs$0(IndexAuditUpgradeIT.java:70)
00:00:46 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
00:00:46 > ... 39 more
00:00:46 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=2431, key=node-2}, {doc_count=2175, key=node-1}, {doc_count=1983, key=upgraded-node-0}, {doc_count=1794, key=upgraded-node-1}, {doc_count=891, key=node-0}]
00:00:46 > Expected: a collection with size <6>
00:00:46 > but: collection size was <5>
00:00:46 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testAuditLogs$0(IndexAuditUpgradeIT.java:70)
00:00:46 2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+6.4+bwc-tests/x-pack/qa/rolling-upgrade/with-system-key/build/testrun/v6.3.1#upgradedClusterTestRunner/J0/temp/org.elasticsearch.upgrades.IndexAuditUpgradeIT_57D936B12E792F96-001
00:00:46 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
00:00:46 > ... 39 more
00:00:46 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=2431, key=node-2}, {doc_count=2175, key=node-1}, {doc_count=1983, key=upgraded-node-0}, {doc_count=1794, key=upgraded-node-1}, {doc_count=891, key=node-0}]
00:00:46 2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=174, maxMBSortInHeap=7.47073894872366, sim=RandomSimilarity(queryNorm=true): {}, locale=bg, timezone=Pacific/Marquesas
00:00:46 2> NOTE: Linux 4.4.0-1061-aws amd64/Oracle Corporation 1.8.0_181 (64-bit)/cpus=16,threads=1,free=433534592,total=514850816
00:00:46 > Expected: a collection with size <6>
00:00:46 > but: collection size was <5>
00:00:46 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:46 2> NOTE: All tests run in this JVM: [IndexingIT, RollupIDUpgradeIT, TokenBackwardsCompatibilityIT, IndexAuditUpgradeIT]
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testAuditLogs$0(IndexAuditUpgradeIT.java:70)
00:00:46 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
00:00:46 > ... 39 more
00:00:46 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=2431, key=node-2}, {doc_count=2175, key=node-1}, {doc_count=1983, key=upgraded-node-0}, {doc_count=1794, key=upgraded-node-1}, {doc_count=891, key=node-0}]
00:00:46 > Expected: a collection with size <6>
00:00:46 > but: collection size was <5>
00:00:46 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testAuditLogs$0(IndexAuditUpgradeIT.java:70)
00:00:46 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
00:00:46 > ... 39 more
00:00:46 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=2431, key=node-2}, {doc_count=2175, key=node-1}, {doc_count=1983, key=upgraded-node-0}, {doc_count=1794, key=upgraded-node-1}, {doc_count=891, key=node-0}]
00:00:46 > Expected: a collection with size <6>
00:00:46 > but: collection size was <5>
00:00:46 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testAuditLogs$0(IndexAuditUpgradeIT.java:70)
00:00:46 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
00:00:46 > ... 39 more
00:00:46 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=2431, key=node-2}, {doc_count=2175, key=node-1}, {doc_count=1983, key=upgraded-node-0}, {doc_count=1794, key=upgraded-node-1}, {doc_count=891, key=node-0}]
00:00:46 > Expected: a collection with size <6>
00:00:46 > but: collection size was <5>
00:00:46 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testAuditLogs$0(IndexAuditUpgradeIT.java:70)
00:00:46 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
00:00:46 > ... 39 more
00:00:46 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=2431, key=node-2}, {doc_count=2175, key=node-1}, {doc_count=1983, key=upgraded-node-0}, {doc_count=1794, key=upgraded-node-1}, {doc_count=891, key=node-0}]
00:00:46 > Expected: a collection with size <6>
00:00:46 > but: collection size was <5>
00:00:46 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testAuditLogs$0(IndexAuditUpgradeIT.java:70)
00:00:46 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
00:00:46 > ... 39 more
00:00:46 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=2431, key=node-2}, {doc_count=2175, key=node-1}, {doc_count=1983, key=upgraded-node-0}, {doc_count=1794, key=upgraded-node-1}, {doc_count=891, key=node-0}]
00:00:46 > Expected: a collection with size <6>
00:00:46 > but: collection size was <5>
00:00:46 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testAuditLogs$0(IndexAuditUpgradeIT.java:70)
00:00:46 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
00:00:46 > ... 39 more
00:00:46 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=2431, key=node-2}, {doc_count=2175, key=node-1}, {doc_count=1983, key=upgraded-node-0}, {doc_count=1794, key=upgraded-node-1}, {doc_count=891, key=node-0}]
00:00:46 > Expected: a collection with size <6>
00:00:46 > but: collection size was <5>
00:00:46 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testAuditLogs$0(IndexAuditUpgradeIT.java:70)
00:00:46 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
00:00:46 > ... 39 more
00:00:46 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=2431, key=node-2}, {doc_count=2175, key=node-1}, {doc_count=2025, key=upgraded-node-0}, {doc_count=1841, key=upgraded-node-1}, {doc_count=891, key=node-0}]
00:00:46 > Expected: a collection with size <6>
00:00:46 > but: collection size was <5>
00:00:46 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testAuditLogs$0(IndexAuditUpgradeIT.java:70)
00:00:46 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
00:00:46 > ... 39 more
00:00:46 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=2431, key=node-2}, {doc_count=2175, key=node-1}, {doc_count=2141, key=upgraded-node-0}, {doc_count=1933, key=upgraded-node-1}, {doc_count=891, key=node-0}]
00:00:46 > Expected: a collection with size <6>
00:00:46 > but: collection size was <5>
00:00:46 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testAuditLogs$0(IndexAuditUpgradeIT.java:70)
00:00:46 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
00:00:46 > ... 39 more
00:00:46 > Suppressed: java.lang.AssertionError: Found node buckets [{doc_count=2431, key=node-2}, {doc_count=2212, key=upgraded-node-0}, {doc_count=2175, key=node-1}, {doc_count=2000, key=upgraded-node-1}, {doc_count=891, key=node-0}]
00:00:46 > Expected: a collection with size <6>
00:00:46 > but: collection size was <5>
00:00:46 > at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.assertNumUniqueNodeNameBuckets(IndexAuditUpgradeIT.java:126)
00:00:46 > at org.elasticsearch.upgrades.IndexAuditUpgradeIT.lambda$testAuditLogs$0(IndexAuditUpgradeIT.java:70)
00:00:46 > at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:818)
00:00:46 > ... 39 more
00:00:46 Completed [4/6] in 10.40s, 1 test, 1 failure <<< FAILURES!
Frequency
It happened 8 times in August, not super frequent but seems consistent.
Metadata
Metadata
Assignees
Type
Fields
Give feedbackNo fields configured for issues without a type.