Skip to content

[CI] FullClusterRestartIT.testSearch fails on master #25311

@dimitris-athanasiou

Description

@dimitris-athanasiou

Link to CI: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+bwc-tests/81/console

Reproduce with:

gradle :qa:full-cluster-restart:v5.0.2#upgradedClusterTestRunner -Dtests.seed=6FCA8286356192B5 -Dtests.class=org.elasticsearch.upgrades.FullClusterRestartIT -Dtests.method="testSearch" -Dtests.security.manager=true -Dtests.locale=id -Dtests.timezone=Etc/GMT+7

Could not reproduce it.

Full error:

Suite: org.elasticsearch.upgrades.FullClusterRestartIT
  1> [2017-06-20T08:46:31,294][WARN ][o.e.b.JNANatives         ] unable to install syscall filter: 
  1> java.lang.UnsupportedOperationException: seccomp unavailable: CONFIG_SECCOMP not compiled into kernel, CONFIG_SECCOMP and CONFIG_SECCOMP_FILTER are needed
  1> 	at org.elasticsearch.bootstrap.SystemCallFilter.linuxImpl(SystemCallFilter.java:363) ~[elasticsearch-6.0.0-alpha3-SNAPSHOT.jar:6.0.0-alpha3-SNAPSHOT]
  1> 	at org.elasticsearch.bootstrap.SystemCallFilter.init(SystemCallFilter.java:638) ~[elasticsearch-6.0.0-alpha3-SNAPSHOT.jar:6.0.0-alpha3-SNAPSHOT]
  1> 	at org.elasticsearch.bootstrap.JNANatives.tryInstallSystemCallFilter(JNANatives.java:215) [elasticsearch-6.0.0-alpha3-SNAPSHOT.jar:6.0.0-alpha3-SNAPSHOT]
  1> 	at org.elasticsearch.bootstrap.Natives.tryInstallSystemCallFilter(Natives.java:99) [elasticsearch-6.0.0-alpha3-SNAPSHOT.jar:6.0.0-alpha3-SNAPSHOT]
  1> 	at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:110) [elasticsearch-6.0.0-alpha3-SNAPSHOT.jar:6.0.0-alpha3-SNAPSHOT]
  1> 	at org.elasticsearch.bootstrap.BootstrapForTesting.<clinit>(BootstrapForTesting.java:80) [framework-6.0.0-alpha3-SNAPSHOT.jar:6.0.0-alpha3-SNAPSHOT]
  1> 	at org.elasticsearch.test.ESTestCase.<clinit>(ESTestCase.java:186) [framework-6.0.0-alpha3-SNAPSHOT.jar:6.0.0-alpha3-SNAPSHOT]
  1> 	at java.lang.Class.forName0(Native Method) ~[?:1.8.0_131]
  1> 	at java.lang.Class.forName(Class.java:348) [?:1.8.0_131]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:592) [randomizedtesting-runner-2.5.0.jar:?]
  1> [2017-06-20T08:46:31,313][WARN ][o.e.b.JNANatives         ] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
  1> [2017-06-20T08:46:31,313][WARN ][o.e.b.JNANatives         ] This can result in part of the JVM being swapped out.
  1> [2017-06-20T08:46:31,314][WARN ][o.e.b.JNANatives         ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
  1> [2017-06-20T08:46:31,314][WARN ][o.e.b.JNANatives         ] These can be adjusted by modifying /etc/security/limits.conf, for example: 
  1> 	# allow user 'jenkins' mlockall
  1> 	jenkins soft memlock unlimited
  1> 	jenkins hard memlock unlimited
  1> [2017-06-20T08:46:31,314][WARN ][o.e.b.JNANatives         ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
  1> [2017-06-20T01:46:32,052][INFO ][o.e.u.FullClusterRestartIT] [testSingleDoc]: before test
  1> [2017-06-20T01:46:32,074][INFO ][o.e.u.FullClusterRestartIT] initializing REST clients against [http://[::1]:33081, http://[::1]:39135]
  1> [2017-06-20T01:46:32,826][INFO ][o.e.u.FullClusterRestartIT] [testSingleDoc]: after test
  1> [2017-06-20T01:46:32,847][INFO ][o.e.u.FullClusterRestartIT] [testNewReplicasWork]: before test
  1> [2017-06-20T01:46:33,367][INFO ][o.e.u.FullClusterRestartIT] [testNewReplicasWork]: after test
  1> [2017-06-20T01:46:33,372][INFO ][o.e.u.FullClusterRestartIT] [testSearch]: before test
  1> [2017-06-20T01:46:33,384][INFO ][o.e.u.FullClusterRestartIT] --> testing basic search
  1> [2017-06-20T01:46:33,393][INFO ][o.e.u.FullClusterRestartIT] Found 2482 in old index
  1> [2017-06-20T01:46:33,393][INFO ][o.e.u.FullClusterRestartIT] --> testing basic search with sort
  1> [2017-06-20T01:46:33,422][INFO ][o.e.u.FullClusterRestartIT] --> testing exists filter
  1> [2017-06-20T01:46:33,443][INFO ][o.e.u.FullClusterRestartIT] --> testing _all search
  1> [2017-06-20T01:46:34,195][INFO ][o.e.u.FullClusterRestartIT] [testSearch]: after test
  2> REPRODUCE WITH: gradle :qa:full-cluster-restart:v5.0.2#upgradedClusterTestRunner -Dtests.seed=6FCA8286356192B5 -Dtests.class=org.elasticsearch.upgrades.FullClusterRestartIT -Dtests.method="testSearch" -Dtests.security.manager=true -Dtests.locale=id -Dtests.timezone=Etc/GMT+7
FAILURE 0.84s | FullClusterRestartIT.testSearch <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: expected:<0> but was:<519772>
   > 	at __randomizedtesting.SeedInfo.seed([6FCA8286356192B5:FD7317D37178CA1A]:0)
   > 	at org.elasticsearch.upgrades.FullClusterRestartIT.assertUpgradeWorks(FullClusterRestartIT.java:435)
   > 	at org.elasticsearch.upgrades.FullClusterRestartIT.testSearch(FullClusterRestartIT.java:149)
   > 	at java.lang.Thread.run(Thread.java:748)
  1> [2017-06-20T01:46:34,229][INFO ][o.e.u.FullClusterRestartIT] [testSnapshotRestore]: before test
  1> [2017-06-20T01:46:34,908][INFO ][o.e.u.FullClusterRestartIT] There are still tasks running after this test that might break subsequent tests [internal:index/shard/recovery/file_chunk, internal:index/shard/recovery/start_recovery].
  1> [2017-06-20T01:46:34,909][INFO ][o.e.u.FullClusterRestartIT] [testSnapshotRestore]: after test
  1> [2017-06-20T01:46:34,917][INFO ][o.e.u.FullClusterRestartIT] [testRecovery]: before test
  1> [2017-06-20T01:46:35,010][INFO ][o.e.u.FullClusterRestartIT] There are still tasks running after this test that might break subsequent tests [internal:index/shard/recovery/file_chunk, internal:index/shard/recovery/start_recovery].
  1> [2017-06-20T01:46:35,010][INFO ][o.e.u.FullClusterRestartIT] [testRecovery]: after test
  1> [2017-06-20T01:46:35,026][INFO ][o.e.u.FullClusterRestartIT] [testAliasWithBadName]: before test
  1> [2017-06-20T01:46:35,052][ERROR][o.e.u.FullClusterRestartIT] clusterState={cluster_name=full-cluster-restart, compressed_size_in_bytes=3199, metadata={indices={restored_testsnapshotrestore={settings={index={number_of_shards=5, provided_name=testsnapshotrestore, creation_date=1497948353547, number_of_replicas=1, uuid=5YSqwvBaRSauR32PZ2lrzA, version={upgraded=6000003, created=5000299}}}, mappings={doc={_all={enabled=true}, properties={field={type=text, fields={keyword={ignore_above=256, type=keyword}}}}}}, aliases=[], state=open, in_sync_allocations={0=[gCTVGM0IRMu6FiCW_wafEQ], 1=[-lf2VFE8Tve2nHYQKJ7cDg], 2=[vQV9eUwUS0KjWcuqWVXDPA], 3=[2Wc6p_sCRaaqSNkEknX1Uw], 4=[YJi_6tgCTvOKDs0u9AEGHA]}, primary_terms={0=2, 1=2, 2=2, 3=2, 4=2}}, testsearch={settings={index={refresh_interval=1s, number_of_shards=1, provided_name=testsearch, creation_date=1497948357550, number_of_replicas=0, uuid=0gFkIub6QdW3sFf6hUO25A, version={upgraded=6000003, created=5000299}}}, mappings={doc={_all={enabled=true}, properties={bool={type=boolean}, field={properties={with={properties={dots={type=text, fields={keyword={ignore_above=256, type=keyword}}}}}}}, string={type=text}, binary={store=true, type=binary}, foo={type=text, fields={keyword={ignore_above=256, type=keyword}}}, float={type=float}, dots_in_field_names={type=text}, int={type=long}}}}, aliases=[], state=open, in_sync_allocations={0=[aDrIAXM7RmGaGB7RPqslIw]}, primary_terms={0=2}}, testaliaswithbadname={settings={index={number_of_shards=1, provided_name=testaliaswithbadname, creation_date=1497948347228, number_of_replicas=0, uuid=lx09u25HQdyp_KLfb35neA, version={upgraded=6000003, created=5000299}}}, mappings={doc={_all={enabled=true}, properties={key={type=keyword}}}}, aliases=[#testaliaswithbadname], state=open, in_sync_allocations={0=[5t10L5PXQgCT43ZjRCMtQA]}, primary_terms={0=2}}, testsingledoc={settings={index={number_of_shards=5, provided_name=testsingledoc, creation_date=1497948330341, number_of_replicas=1, uuid=M71hphp7SR6nsXv8vQKccg, version={upgraded=6000003, created=5000299}}}, mappings={doc={_all={enabled=true}, properties={test={type=text, fields={keyword={ignore_above=256, type=keyword}}}}}}, aliases=[], state=open, in_sync_allocations={0=[aBqjfD-QQaOl9ttQcAtB5Q, 6lc8KdrtSKqgThoU4xN-bw], 1=[bmfqVGrbT9OvoXwrKWesgQ, VY4FoetYQT-LA536Njtlgg], 2=[nW_fTTjIQbGI_uAj8N_d6Q, yyjITc0gT-qdawPDzmR5Uw], 3=[fT_K4f37Sma_TuWjChDCig, gSh_oykiRg-ufZKiVuRtQw], 4=[TzVLSWQ1TBq6_I4OSA-jEg, yxZyliQUSZGKE_NnNKAsfA]}, primary_terms={0=2, 1=2, 2=2, 3=2, 4=2}}, testnewreplicaswork={settings={index={number_of_shards=1, provided_name=testnewreplicaswork, creation_date=1497948331660, number_of_replicas=1, uuid=JMpUI6ExQXCQNseTRxMhVg, version={upgraded=6000003, created=5000299}}}, mappings={doc={_all={enabled=true}, properties={field={type=text}}}}, aliases=[], state=open, in_sync_allocations={0=[RBqacL0QSGuGpcT3SnoqXA, Jq6cXpdrQ_Gyromx5iKhdQ]}, primary_terms={0=2}}, testsnapshotrestore={settings={index={number_of_shards=5, provided_name=testsnapshotrestore, creation_date=1497948353547, number_of_replicas=1, uuid=jKVtQRQeQtmcf4EAoFsMlQ, version={upgraded=6000003, created=5000299}}}, mappings={doc={_all={enabled=true}, properties={field={type=text, fields={keyword={ignore_above=256, type=keyword}}}}}}, aliases=[], state=open, in_sync_allocations={0=[eCQpxsSrQb-mW3p55MC92Q, hIbMtnj2QvqHXnmavttFXA], 1=[NY5pk7fpSvK2jGXbO3hAsw, w5Qg9IfBQPCVG5IdIA1T0g], 2=[jlLuniShQxaREOf19wo7dQ, cIpyM7g8SmSG8nZnh1vhPA], 3=[USGzVDKtRvS_1STKFqMpnw, d6Ckq7XYQJKwn1WMz6CGuA], 4=[j6OBWcWJT7S2FmXgaDDFUg, kIPn-9otTMuNHb-zSUhcAQ]}, primary_terms={0=2, 1=2, 2=2, 3=2, 4=2}}, testrecovery={settings={index={number_of_shards=5, provided_name=testrecovery, creation_date=1497948348582, number_of_replicas=1, uuid=glGyYBQQR7SWlFOrQjS3HQ, version={upgraded=6000003, created=5000299}}}, mappings={doc={_all={enabled=true}, properties={field={type=text, fields={keyword={ignore_above=256, type=keyword}}}}}}, aliases=[], state=open, in_sync_allocations={0=[qBeilrG6SbKZgEyDFGqnTg, TeiSagr4So-Unx5-d2pd6w], 1=[syrkQ478RUmAabauMhVA1g, vNjfVU5LRC2KEBAKnXaGlQ], 2=[_p4VoqjISbW46-Jpy8CmPQ, nlmREv_MTSKQ0SNXG_d4Gw], 3=[NamlkbNNRSS77J_nv7kWdA, S2EVbn48Tt2y9Zl3yAzmjA], 4=[w677kVvFRdmcFNfdbZyO1A, Jx6nu2-RRPmx_XX1jcFzIQ]}, primary_terms={0=2, 1=2, 2=2, 3=2, 4=2}}, info={settings={index={number_of_shards=5, provided_name=info, creation_date=1497948348164, number_of_replicas=1, uuid=8vGlHvyKSUyDRGpJvM_0Pg, version={upgraded=6000003, created=5000299}}}, mappings={doc={_all={enabled=true}, properties={value={type=text, fields={keyword={ignore_above=256, type=keyword}}}}}}, aliases=[], state=open, in_sync_allocations={0=[CBfSbTVESnKzkYC6NHYG0A, -4S7fFFaSuqnie_qXw51iA], 1=[MsfoHJI1SmyaAEfL3q8a-A, m9eTciEGQtKmTPxjl0_PnA], 2=[CPhCk50tQV2f_Eo99iASDw, daNPgIFsRKeEHuodR_opUQ], 3=[SeKnywUyRi2uVE_8CeGV7A, nI_vmrLaRZSZUTDX84Tgpg], 4=[6u6Y9mqwQ52EZrIU7NJc0Q, NsT677n7Sd6t0sPsHNCFyA]}, primary_terms={0=2, 1=2, 2=2, 3=2, 4=2}}}, cluster_uuid=pb6OYrMmT6mBUSbmB3LAfw, repositories={repo={settings={compress=true, location=/var/lib/jenkins/workspace/elastic+elasticsearch+master+bwc-tests/qa/full-cluster-restart/build/cluster/shared/repo}, type=fs}}, templates={}, index-graveyard={tombstones=[{index={index_uuid=lWxKhL6DSyilySChFRrWnQ, index_name=restored_testsnapshotrestore}, delete_date_in_millis=1497948394266}]}}}
  2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+master+bwc-tests/qa/full-cluster-restart/build/testrun/v5.0.2#upgradedClusterTestRunner/J0/temp/org.elasticsearch.upgrades.FullClusterRestartIT_6FCA8286356192B5-001
  1> [2017-06-20T01:46:35,185][INFO ][o.e.u.FullClusterRestartIT] There are still tasks running after this test that might break subsequent tests [internal:index/shard/recovery/file_chunk, internal:index/shard/recovery/start_recovery].
  2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1823, maxMBSortInHeap=5.933591739645208, sim=RandomSimilarity(queryNorm=true): {}, locale=id, timezone=Etc/GMT+7
  1> [2017-06-20T01:46:35,185][INFO ][o.e.u.FullClusterRestartIT] [testAliasWithBadName]: after test
  2> NOTE: Linux 2.6.32-642.15.1.el6.x86_64 amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=466418616,total=514850816
  2> NOTE: All tests run in this JVM: [FullClusterRestartIT]
Completed [1/1] in 4.79s, 6 tests, 1 failure <<< FAILURES!

Metadata

Metadata

Assignees

Labels

:Core/Infra/CoreCore issues without another label>testIssues or PRs that are addressing/adding tests>test-failureTriaged test failures from CIv6.0.0-beta1

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions