Skip to content

[CI] Problems in mixed cluster cleanup in ESRestTestCase#wipeRollupJobs #36044

@cbuescher

Description

@cbuescher

Build: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+default-distro-bwc-tests/131/console

This one has a bunch of failures in the mixed cluster tests:

org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT test {p0=index/60_refresh/refresh=wait_for waits until changes are visible in search}
org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT test {p0=indices.put_template/10_basic/Put multiple template}
org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT test {p0=indices.update_aliases/20_routing/Numeric Routing}
org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT test {p0=nodes.stats/10_basic/Nodes stats level}
org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT test {p0=nodes.stats/11_indices_metrics/Metric - multi}
org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT test {p0=search.aggregation/40_range/Date Range Missing}
org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT test {p0=search/160_exists_query/Test exists query on mapped date field}
org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT test {p0=search/160_exists_query/Test exists query on unmapped geo_point field}
org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT test {p0=search/30_limits/Docvalues_fields size limit}
org.elasticsearch.backwards.IndexingIT testIndexVersionPropagation

The failures look similar to this, so it probably not be the tests itself that fail but some cleanup:

org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:46698], URI [/_rollup/job/_all], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"_rollup","index_uuid":"_na_","index":"_rollup"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"_rollup","index_uuid":"_na_","index":"_rollup"},"status":404}
	at __randomizedtesting.SeedInfo.seed([3DF3A00DADE684BC:B5A79FD7031AE944]:0)
	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:687)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218)
	at org.elasticsearch.test.rest.ESRestTestCase.wipeRollupJobs(ESRestTestCase.java:477)
	at org.elasticsearch.test.rest.ESRestTestCase.wipeCluster(ESRestTestCase.java:407)
	at org.elasticsearch.test.rest.ESRestTestCase.cleanUpCluster(ESRestTestCase.java:193)

I tried to reproduce with one of the reproduce lines on master:

./gradlew :qa:mixed-cluster:v6.6.0#mixedClusterTestRunner \
  -Dtests.seed=3DF3A00DADE684BC \
  -Dtests.class=org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT \
  -Dtests.method="test {p0=search/160_exists_query/Test exists query on mapped date field}" \
  -Dtests.security.manager=true \
  -Dtests.locale=ar \
  -Dtests.timezone=Kwajalein \
  -Dtests.distribution=zip \
  -Dcompiler.java=11 \
  -Druntime.java=8

And got this:

ERROR   7.60s | MixedClusterClientYamlTestSuiteIT.test {p0=search/160_exists_query/Test exists query on mapped date field} <<< FAILURES!
   > Throwable #1: org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:52996], URI [/_rollup/job/_all], status line [HTTP/1.1 404 Not Found]
   > {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"_rollup","index_uuid":"_na_","index":"_rollup"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"_rollup","index_uuid":"_na_","index":"_rollup"},"status":404}
   >    at __randomizedtesting.SeedInfo.seed([3DF3A00DADE684BC:B5A79FD7031AE944]:0)
   >    at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:687)
   >    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218)
   >    at org.elasticsearch.test.rest.ESRestTestCase.wipeRollupJobs(ESRestTestCase.java:477)
   >    at org.elasticsearch.test.rest.ESRestTestCase.wipeCluster(ESRestTestCase.java:407)
   >    at org.elasticsearch.test.rest.ESRestTestCase.cleanUpCluster(ESRestTestCase.java:193)
   >    at java.lang.Thread.run(Thread.java:748)
   > Caused by: org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:52996], URI [/_rollup/job/_all], status line [HTTP/1.1 404 Not Found]
   > {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"_rollup","index_uuid":"_na_","index":"_rollup"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_expression","resource.id":"_rollup","index_uuid":"_na_","index":"_rollup"},"status":404}
   >    at org.elasticsearch.client.RestClient$1.completed(RestClient.java:307)
   >    at org.elasticsearch.client.RestClient$1.completed(RestClient.java:292)
   >    at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
   >    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
   >    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
   >    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
   >    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
   >    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
   >    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
   >    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
   >    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
   >    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
   >    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
   >    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
   >    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   >    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
   >    ... 1 more
  2> NOTE: leaving temporary files on disk at: /Users/christoph/workspace/elasticsearch/qa/mixed-cluster/build/testrun/v6.6.0#mixedClusterTestRunner/J0/temp/org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT_3DF3A00DADE684BC-001
  2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=1457, maxMBSortInHeap=7.982721853983626, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@138084b3), locale=ar, timezone=Kwajalein
  2> NOTE: Mac OS X 10.14.1 x86_64/Oracle Corporation 1.8.0_181 (64-bit)/cpus=8,threads=1,free=445255424,total=514850816
  2> NOTE: All tests run in this JVM: [MixedClusterClientYamlTestSuiteIT]

Metadata

Metadata

Assignees

Labels

:StorageEngine/RollupTurn fine-grained time-based data into coarser-grained data>test-failureTriaged test failures from CI

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions