This has been failing a bunch of times on 7.10 recently:
https://gradle-enterprise.elastic.co/s/ppzyiud65lopu
org.elasticsearch.xpack.ml.integration.TooManyJobsIT > testSingleNode FAILED
java.lang.AssertionError: Could not open job because no suitable nodes were found, allocation explanation [Not opening job [max-number-of-jobs-limit-job-7] on node [{node_t1}{ml.machine_memory=0}{ml.max_open_jobs=6}], because this node is full. Number of opened jobs [6], xpack.ml.max_open_jobs [6]]
at __randomizedtesting.SeedInfo.seed([1A5A7D808B0C9B12:FAEAE838E7C43CA8]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.elasticsearch.xpack.ml.integration.TooManyJobsIT.verifyMaxNumberOfJobsLimit(TooManyJobsIT.java:172)
at org.elasticsearch.xpack.ml.integration.TooManyJobsIT.testSingleNode(TooManyJobsIT.java:129)
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
:x-pack:plugin:ccr:qa:restart:followClusterRestartTest
REPRODUCE WITH: ./gradlew ':x-pack:plugin:ml:internalClusterTest' --tests "org.elasticsearch.xpack.ml.integration.TooManyJobsIT.testSingleNode" -Dtests.seed=1A5A7D808B0C9B12 -Dtests.security.manager=true -Dtests.locale=zh -Dtests.timezone=Etc/GMT-1 -Druntime.java=8
org.elasticsearch.xpack.ml.integration.TooManyJobsIT > testMultipleNodes FAILED
java.lang.AssertionError: Could not open job because no suitable nodes were found, allocation explanation [Not opening job [max-number-of-jobs-limit-job-10] on node [{node_t2}{ml.machine_memory=0}{ml.max_open_jobs=3}], because this node is full. Number of opened jobs [3], xpack.ml.max_open_jobs [3]|Not opening job [max-number-of-jobs-limit-job-10] on node [{node_t3}{ml.machine_memory=0}{ml.max_open_jobs=3}], because this node is full. Number of opened jobs [3], xpack.ml.max_open_jobs [3]|Not opening job [max-number-of-jobs-limit-job-10] on node [{node_t1}{ml.machine_memory=0}{ml.max_open_jobs=3}], because this node is full. Number of opened jobs [3], xpack.ml.max_open_jobs [3]]
at __randomizedtesting.SeedInfo.seed([1A5A7D808B0C9B12:B4CFD6BE498345BD]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.elasticsearch.xpack.ml.integration.TooManyJobsIT.verifyMaxNumberOfJobsLimit(TooManyJobsIT.java:172)
at org.elasticsearch.xpack.ml.integration.TooManyJobsIT.testMultipleNodes(TooManyJobsIT.java:133)
REPRODUCE WITH: ./gradlew ':x-pack:plugin:ml:internalClusterTest' --tests "org.elasticsearch.xpack.ml.integration.TooManyJobsIT.testMultipleNodes" -Dtests.seed=1A5A7D808B0C9B12 -Dtests.security.manager=true -Dtests.locale=zh -Dtests.timezone=Etc/GMT-1 -Druntime.java=8
Interestingly enough, instances of this failure coincide with the following REST test failure twice today.
REPRODUCE WITH: ./gradlew ':x-pack:plugin:yamlRestTest' --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=ml/jobs_crud/Test put job with model_memory_limit as string and lazy open}" -Dtests.seed=1A5A7D808B0C9B12 -Dtests.security.manager=true -Dtests.locale=de-DE -Dtests.timezone=Africa/Ouagadougou -Druntime.java=8 -Dtests.rest.blacklist=getting_started/10_monitor_cluster_health/*
org.elasticsearch.xpack.test.rest.XPackRestIT > test {p0=ml/ml_info/Test ml info} FAILED
java.lang.AssertionError: Failure at [ml/ml_info:21]: field [limits.effective_max_model_memory_limit] was expected to be of type String but is an instanceof [null]
Expected: an instance of java.lang.String
but: null
at __randomizedtesting.SeedInfo.seed([1A5A7D808B0C9B12:920E425A25F0F6EA]:0)
at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:414)
at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:391)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by:
java.lang.AssertionError: field [limits.effective_max_model_memory_limit] was expected to be of type String but is an instanceof [null]
Expected: an instance of java.lang.String
but: null
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
at org.junit.Assert.assertThat(Assert.java:956)
at org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:63)
at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:76)
at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:407)
... 37 more
This has been failing a bunch of times on 7.10 recently:
https://gradle-enterprise.elastic.co/s/ppzyiud65lopu
Interestingly enough, instances of this failure coincide with the following REST test failure twice today.