Skip to content

Failures in ML TooManyJobsIT on Debian 8 #66885

@original-brownbear

Description

@original-brownbear

This has been failing a bunch of times on 7.10 recently:

https://gradle-enterprise.elastic.co/s/ppzyiud65lopu


org.elasticsearch.xpack.ml.integration.TooManyJobsIT > testSingleNode FAILED
    java.lang.AssertionError: Could not open job because no suitable nodes were found, allocation explanation [Not opening job [max-number-of-jobs-limit-job-7] on node [{node_t1}{ml.machine_memory=0}{ml.max_open_jobs=6}], because this node is full. Number of opened jobs [6], xpack.ml.max_open_jobs [6]]
        at __randomizedtesting.SeedInfo.seed([1A5A7D808B0C9B12:FAEAE838E7C43CA8]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.elasticsearch.xpack.ml.integration.TooManyJobsIT.verifyMaxNumberOfJobsLimit(TooManyJobsIT.java:172)
        at org.elasticsearch.xpack.ml.integration.TooManyJobsIT.testSingleNode(TooManyJobsIT.java:129)
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
:x-pack:plugin:ccr:qa:restart:followClusterRestartTest
REPRODUCE WITH: ./gradlew ':x-pack:plugin:ml:internalClusterTest' --tests "org.elasticsearch.xpack.ml.integration.TooManyJobsIT.testSingleNode" -Dtests.seed=1A5A7D808B0C9B12 -Dtests.security.manager=true -Dtests.locale=zh -Dtests.timezone=Etc/GMT-1 -Druntime.java=8

org.elasticsearch.xpack.ml.integration.TooManyJobsIT > testMultipleNodes FAILED
    java.lang.AssertionError: Could not open job because no suitable nodes were found, allocation explanation [Not opening job [max-number-of-jobs-limit-job-10] on node [{node_t2}{ml.machine_memory=0}{ml.max_open_jobs=3}], because this node is full. Number of opened jobs [3], xpack.ml.max_open_jobs [3]|Not opening job [max-number-of-jobs-limit-job-10] on node [{node_t3}{ml.machine_memory=0}{ml.max_open_jobs=3}], because this node is full. Number of opened jobs [3], xpack.ml.max_open_jobs [3]|Not opening job [max-number-of-jobs-limit-job-10] on node [{node_t1}{ml.machine_memory=0}{ml.max_open_jobs=3}], because this node is full. Number of opened jobs [3], xpack.ml.max_open_jobs [3]]
        at __randomizedtesting.SeedInfo.seed([1A5A7D808B0C9B12:B4CFD6BE498345BD]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.junit.Assert.assertTrue(Assert.java:41)
        at org.elasticsearch.xpack.ml.integration.TooManyJobsIT.verifyMaxNumberOfJobsLimit(TooManyJobsIT.java:172)
        at org.elasticsearch.xpack.ml.integration.TooManyJobsIT.testMultipleNodes(TooManyJobsIT.java:133)
REPRODUCE WITH: ./gradlew ':x-pack:plugin:ml:internalClusterTest' --tests "org.elasticsearch.xpack.ml.integration.TooManyJobsIT.testMultipleNodes" -Dtests.seed=1A5A7D808B0C9B12 -Dtests.security.manager=true -Dtests.locale=zh -Dtests.timezone=Etc/GMT-1 -Druntime.java=8

Interestingly enough, instances of this failure coincide with the following REST test failure twice today.

REPRODUCE WITH: ./gradlew ':x-pack:plugin:yamlRestTest' --tests "org.elasticsearch.xpack.test.rest.XPackRestIT.test {p0=ml/jobs_crud/Test put job with model_memory_limit as string and lazy open}" -Dtests.seed=1A5A7D808B0C9B12 -Dtests.security.manager=true -Dtests.locale=de-DE -Dtests.timezone=Africa/Ouagadougou -Druntime.java=8 -Dtests.rest.blacklist=getting_started/10_monitor_cluster_health/*

org.elasticsearch.xpack.test.rest.XPackRestIT > test {p0=ml/ml_info/Test ml info} FAILED
    java.lang.AssertionError: Failure at [ml/ml_info:21]: field [limits.effective_max_model_memory_limit] was expected to be of type String but is an instanceof [null]
    Expected: an instance of java.lang.String
         but: null
        at __randomizedtesting.SeedInfo.seed([1A5A7D808B0C9B12:920E425A25F0F6EA]:0)
        at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:414)
        at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:391)
        at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
        at java.lang.Thread.run(Thread.java:748)

        Caused by:
        java.lang.AssertionError: field [limits.effective_max_model_memory_limit] was expected to be of type String but is an instanceof [null]
        Expected: an instance of java.lang.String
             but: null
            at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
            at org.junit.Assert.assertThat(Assert.java:956)
            at org.elasticsearch.test.rest.yaml.section.MatchAssertion.doAssert(MatchAssertion.java:63)
            at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:76)
            at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:407)
            ... 37 more

Metadata

Metadata

Assignees

Labels

:mlMachine learning>test-failureTriaged test failures from CITeam:MLMeta label for the ML team

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions