[CI] IndexAuditTrail fails to start - cannot read security-version #37062
Description
On 6.6:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.6+periodic/80/console
Seems to have failed twice (previous but the earlier failure may have been related to the underlying hardware.
Doesn't reproduce for me.
As a CI failure it seems pretty rare, but I wonder if there's some sort of timing/concurrency issue that could be an actual production problem.
Log:
00:03:38 | [2019-01-02T00:01:44,802][ERROR][o.e.x.s.a.i.IndexAuditTrail] [upgraded-node-2] failed to start index audit trail services
00:03:38 | java.lang.IllegalStateException: Cannot read security-version string in index .security_audit_log-2019.01.02
00:03:38 | at org.elasticsearch.xpack.security.audit.index.IndexAuditTrail.updateCurrentIndexMappingsIfNecessary(IndexAuditTrail.java:404) ~[x-pack-security-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
00:03:38 | at org.elasticsearch.xpack.security.audit.index.IndexAuditTrail.start(IndexAuditTrail.java:373) ~[x-pack-security-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
00:03:38 | at org.elasticsearch.xpack.security.audit.index.IndexAuditTrail$2.doRun(IndexAuditTrail.java:235) [x-pack-security-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
00:03:38 | at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
00:03:38 | at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
00:03:38 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
00:03:38 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
00:03:38 | at java.lang.Thread.run(Thread.java:834) [?:?]
00:03:38 | [2019-01-02T00:01:44,814][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [upgraded-node-2] fatal error in thread [elasticsearch[upgraded-node-2][generic][T#1]], exiting
00:03:38 | java.lang.AssertionError: security lifecycle services startup failed
00:03:38 | at org.elasticsearch.xpack.security.audit.index.IndexAuditTrail$2.onFailure(IndexAuditTrail.java:230) ~[?:?]
00:03:38 | at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:744) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
00:03:38 | at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) ~[elasticsearch-6.6.0-SNAPSHOT.jar:6.6.0-SNAPSHOT]
00:03:38 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
00:03:38 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
00:03:38 | at java.lang.Thread.run(Thread.java:834) [?:?]
00:03:38 |-----------------------------------------
Metadata
Metadata
Assignees
Type
Fields
Give feedbackNo fields configured for issues without a type.