Skip to content

XPack Usage API can return 503 due to missing primary #45250

@Mpdreamz

Description

@Mpdreamz

GET /_xpack/usage?pretty=true&error_trace=true

{
  "error" : {
    "root_cause" : [
      {
        "type" : "unavailable_shards_exception",
        "reason" : "at least one primary shard for the security index is unavailable",
        "stack_trace" : "UnavailableShardsException[at least one primary shard for the security index is unavailable]\r\n\tat org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:146)\r\n\tat org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getUserCount(NativeUsersStore.java:171)\r\n\tat org.elasticsearch.xpack.security.authc.esnative.NativeRealm.lambda$usageStats$1(NativeRealm.java:53)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$usageStats$5(CachingUsernamePasswordRealm.java:203)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\r\n\tat org.elasticsearch.xpack.core.security.authc.Realm.usageStats(Realm.java:137)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.usageStats(CachingUsernamePasswordRealm.java:201)\r\n\tat org.elasticsearch.xpack.security.authc.esnative.NativeRealm.usageStats(NativeRealm.java:52)\r\n\tat org.elasticsearch.xpack.security.authc.Realms.usageStats(Realms.java:243)\r\n\tat org.elasticsearch.xpack.security.SecurityFeatureSet.usage(SecurityFeatureSet.java:144)\r\n\tat org.elasticsearch.xpack.core.action.TransportXPackUsageAction.lambda$masterOperation$0(TransportXPackUsageAction.java:70)\r\n\tat org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102)\r\n\tat org.elasticsearch.xpack.core.action.TransportXPackUsageAction.masterOperation(TransportXPackUsageAction.java:93)\r\n\tat org.elasticsearch.xpack.core.action.TransportXPackUsageAction.masterOperation(TransportXPackUsageAction.java:31)\r\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:127)\r\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:208)\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\r\n\tat java.base/java.lang.Thread.run(Thread.java:835)\r\n"
      }
    ],
    "type" : "unavailable_shards_exception",
    "reason" : "at least one primary shard for the security index is unavailable",
    "stack_trace" : "UnavailableShardsException[at least one primary shard for the security index is unavailable]\r\n\tat org.elasticsearch.xpack.security.support.SecurityIndexManager.getUnavailableReason(SecurityIndexManager.java:146)\r\n\tat org.elasticsearch.xpack.security.authc.esnative.NativeUsersStore.getUserCount(NativeUsersStore.java:171)\r\n\tat org.elasticsearch.xpack.security.authc.esnative.NativeRealm.lambda$usageStats$1(NativeRealm.java:53)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.lambda$usageStats$5(CachingUsernamePasswordRealm.java:203)\r\n\tat org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61)\r\n\tat org.elasticsearch.xpack.core.security.authc.Realm.usageStats(Realm.java:137)\r\n\tat org.elasticsearch.xpack.security.authc.support.CachingUsernamePasswordRealm.usageStats(CachingUsernamePasswordRealm.java:201)\r\n\tat org.elasticsearch.xpack.security.authc.esnative.NativeRealm.usageStats(NativeRealm.java:52)\r\n\tat org.elasticsearch.xpack.security.authc.Realms.usageStats(Realms.java:243)\r\n\tat org.elasticsearch.xpack.security.SecurityFeatureSet.usage(SecurityFeatureSet.java:144)\r\n\tat org.elasticsearch.xpack.core.action.TransportXPackUsageAction.lambda$masterOperation$0(TransportXPackUsageAction.java:70)\r\n\tat org.elasticsearch.xpack.core.common.IteratingActionListener.run(IteratingActionListener.java:102)\r\n\tat org.elasticsearch.xpack.core.action.TransportXPackUsageAction.masterOperation(TransportXPackUsageAction.java:93)\r\n\tat org.elasticsearch.xpack.core.action.TransportXPackUsageAction.masterOperation(TransportXPackUsageAction.java:31)\r\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:127)\r\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:208)\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751)\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\r\n\tat java.base/java.lang.Thread.run(Thread.java:835)\r\n"
  },
  "status" : 503
}

Hitting this intermittently on our CI, do not have an exact reproduce playlist.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions