Skip to content

An error occurred while calling the create inference API in the ES cluster upgraded from version 8.9 to 8.15. #112694

Closed
Closed
An error occurred while calling the create inference API in the ES cluster upgraded from version 8.9 to 8.15.#112694
Assignees
ldematte
Labels
:Core/Infra/CoreCore issues without another label>bugTeam:Core/InfraMeta label for core/infra team
@Huaixinww

Description

@Huaixinww
Huaixinww
opened on Sep 10, 2024

Elasticsearch Version

8.9

Installed Plugins

No response

Java Version

bundled

OS Version

x86_64 GNU/Linux

Problem Description

After upgrading our ES cluster from version 8.9 to 8.15, we encountered the following error when calling the create inference API:

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "Failed to store inference endpoint [os-embeddings]"
      }
    ],
    "type": "status_exception",
    "reason": "Failed to store inference endpoint [os-embeddings]",
    "caused_by": {
      "type": "null_pointer_exception",
      "reason": null
    }
  },
  "status": 500
}

We noticed that in the 8.15 cluster, the mappings_versions field in the nodes_versions parameter of GET _cluster/state contains information about relevant system indices.

"nodes_versions": [
    {
      "node_id": "XpcbHlUSQR6Nq3iQ9ZmBHA",
      "transport_version": "8702003",
      "mappings_versions": {
        ".triggered_watches": {
          "version": 1,
          "hash": -502826165
        },
        ".secrets-inference": {
          "version": 1,
          "hash": -1434574148
        },
        ...  # other system index mappings_versions
      }
    },

However, in the cluster upgraded from 8.9, this mappings_versions field is empty.

"nodes_versions": [
    {
      "node_id": "7SHcBQoRSSOBA2KXyz42hQ",
      "transport_version": "8702003",
      "mappings_versions": {}
    },
    {
      "node_id": "KFVdkkGDQ5qx-SoxbK8zxg",
      "transport_version": "8702003",
      "mappings_versions": {}
    },
    {
      "node_id": "RpOqtyQJRBuxHpq78T_8Gw",
      "transport_version": "8702003",
      "mappings_versions": {}
    }
  ]

This causes a NPE when calling the inference API, as shown in the following error trace:

{
  "error": {
    "root_cause": [
      {
        "type": "status_exception",
        "reason": "Failed to store inference endpoint [os-embeddings]",
        "stack_trace": """org.elasticsearch.ElasticsearchStatusException: Failed to store inference endpoint [os-embeddings]
	at org.elasticsearch.xpack.inference.registry.ModelRegistry.lambda$getStoreModelListener$5(ModelRegistry.java:334)
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:249)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
	at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:202)
	at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:196)
	at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:307)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
	at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
	at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:307)
	at org.elasticsearch.action.bulk.BulkOperation.completeBulkOperation(BulkOperation.java:396)
	at org.elasticsearch.action.bulk.BulkOperation.redirectFailuresOrCompleteBulkOperation(BulkOperation.java:391)
	at org.elasticsearch.action.bulk.BulkOperation.executeBulkRequestsByShard(BulkOperation.java:356)
	at org.elasticsearch.action.bulk.BulkOperation.doRun(BulkOperation.java:169)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:539)
	at org.elasticsearch.action.bulk.TransportBulkAction$1.doRun(TransportBulkAction.java:324)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.NullPointerException
	at java.util.Objects.requireNonNull(Objects.java:233)
	at org.elasticsearch.indices.SystemIndexDescriptor.getDescriptorCompatibleWith(SystemIndexDescriptor.java:622)
	at org.elasticsearch.action.admin.indices.create.AutoCreateAction$TransportAction$CreateIndexTask.execute(AutoCreateAction.java:307)
	at org.elasticsearch.action.admin.indices.create.AutoCreateAction$TransportAction.lambda$new$0(AutoCreateAction.java:120)
	at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1078)
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1041)
	at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:240)
	at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1694)
	at org.elasticsearch.action.ActionListener.run(ActionListener.java:444)
	at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1691)
	at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1286)
	at org.elasticsearch.action.ActionListener.run(ActionListener.java:444)
	at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1265)
	... 5 more
"""
      }
    ],
    "type": "status_exception",
    "reason": "Failed to store inference endpoint [os-embeddings]",
    "caused_by": {
      "type": "null_pointer_exception",
      "reason": null,
      "stack_trace": """java.lang.NullPointerException
	at java.util.Objects.requireNonNull(Objects.java:233)
	at org.elasticsearch.indices.SystemIndexDescriptor.getDescriptorCompatibleWith(SystemIndexDescriptor.java:622)
	at org.elasticsearch.action.admin.indices.create.AutoCreateAction$TransportAction$CreateIndexTask.execute(AutoCreateAction.java:307)
	at org.elasticsearch.action.admin.indices.create.AutoCreateAction$TransportAction.lambda$new$0(AutoCreateAction.java:120)
	at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1078)
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1041)
	at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:240)
	at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1694)
	at org.elasticsearch.action.ActionListener.run(ActionListener.java:444)
	at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1691)
	at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1286)
	at org.elasticsearch.action.ActionListener.run(ActionListener.java:444)
	at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1265)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.lang.Thread.run(Thread.java:1583)
"""
    },
    "stack_trace": """org.elasticsearch.ElasticsearchStatusException: Failed to store inference endpoint [os-embeddings]
	at org.elasticsearch.xpack.inference.registry.ModelRegistry.lambda$getStoreModelListener$5(ModelRegistry.java:334)
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:249)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
	at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:202)
	at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:196)
	at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:307)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32)
	at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95)
	at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:307)
	at org.elasticsearch.action.bulk.BulkOperation.completeBulkOperation(BulkOperation.java:396)
	at org.elasticsearch.action.bulk.BulkOperation.redirectFailuresOrCompleteBulkOperation(BulkOperation.java:391)
	at org.elasticsearch.action.bulk.BulkOperation.executeBulkRequestsByShard(BulkOperation.java:356)
	at org.elasticsearch.action.bulk.BulkOperation.doRun(BulkOperation.java:169)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:539)
	at org.elasticsearch.action.bulk.TransportBulkAction$1.doRun(TransportBulkAction.java:324)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.NullPointerException
	at java.util.Objects.requireNonNull(Objects.java:233)
	at org.elasticsearch.indices.SystemIndexDescriptor.getDescriptorCompatibleWith(SystemIndexDescriptor.java:622)
	at org.elasticsearch.action.admin.indices.create.AutoCreateAction$TransportAction$CreateIndexTask.execute(AutoCreateAction.java:307)
	at org.elasticsearch.action.admin.indices.create.AutoCreateAction$TransportAction.lambda$new$0(AutoCreateAction.java:120)
	at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1078)
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1041)
	at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:240)
	at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1694)
	at org.elasticsearch.action.ActionListener.run(ActionListener.java:444)
	at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1691)
	at org.elasticsearch.cluster.service.MasterService$5.lambda$doRun$0(MasterService.java:1286)
	at org.elasticsearch.action.ActionListener.run(ActionListener.java:444)
	at org.elasticsearch.cluster.service.MasterService$5.doRun(MasterService.java:1265)
	... 5 more
"""
  },
  "status": 500
}

Steps to Reproduce

  1. An ES cluster upgraded from version 8.9 to 8.15.
  2. Calling the create inference API interface.

Logs (if relevant)

No response

Metadata

Metadata

Assignees

Labels

:Core/Infra/CoreCore issues without another label>bugTeam:Core/InfraMeta label for core/infra team

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions