Skip to content

[BUG] [class_not_found_exception] during rolling upgrade on security enabled cluster #1259

@skkosuri-amzn

Description

@skkosuri-amzn

Describe the bug

curl https://localhost:9202/_cluster/settings?pretty -u admin:admin --insecure
{
"error" : {
"root_cause" : [
{
"type" : "class_not_found_exception",
"reason" : "class_not_found_exception: org.opensearch.security.user.User"
}
],
"type" : "exception",
"reason" : "java.lang.ClassNotFoundException: org.opensearch.security.user.User",
"caused_by" : {
"type" : "class_not_found_exception",
"reason" : "class_not_found_exception: org.opensearch.security.user.User"
}
},
"status" : 500
}

To Reproduce
Steps to reproduce the behavior:

  1. Setup 3 node ODFE cluster (yml changes 9300, 9301, 9302)
  2. Ingest data to an index.
  3. Setup few cluster settings.

curl -X PUT -H 'Content-Type:application/json' https://localhost:9200/_cluster/settings?pretty -u admin:admin --insecure -d'
{
"persistent" : {
"indices.recovery.max_bytes_per_sec" : "50mb"
}
}
'
curl -X PUT -H 'Content-Type:application/json' https://localhost:9200/_cluster/settings?pretty -u admin:admin --insecure -d'
{
"transient" : {
"indices.recovery.max_bytes_per_sec" : "10mb"
}
}
'

  1. Shutdown one node and replace with OpenSearch node.
    Copy the data folder and yml config to OpenSearch
  2. Do a GET /_cluster/settings on the OpenSearch node.
  3. The above class_not_found_exception is shown.

Host/Environment (please complete the following information):
Linux AL2

Additional context

  1. After OpenSearch node is added, I see that OpenSearch-security shard is not initialized on OpenSearch node. That could be the main reason for this error.

curl https://localhost:9200/_cluster/health?pretty -u admin:admin --insecure
{
"cluster_name" : "elasticsearch",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 8,
"active_shards" : 16,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 94.11764705882352
}

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions