Parent ticket: #101016

Summary

We never really talked about which index settings and ILM policy would be best for all RAC indices by default. What we have in the code is more like a draft and an issue was found with the default policy: #111029

Index settings are minimalistic and defined in the technical component template:

Ideas for the policy

We could use the .siem-signals policy as a base because it was in production for some time:

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "30d"
          }
        }
      }
    }
  }
}

30d and 50gb are the recommended defaults, and max_size is deprecated in favour of max_primary_shard_size:

So it could be

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "30d"
          }
        }
      }
    }
  }
}

Some default policies also include a _meta object with managed: true and a description.

Any other settings that would make sense? Data tiers, priorities, etc?

Ideas for the settings

I was thinking about something like that:

  settings: {
    number_of_shards: 1,
    auto_expand_replicas: '0-1',
    'mapping.total_fields.limit': 10000,
    'sort.field': '@timestamp',
    'sort.order': 'desc',
  },

• auto_expand_replicas: '0-1' can be useful in simple 1-node setups (otherwise indices won’t be healthy); it’s used in event_log plugin for example
• ‘mapping.total_fields.limit’: 10000 is used in Security Solution for .siem-signals indices
• default sorting could be probably useful to make all queries faster