Add ILM policy for new stack monitoring data streams by jbaiera · Pull Request #82498 · elastic/elasticsearch

jbaiera · 2022-01-12T21:56:00Z

Adds a templated ILM policy with a 3 day or 50 gb rollover and a 3 day retention period by default. The retention period checks the value set in xpack.monitoring.history.duration and uses that for the retention period if it is present. Depending on which default it uses, the _meta field is updated with the reasoning for picking the default retention value:

The defaults

{
    ".monitoring-8-ilm-policy": {
        "version": 1,
        "modified_date": "2022-01-12T21:53:55.278Z",
        "policy": {
            "phases": {
                "warm": {
                    "min_age": "0ms",
                    "actions": {
                        "forcemerge": {
                            "max_num_segments": 1
                        }
                    }
                },
                "hot": {
                    "min_age": "0ms",
                    "actions": {
                        "rollover": {
                            "max_primary_shard_size": "50gb",
                            "max_age": "3d"
                        }
                    }
                },
                "delete": {
                    "min_age": "3d",
                    "actions": {
                        "delete": {
                            "delete_searchable_snapshot": true
                        }
                    }
                }
            },
            "_meta": {
                "defaults": {
                    "delete_min_age": "Using value of [3d] based on the monitoring plugin default"
                },
                "managed": true,
                "description": "Index lifecycle policy generated for [monitoring-*-8] data streams"
            }
        },
        "in_use_by": {
            "indices": [],
            "data_streams": [],
            "composable_templates": [
                ".monitoring-beats-mb",
                ".monitoring-kibana-mb",
                ".monitoring-es-mb",
                ".monitoring-logstash-mb"
            ]
        }
    }
}

Setting xpack.monitoring.history.duration to the value 4d at start up

{
    ".monitoring-8-ilm-policy": {
        "version": 1,
        "modified_date": "2022-01-12T20:42:01.835Z",
        "policy": {
            "phases": {
                "warm": {
                    "min_age": "0ms",
                    "actions": {
                        "forcemerge": {
                            "max_num_segments": 1
                        }
                    }
                },
                "hot": {
                    "min_age": "0ms",
                    "actions": {
                        "rollover": {
                            "max_primary_shard_size": "50gb",
                            "max_age": "3d"
                        }
                    }
                },
                "delete": {
                    "min_age": "4d",
                    "actions": {
                        "delete": {
                            "delete_searchable_snapshot": true
                        }
                    }
                }
            },
            "_meta": {
                "defaults": {
                    "delete_min_age": "Using value of [4d] based on the value of the [xpack.monitoring.history.duration] setting at node startup"
                },
                "managed": true,
                "description": "Index lifecycle policy generated for [monitoring-*-8] data streams"
            }
        },
        "in_use_by": {
            "indices": [],
            "data_streams": [],
            "composable_templates": [
                ".monitoring-beats-mb",
                ".monitoring-kibana-mb",
                ".monitoring-es-mb",
                ".monitoring-logstash-mb"
            ]
        }
    }
}

Also included is a small enhancement to allow for template variable replacement in ILM policy resource files.

elasticmachine · 2022-01-12T21:56:04Z

Pinging @elastic/es-data-management (Team:Data Management)

jbaiera · 2022-01-12T21:56:43Z

relates to #81839

dakrone

Generally looks good, I left a couple of comments.

Can you also add a test for this similar to what we do for StackTemplateRegistryTests? That way we can test the duration copying from the setting into the ILM policy and ensure it works correctly.

...k/plugin/core/src/main/java/org/elasticsearch/xpack/core/template/LifecyclePolicyConfig.java

x-pack/plugin/core/src/main/resources/monitoring-mb-ilm-policy.json

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

jbaiera · 2022-01-14T19:58:19Z

@elasticmachine update branch

dakrone

LGTM, I left two suuuper minor things

...toring/src/test/java/org/elasticsearch/xpack/monitoring/MonitoringTemplateRegistryTests.java

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

matschaffer · 2022-01-17T04:32:39Z

Reviewing this made me notice we're going to need a monitoring-entsearch as well. Looks like the mapping isn't there atm anyway so we'll just have to remember to include ILM when we add it (probably as part of elastic/kibana#121975)

…policy.

matschaffer

I gave this a try locally and LGTM

jbaiera · 2022-01-18T18:02:20Z

@elasticmachine update branch

Adds a templated ILM policy with a 3 day or 50 gb rollover and a 3 day retention period by default. The retention period checks the value set in xpack.monitoring.history.duration and uses that for the retention period if it is present. Depending on which default it uses, the _meta field is updated with the reasoning for picking the default retention value. Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> # Conflicts: # x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/template/LifecyclePolicyConfig.java # x-pack/plugin/ml/qa/native-multi-node-tests/src/javaRestTest/java/org/elasticsearch/xpack/ml/integration/MlNativeIntegTestCase.java

* Upgrade documentation for xpack.monitoring.history.duration * A few updates now that #82498 has been opened with a concrete policy. * Grammar fix * Update docs/reference/migration/migrate_8_0/cluster-node-setting-changes.asciidoc Co-authored-by: James Baiera <james.baiera@gmail.com> Co-authored-by: James Baiera <james.baiera@gmail.com>

…82295) * Upgrade documentation for xpack.monitoring.history.duration * A few updates now that elastic#82498 has been opened with a concrete policy. * Grammar fix * Update docs/reference/migration/migrate_8_0/cluster-node-setting-changes.asciidoc Co-authored-by: James Baiera <james.baiera@gmail.com> Co-authored-by: James Baiera <james.baiera@gmail.com>

…82920) * Upgrade documentation for xpack.monitoring.history.duration * A few updates now that #82498 has been opened with a concrete policy. * Grammar fix * Update docs/reference/migration/migrate_8_0/cluster-node-setting-changes.asciidoc Co-authored-by: James Baiera <james.baiera@gmail.com> Co-authored-by: James Baiera <james.baiera@gmail.com> Co-authored-by: James Baiera <james.baiera@gmail.com>

…82892) Adds a templated ILM policy with a 3 day or 50 gb rollover and a 3 day retention period by default. The retention period checks the value set in xpack.monitoring.history.duration and uses that for the retention period if it is present. Depending on which default it uses, the _meta field is updated with the reasoning for picking the default retention value. Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com> Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>

jbaiera added 2 commits January 12, 2022 15:45

Add templated ILM policy for monitoring data streams

0611754

Appease precommit

358fc6f

jbaiera added >enhancement :Core/Infra/Monitoring DEPRECATED, DO NOT USE v8.0.0 v8.1.0 labels Jan 12, 2022

elasticmachine added the Team:Data Management (obsolete) DO NOT USE. This team no longer exists. label Jan 12, 2022

dakrone requested changes Jan 12, 2022

View reviewed changes

...k/plugin/core/src/main/java/org/elasticsearch/xpack/core/template/LifecyclePolicyConfig.java Outdated Show resolved Hide resolved

x-pack/plugin/core/src/main/resources/monitoring-mb-ilm-policy.json Outdated Show resolved Hide resolved

jbaiera and others added 2 commits January 13, 2022 11:56

Simplify constructor

6360ab4

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

Add monitoring template registry tests

d472b64

jbaiera requested a review from dakrone January 14, 2022 19:58

Merge branch 'master' into monitoring-ilm-policy

053caa2

dakrone approved these changes Jan 14, 2022

View reviewed changes

...toring/src/test/java/org/elasticsearch/xpack/monitoring/MonitoringTemplateRegistryTests.java Outdated Show resolved Hide resolved

...toring/src/test/java/org/elasticsearch/xpack/monitoring/MonitoringTemplateRegistryTests.java Outdated Show resolved Hide resolved

jbaiera and others added 4 commits January 14, 2022 15:29

Update error message

a7f53a5

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

Update error message part 2

731e8f4

Co-authored-by: Lee Hinman <dakrone@users.noreply.github.com>

Update error message part 3

e465b00

Appease the style check monster

1826358

matschaffer added a commit to matschaffer/elasticsearch that referenced this pull request Jan 17, 2022

A few updates now that elastic#82498 has been opened with a concrete …

15f05ad

…policy.

matschaffer approved these changes Jan 17, 2022

View reviewed changes

elasticmachine and others added 4 commits January 18, 2022 11:02

Merge branch 'master' into monitoring-ilm-policy

66626fd

Ignore monitoring ilm policy during test cluster cleanup

d49b71e

see spotless run

a2c055a

Add ILM deps to downstream tests

1f0163b

matschaffer mentioned this pull request Jan 20, 2022

Upgrade documentation for xpack.monitoring.history.duration #82295

Merged

jbaiera merged commit 98dc0eb into elastic:master Jan 20, 2022

jbaiera deleted the monitoring-ilm-policy branch January 20, 2022 19:52

jbaiera added the backport pending label Jan 20, 2022

jbaiera mentioned this pull request Jan 20, 2022

[8.0] Add ILM policy for new stack monitoring data streams (#82498) #82892

Merged

jbaiera removed the backport pending label Jan 24, 2022

jakelandis mentioned this pull request Feb 9, 2022

Use ILM to manage internal indices #38470

Closed

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ILM policy for new stack monitoring data streams#82498

Add ILM policy for new stack monitoring data streams#82498
jbaiera merged 13 commits intoelastic:masterfrom
jbaiera:monitoring-ilm-policy

jbaiera commented Jan 12, 2022

Uh oh!

elasticmachine commented Jan 12, 2022

Uh oh!

jbaiera commented Jan 12, 2022

Uh oh!

dakrone left a comment

Uh oh!

Uh oh!

Uh oh!

jbaiera commented Jan 14, 2022

Uh oh!

dakrone left a comment

Uh oh!

Uh oh!

Uh oh!

matschaffer commented Jan 17, 2022

Uh oh!

matschaffer left a comment

Uh oh!

jbaiera commented Jan 18, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

jbaiera commented Jan 12, 2022

Uh oh!

elasticmachine commented Jan 12, 2022

Uh oh!

jbaiera commented Jan 12, 2022

Uh oh!

dakrone left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jbaiera commented Jan 14, 2022

Uh oh!

dakrone left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

matschaffer commented Jan 17, 2022

Uh oh!

matschaffer left a comment

Choose a reason for hiding this comment

Uh oh!

jbaiera commented Jan 18, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants