Add the frozen tier node role and ILM phase#68605
Merged
dakrone merged 10 commits intoelastic:masterfrom Feb 5, 2021
Merged
Conversation
This commit adds the `data_frozen` node role as part of the formalization of data tiers. It also adds the `"frozen"` phase to ILM, currently allowing the same actions as the existing cold phase. The frozen phase is intended to be used for data even less frequently searched than the cold phase, and will eventually be loosely tied to data using partial searchable snapshots (as oppposed to full searchable snapshots in the cold phase). Relates to elastic#60848
Collaborator
|
Pinging @elastic/es-core-features (Team:Core/Features) |
dakrone
commented
Feb 5, 2021
jasontedor
reviewed
Feb 5, 2021
Member
jasontedor
left a comment
There was a problem hiding this comment.
LGTM. The only question is around the descriptions of the querying levels in the tiers. I offered a suggestion of cold -> infrequently, frozen -> rarely, but I am open to other suggestions. The seldom, more seldom reads oddly to me.
| if ((dataRoles.size() > 1 || dataRoles.get(0).roleName().equals("data_cold") == false) || | ||
| roles.contains(DiscoveryNodeRole.MASTER_ROLE)) { | ||
| // if the node is not a dedicated cold node, use the default | ||
| if (dataRoles.stream().allMatch(dn -> dn.roleName().equals("data_cold") || dn.roleName().equals("data_frozen")) == false) { |
jasontedor
reviewed
Feb 5, 2021
jasontedor
previously approved these changes
Feb 5, 2021
Member
jasontedor
left a comment
There was a problem hiding this comment.
I left one more comment about a code comment, but I don't think we need another round.
We discovered that we need to separate the tier preferences for the cache index and mounted indices, so we'll take another review round here.
A full copy mount prefers cold -> warm -> hot A partial mount prefers frozen -> cold -> warm -> hot
dakrone
added a commit
to dakrone/elasticsearch
that referenced
this pull request
Feb 5, 2021
This commit adds the `data_frozen` node role as part of the formalization of data tiers. It also adds the `"frozen"` phase to ILM, currently allowing the same actions as the existing cold phase. The frozen phase is intended to be used for data even less frequently searched than the cold phase, and will eventually be loosely tied to data using partial searchable snapshots (as oppposed to full searchable snapshots in the cold phase). Relates to elastic#60848
dakrone
added a commit
that referenced
this pull request
Feb 5, 2021
This commit adds the `data_frozen` node role as part of the formalization of data tiers. It also adds the `"frozen"` phase to ILM, currently allowing the same actions as the existing cold phase. The frozen phase is intended to be used for data even less frequently searched than the cold phase, and will eventually be loosely tied to data using partial searchable snapshots (as oppposed to full searchable snapshots in the cold phase). Relates to #60848
Member
|
I think we missed one place in the docs which is docs/reference/modules/node.asciidoc. Leaving this here as a reminder for one of us to pick this up on Monday. |
Member
dakrone
added a commit
to dakrone/elasticsearch
that referenced
this pull request
Feb 8, 2021
This commit adds support for the recently introduced partial searchable snapshot (elastic#68509) to ILM. Searchable snapshot ILM actions may now be specified with a `storage` option, specifying either `full_copy` or `shared_cache` (similar to the "mount" API) to mount either a full or partial searchable snapshot: ```json PUT _ilm/policy/my_policy { "policy": { "phases": { "cold": { "actions": { "searchable_snapshot" : { "snapshot_repository" : "backing_repo", "storage": "shared_cache" } } } } } } ``` Internally, If more than one searchable snapshot action is specified (for example, a full searchable snapshot in the "cold" phase and a partial searchable snapshot in the "frozen" phase) ILM will re-use the existing snapshot when doing the second mount since a second snapshot is not required. Currently this is allowed for actions that use the same repository, however, multiple `searchable_snapshot` actions for the same index that use different repositories is not allowed (the ERROR state is entered). We plan to allow this in the future in subsequent work. If the `storage` option is not specified in the `searchable_snapshot` action, the mount type defaults to "shared_cache" in the frozen phase and "full_copy" in all other phases. Relates to elastic#68605
elasticmachine
pushed a commit
to costin/elasticsearch
that referenced
this pull request
Feb 9, 2021
This commit adds support for the recently introduced partial searchable snapshot (elastic#68509) to ILM. Searchable snapshot ILM actions may now be specified with a `storage` option, specifying either `full_copy` or `shared_cache` (similar to the "mount" API) to mount either a full or partial searchable snapshot: ```json PUT _ilm/policy/my_policy { "policy": { "phases": { "cold": { "actions": { "searchable_snapshot" : { "snapshot_repository" : "backing_repo", "storage": "shared_cache" } } } } } } ``` Internally, If more than one searchable snapshot action is specified (for example, a full searchable snapshot in the "cold" phase and a partial searchable snapshot in the "frozen" phase) ILM will re-use the existing snapshot when doing the second mount since a second snapshot is not required. Currently this is allowed for actions that use the same repository, however, multiple `searchable_snapshot` actions for the same index that use different repositories is not allowed (the ERROR state is entered). We plan to allow this in the future in subsequent work. If the `storage` option is not specified in the `searchable_snapshot` action, the mount type defaults to "shared_cache" in the frozen phase and "full_copy" in all other phases. Relates to elastic#68605
dakrone
added a commit
to dakrone/elasticsearch
that referenced
this pull request
Feb 9, 2021
This commit adds support for the recently introduced partial searchable snapshot (elastic#68509) to ILM. Searchable snapshot ILM actions may now be specified with a `storage` option, specifying either `full_copy` or `shared_cache` (similar to the "mount" API) to mount either a full or partial searchable snapshot: ```json PUT _ilm/policy/my_policy { "policy": { "phases": { "cold": { "actions": { "searchable_snapshot" : { "snapshot_repository" : "backing_repo", "storage": "shared_cache" } } } } } } ``` Internally, If more than one searchable snapshot action is specified (for example, a full searchable snapshot in the "cold" phase and a partial searchable snapshot in the "frozen" phase) ILM will re-use the existing snapshot when doing the second mount since a second snapshot is not required. Currently this is allowed for actions that use the same repository, however, multiple `searchable_snapshot` actions for the same index that use different repositories is not allowed (the ERROR state is entered). We plan to allow this in the future in subsequent work. If the `storage` option is not specified in the `searchable_snapshot` action, the mount type defaults to "shared_cache" in the frozen phase and "full_copy" in all other phases. Relates to elastic#68605
dakrone
added a commit
that referenced
this pull request
Feb 9, 2021
…68762) This commit adds support for the recently introduced partial searchable snapshot (#68509) to ILM. Searchable snapshot ILM actions may now be specified with a `storage` option, specifying either `full_copy` or `shared_cache` (similar to the "mount" API) to mount either a full or partial searchable snapshot: `json PUT _ilm/policy/my_policy { "policy": { "phases": { "cold": { "actions": { "searchable_snapshot" : { "snapshot_repository" : "backing_repo", "storage": "shared_cache" } } } } } } ` Internally, If more than one searchable snapshot action is specified (for example, a full searchable snapshot in the "cold" phase and a partial searchable snapshot in the "frozen" phase) ILM will re-use the existing snapshot when doing the second mount since a second snapshot is not required. Currently this is allowed for actions that use the same repository, however, multiple `searchable_snapshot` actions for the same index that use different repositories is not allowed (the ERROR state is entered). We plan to allow this in the future in subsequent work. If the `storage` option is not specified in the `searchable_snapshot` action, the mount type defaults to "shared_cache" in the frozen phase and "full_copy" in all other phases. Relates to #68605
34 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit adds the
data_frozennode role as part of the formalization of data tiers. It alsoadds the
"frozen"phase to ILM, currently allowing the same actions as the existing cold phase.The frozen phase is intended to be used for data even less frequently searched than the cold phase,
and will eventually be loosely tied to data using partial searchable snapshots (as oppposed to full
searchable snapshots in the cold phase).
Relates to #60848