Avoid regular indices in frozen tier by henningandersen · Pull Request #70141 · elastic/elasticsearch

henningandersen · 2021-03-09T11:43:55Z

The frozen tier will be dedicated for partially cached searchable
snapshots. This PR ensures that we do not allow allocating regular
indices (including fully cached searchable snapshots) to the frozen
tier.

The frozen tier will be dedicated for partially cached searchable snapshots. This PR ensures that we do not allow allocating regular indices (including fully cached searchable snapshots) to the frozen tier.

elasticmachine · 2021-03-09T11:43:58Z

Pinging @elastic/es-distributed (Team:Distributed)

elasticmachine · 2021-03-09T11:43:58Z

Pinging @elastic/es-core-features (Team:Core/Features)

A setting validator can declare settings that the validation depends on, but when updating index settings, we eagerly validate the settings before submitting the cluster state update and here we do not know the existing settings. With this commit, we ensure that the pre-validation works with such validators, letting the validator validate syntax (validate(value)) only. Relates elastic#70141.

Blob cache index not on frozen. When cloning frozen have to specify the tier

henningandersen · 2021-03-09T13:18:25Z

server/src/main/java/org/elasticsearch/common/settings/AbstractScopedSettings.java

+        if (validateDependencies) {
+            setting.get(settings);
+        } else {
+            setting.validateWithoutDependencies(settings);
+        }


This change and the associated change in SecureSettings and tests have been split into #70144

These tests are going to be touched by other work, will wait merging this until that is done.

henningandersen

Note to self to remove AwaitsFix before merge. Those tests are fixed by #70158, which I will integrate here before merging.

Now done.

...de/src/javaRestTest/java/org/elasticsearch/xpack/ilm/actions/SearchableSnapshotActionIT.java

…_not_on_frozen_tier

henningandersen · 2021-03-09T18:59:36Z

@elasticmachine run elasticsearch-ci/bwc

…_not_on_frozen_tier

Old 7.12/7.13 clusters will create the .snapshot-blob-cache index with tier preference including frozen, which causes cluster state updates to fail when upgrading. Disabling bwc until this is in 7.12.

ywelsch

I've just left one small comment, otherwise looking good. I did not review the dependent setting validation logic.

ywelsch · 2021-03-10T16:20:21Z

.../src/main/java/org/elasticsearch/xpack/searchablesnapshots/SearchableSnapshotsConstants.java


+    public static boolean isPartialSearchableSnapshotIndex(Map<Setting<?>, Object> indexSettings) {
+        return SNAPSHOT_DIRECTORY_FACTORY_KEY.equals(indexSettings.get(INDEX_STORE_TYPE_SETTING))
+            && (boolean) indexSettings.get(SNAPSHOT_PARTIAL_SETTING);


Could this method throw a NPE if indexSettings does not contain SNAPSHOT_PARTIAL_SETTING?

Yes, it could, but the expectation is that the map contains the two settings. I have added assertions to enforce this.

henningandersen · 2021-03-11T06:38:56Z

Thanks for reviewing, @dakrone and @ywelsch , I integrated the settings changes from #70144 so they no longer appear in this PR. This is ready for another round.

ywelsch

LGTM

…_not_on_frozen_tier

The frozen tier will be dedicated for partially cached searchable snapshots. This PR ensures that we do not allow allocating regular indices (including fully cached searchable snapshots) to the frozen tier.

Reenable bwc tests now that elastic#70141 has been backported.

Reenable bwc tests now that #70141 has been backported.

This commit converts the index metadata of searchable snapshot indices using the `shared_cache` storage type to: - Remove all the `index.routing.allocation.(include|exclude|require)._tier` settings - Sets `index.routing.allocation.include._tier_preference` to `data_frozen` automatically when the index metadata is read This is in preperation to enforcing that the `_tier_preference` setting is always set to `data_frozen` for shared cache SBIs. Relates to elastic#70846, elastic#71013, elastic#70786, elastic#70141

This commit converts the index metadata of searchable snapshot indices using the `shared_cache` storage type to: - Remove all the `index.routing.allocation.(include|exclude|require)._tier` settings - Sets `index.routing.allocation.include._tier_preference` to `data_frozen` automatically when the index metadata is read This is in preperation to enforcing that the `_tier_preference` setting is always set to `data_frozen` for shared cache SBIs. Relates to #70846, #71013, #70786, #70141

This commit converts the index metadata of searchable snapshot indices using the `shared_cache` storage type to: - Remove all the `index.routing.allocation.(include|exclude|require)._tier` settings - Sets `index.routing.allocation.include._tier_preference` to `data_frozen` automatically when the index metadata is read This is in preperation to enforcing that the `_tier_preference` setting is always set to `data_frozen` for shared cache SBIs. Relates to elastic#70846, elastic#71013, elastic#70786, elastic#70141

…#71129) This commit converts the index metadata of searchable snapshot indices using the `shared_cache` storage type to: - Remove all the `index.routing.allocation.(include|exclude|require)._tier` settings - Sets `index.routing.allocation.include._tier_preference` to `data_frozen` automatically when the index metadata is read This is in preperation to enforcing that the `_tier_preference` setting is always set to `data_frozen` for shared cache SBIs. Relates to #70846, #71013, #70786, #70141

Avoid regular indices in frozen tier

da7aec3

The frozen tier will be dedicated for partially cached searchable snapshots. This PR ensures that we do not allow allocating regular indices (including fully cached searchable snapshots) to the frozen tier.

henningandersen added >non-issue :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs :Core/Features/Features v8.0.0 v7.12.0 v7.13.0 labels Mar 9, 2021

elasticmachine added Team:Distributed Meta label for distributed team. Team:Data Management (obsolete) DO NOT USE. This team no longer exists. labels Mar 9, 2021

henningandersen mentioned this pull request Mar 9, 2021

Support dependent validation for index settings #70144

Merged

henningandersen added 2 commits March 9, 2021 13:49

Fixes

0d99c4d

Blob cache index not on frozen. When cloning frozen have to specify the tier

spotless

55d4178

henningandersen commented Mar 9, 2021

View reviewed changes

Disable some tests temporarily

98bf727

These tests are going to be touched by other work, will wait merging this until that is done.

henningandersen commented Mar 9, 2021

View reviewed changes

henningandersen added 4 commits March 9, 2021 15:22

Revert half-baked changes.

fc7503a

fix

8734624

Merge remote-tracking branch 'origin/master' into fix_regular_indices…

7259e89

…_not_on_frozen_tier

smaller fixes

2831e28

henningandersen requested review from dakrone and ywelsch March 9, 2021 16:37

Merge remote-tracking branch 'origin/master' into fix_regular_indices…

4460f20

…_not_on_frozen_tier

henningandersen mentioned this pull request Mar 10, 2021

Fix regular indices not on frozen tier validate value #70205

Closed

henningandersen added 2 commits March 10, 2021 10:50

Merge remote-tracking branch 'origin/master' into fix_regular_indices…

16fde1f

…_not_on_frozen_tier

Disable bwc

af02126

Old 7.12/7.13 clusters will create the .snapshot-blob-cache index with tier preference including frozen, which causes cluster state updates to fail when upgrading. Disabling bwc until this is in 7.12.

ywelsch reviewed Mar 10, 2021

View reviewed changes

henningandersen requested a review from ywelsch March 11, 2021 06:37

ywelsch approved these changes Mar 11, 2021

View reviewed changes

henningandersen added 2 commits March 11, 2021 12:56

Merge remote-tracking branch 'origin/master' into fix_regular_indices…

39506a5

…_not_on_frozen_tier

Merge remote-tracking branch 'origin/master' into fix_regular_indices…

7ca4286

…_not_on_frozen_tier

henningandersen merged commit dad23ed into elastic:master Mar 11, 2021

henningandersen added the backport pending label Mar 11, 2021

henningandersen mentioned this pull request Mar 11, 2021

[7.X] Avoid regular indices in frozen tier (#70141) #70313

Merged

henningandersen mentioned this pull request Mar 11, 2021

[7.12] Avoid regular indices in frozen tier (#70141) (#70313) #70315

Merged

henningandersen added a commit to henningandersen/elasticsearch that referenced this pull request Mar 11, 2021

Reenable bwc tests after backport of elastic#70141

747ea3b

Reenable bwc tests now that elastic#70141 has been backported.

henningandersen mentioned this pull request Mar 11, 2021

Reenable bwc tests after backport of #70141 #70321

Merged

henningandersen added a commit that referenced this pull request Mar 11, 2021

Reenable bwc tests after backport of #70141 (#70321)

dddcddd

Reenable bwc tests now that #70141 has been backported.

henningandersen added a commit that referenced this pull request Mar 11, 2021

Reenable bwc tests after backport of #70141 (#70321)

903945e

Reenable bwc tests now that #70141 has been backported.

henningandersen removed the backport pending label Mar 11, 2021

henningandersen mentioned this pull request Mar 18, 2021

[DOCS] Frozen tier dedicated #70542

Merged

dakrone mentioned this pull request Mar 29, 2021

Auto convert shared_cache SBIs to only use frozen tier #71014

Merged

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Avoid regular indices in frozen tier#70141

Avoid regular indices in frozen tier#70141
henningandersen merged 16 commits intoelastic:masterfrom
henningandersen:fix_regular_indices_not_on_frozen_tier

henningandersen commented Mar 9, 2021

Uh oh!

elasticmachine commented Mar 9, 2021

Uh oh!

elasticmachine commented Mar 9, 2021

Uh oh!

henningandersen Mar 9, 2021

Uh oh!

henningandersen left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

henningandersen commented Mar 9, 2021

Uh oh!

ywelsch left a comment

Uh oh!

ywelsch Mar 10, 2021

Uh oh!

henningandersen Mar 10, 2021

Uh oh!

henningandersen commented Mar 11, 2021

Uh oh!

ywelsch left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

henningandersen commented Mar 9, 2021

Uh oh!

elasticmachine commented Mar 9, 2021

Uh oh!

elasticmachine commented Mar 9, 2021

Uh oh!

henningandersen Mar 9, 2021

Choose a reason for hiding this comment

Uh oh!

henningandersen left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

henningandersen commented Mar 9, 2021

Uh oh!

ywelsch left a comment

Choose a reason for hiding this comment

Uh oh!

ywelsch Mar 10, 2021

Choose a reason for hiding this comment

Uh oh!

henningandersen Mar 10, 2021

Choose a reason for hiding this comment

Uh oh!

henningandersen commented Mar 11, 2021

Uh oh!

ywelsch left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

henningandersen left a comment •

edited

Loading