Autoscaling reactive storage decider#65520
Merged
henningandersen merged 40 commits intoelastic:masterfrom Dec 13, 2020
Merged
Conversation
The reactive storage decider will request additional capacity proportional to the size of shards that are either: * unassigned and unable to be allocated with only reason being storage on a node * shards that cannot remain where they are with only reason being storage and cannot be allocated anywhere else * shards that cannot remain where they are and cannot be allocated on any node and at least one node has storage as the only reason for unable to allocate. The reactive storage decider does not try to look into the future, thus at the time the reactive decider asks to scale up, the cluster is already in a need for more storage.
Extracted DiskUsageIntegTestCase from DiskThresholdDeciderIT to allow other tests to easily test functionality relying on disk usage. Relates elastic#65520
henningandersen
added a commit
to henningandersen/elasticsearch
that referenced
this pull request
Nov 26, 2020
Extracted DiskUsageIntegTestCase from DiskThresholdDeciderIT to allow other tests to easily test functionality relying on disk usage. Relates elastic#65520
henningandersen
commented
Nov 27, 2020
...erTest/java/org/elasticsearch/cluster/routing/allocation/decider/DiskThresholdDeciderIT.java
Outdated
Show resolved
Hide resolved
henningandersen
commented
Nov 27, 2020
test/framework/src/main/java/org/elasticsearch/cluster/DiskUsageIntegTestCase.java
Show resolved
Hide resolved
This was referenced Nov 28, 2020
henningandersen
added a commit
to henningandersen/elasticsearch
that referenced
this pull request
Dec 4, 2020
Extracted DiskUsageIntegTestCase from DiskThresholdDeciderIT to allow other tests to easily test functionality relying on disk usage. Relates elastic#65520
…orage_autoscaler_pr_final
henningandersen
added a commit
that referenced
this pull request
Dec 4, 2020
Contributor
Author
|
@elasticmachine update branch |
…orage_autoscaler_pr_final
….com:henningandersen/elasticsearch into enhance_reactive_storage_autoscaler_pr_final
Contributor
Author
|
@elasticmachine update branch |
…orage_autoscaler_pr_final
jasontedor
approved these changes
Dec 13, 2020
Member
jasontedor
left a comment
There was a problem hiding this comment.
This is really great work. I left a few minor comments, but no need for another round.
|
|
||
| ClusterInfo info(); | ||
|
|
||
| SnapshotShardSizeInfo snapshotShardSizeInfo(); |
Member
There was a problem hiding this comment.
Could you add Javadocs to these new methods, and also state?
| DataTier.DATA_CONTENT_NODE_ROLE, | ||
| DataTier.DATA_HOT_NODE_ROLE, | ||
| DataTier.DATA_WARM_NODE_ROLE, | ||
| DataTier.DATA_COLD_NODE_ROLE |
Member
There was a problem hiding this comment.
We'll probably want a test that collects all the roles that return true for DiscoveryNodeRole#canContainData and ensure they are returned in this list. I'm thinking of when we add a role for frozen, ensuring that this list is maintained properly.
| } | ||
|
|
||
| static boolean isDiskOnlyNoDecision(Decision decision) { | ||
| // we consider throttling==yes, throttling should be temporary. |
| */ | ||
| private boolean cannotAllocateDueToStorage(ShardRouting shard, RoutingAllocation allocation) { | ||
| assert allocation.debugDecision() == false; | ||
| allocation.debugDecision(true); |
Member
There was a problem hiding this comment.
Can you leave a comment explaining why we need to enable allocation debugging here?
...src/main/java/org/elasticsearch/xpack/autoscaling/storage/ReactiveStorageDeciderService.java
Show resolved
Hide resolved
...src/main/java/org/elasticsearch/xpack/autoscaling/storage/ReactiveStorageDeciderService.java
Outdated
Show resolved
Hide resolved
| assert assigned >= 0; | ||
| assert unassigned >= 0; | ||
| assert maxShard >= 0; | ||
| String message = unassigned > 0 || assigned > 0 ? "not enough storage available, needs " + (unassigned + assigned) : "storage ok"; |
Member
There was a problem hiding this comment.
I wonder if this should be human readable bytes? So new ByteSizeValue(unassigned + assigned).toString()?
Member
There was a problem hiding this comment.
And if not, bytes should be appended to the message.
jasontedor
added a commit
to jasontedor/elasticsearch
that referenced
this pull request
Dec 13, 2020
* elastic/master: Autoscaling reactive storage decider (elastic#65520) Fix TranslogTests#testStats (elastic#66227)
henningandersen
added a commit
to henningandersen/elasticsearch
that referenced
this pull request
Dec 13, 2020
The reactive storage decider will request additional capacity proportional to the size of shards that are either: * unassigned and unable to be allocated with only reason being storage on a node * shards that cannot remain where they are with only reason being storage and cannot be allocated anywhere else * shards that cannot remain where they are and cannot be allocated on any node and at least one node has storage as the only reason for unable to allocate. The reactive storage decider does not try to look into the future, thus at the time the reactive decider asks to scale up, the cluster is already in a need for more storage.
henningandersen
added a commit
that referenced
this pull request
Dec 14, 2020
The reactive storage decider will request additional capacity proportional to the size of shards that are either: * unassigned and unable to be allocated with only reason being storage on a node * shards that cannot remain where they are with only reason being storage and cannot be allocated anywhere else * shards that cannot remain where they are and cannot be allocated on any node and at least one node has storage as the only reason for unable to allocate. The reactive storage decider does not try to look into the future, thus at the time the reactive decider asks to scale up, the cluster is already in a need for more storage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The reactive storage decider will request additional capacity
proportional to the size of shards that are either:
on a node
storage and cannot be allocated anywhere else
on any node and at least one node has storage as the only reason for
unable to allocate.
The reactive storage decider does not try to look into the future, thus
at the time the reactive decider asks to scale up, the cluster is
already in a need for more storage.