Skip to content

Remove unnecessary shuffle in unassigned shards allocation.#65172

Merged
henningandersen merged 8 commits intoelastic:masterfrom
TencentCloudES:remove_shuffle
Nov 23, 2021
Merged

Remove unnecessary shuffle in unassigned shards allocation.#65172
henningandersen merged 8 commits intoelastic:masterfrom
TencentCloudES:remove_shuffle

Conversation

@howardhuanghua
Copy link
Copy Markdown
Contributor

@howardhuanghua howardhuanghua commented Nov 18, 2020

We always sort unassigned shards before allocating them. Here are the sorting logics:

For exsiting shards allocation:

elasticsearch/server/src/main/java/org/elasticsearch/cluster/routing/allocation/AllocationService.java

Line 443 in caf9cd2

allocation.routingNodes().unassigned().sort(PriorityComparator.getAllocationComparator(allocation)); // sort for priority ordering

For new created shards or peer recovery shards allocation:

elasticsearch/server/src/main/java/org/elasticsearch/cluster/routing/allocation/allocator/BalancedShardsAllocator.java

Line 797 in caf9cd2

ArrayUtil.timSort(primary, comparator);

So, the shuffle should be unnecessary before allocating them. This PR removes related unassigned shards shuffle operations.

@matriv matriv added the :Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) label Nov 25, 2020
@elasticmachine elasticmachine added the Team:Distributed Meta label for distributed team. label Nov 25, 2020
@elasticmachine
Copy link
Copy Markdown
Collaborator

elasticmachine commented Nov 25, 2020

Pinging @elastic/es-distributed (Team:Distributed)

@howardhuanghua
Copy link
Copy Markdown
Contributor Author

howardhuanghua commented Nov 25, 2020 •
edited
Loading

Hi @matriv, thanks for checking PRs, would you please also help to find appropriate members to review this #64753 bug fix? I think it should be omitted.

@matriv
Copy link
Copy Markdown
Contributor

matriv commented Nov 26, 2020

@howardhuanghua

Both PRs are labeled/routed to the teams, so they'll get to them when they have the capacity to do so.

@howardhuanghua
Copy link
Copy Markdown
Contributor Author

howardhuanghua commented Nov 26, 2020

@howardhuanghua

Both PRs are labeled/routed to the teams, so they'll get to them when they have the capacity to do so.

Thanks a lot @matriv .

@henningandersen
Copy link
Copy Markdown
Contributor

henningandersen commented Dec 21, 2020

@elasticmachine check this please

@howardhuanghua
Copy link
Copy Markdown
Contributor Author

howardhuanghua commented Nov 18, 2021

@henningandersen Would you please help to check this PR? I have updated the master branch. Thanks.

henningandersen
henningandersen approved these changes Nov 22, 2021
View reviewed changes
Copy link
Copy Markdown
Contributor

@henningandersen henningandersen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@henningandersen
Copy link
Copy Markdown
Contributor

henningandersen commented Nov 22, 2021

@elasticmachine check this please

@henningandersen henningandersen self-assigned this Nov 22, 2021
@henningandersen
Copy link
Copy Markdown
Contributor

henningandersen commented Nov 22, 2021

@elasticmachine check this please

@henningandersen
Copy link
Copy Markdown
Contributor

henningandersen commented Nov 22, 2021

@elasticmachine check this please

@howardhuanghua
Copy link
Copy Markdown
Contributor Author

howardhuanghua commented Nov 22, 2021

Merged latest master branch. Thanks @henningandersen .

@henningandersen
Copy link
Copy Markdown
Contributor

henningandersen commented Nov 22, 2021

@elasticmachine test this please

@henningandersen
Copy link
Copy Markdown
Contributor

henningandersen commented Nov 23, 2021

@elasticmachine update branch

@howardhuanghua
Copy link
Copy Markdown
Contributor Author

howardhuanghua commented Nov 23, 2021

Branch has been updated.

@henningandersen
Copy link
Copy Markdown
Contributor

henningandersen commented Nov 23, 2021

@elasticmachine test this please

@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

elasticsearchmachine commented Nov 23, 2021

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

@henningandersen
Copy link
Copy Markdown
Contributor

henningandersen commented Nov 23, 2021

@elasticmachine run elasticsearch-ci/part-2

@henningandersen henningandersen merged commit 2ce9cd3 into elastic:master Nov 23, 2021
@henningandersen
Copy link
Copy Markdown
Contributor

henningandersen commented Nov 23, 2021

Thanks for your contribution @howardhuanghua and sorry for the delay on progressing this.

@howardhuanghua
Copy link
Copy Markdown
Contributor Author

howardhuanghua commented Nov 23, 2021

@henningandersen My pleasure, thank you.

weizijun added a commit to weizijun/elasticsearch that referenced this pull request Nov 23, 2021
@weizijun
Merge branch 'upstream/master' into move-tsdb-IT-to-yaml
0f15c86 
* upstream/master:
  Remove obsolete typed legacy index templates (elastic#80937)
  Remove unnecessary shuffle in unassigned shards allocation. (elastic#65172)
  TSDB: Tests for nanosecond timeprecision timestamp just beyond the limit (elastic#80932)

# Conflicts:
#	rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/tsdb/10_settings.yml
weizijun added a commit to weizijun/elasticsearch that referenced this pull request Nov 24, 2021
@weizijun
Merge branch 'upstream/master' into tsdb-required-range-settings
21b60b5 
* upstream/master: (29 commits)
  Fix typo (elastic#80925)
  Increase docker compose timeouts for CI builds
  TSDB: fix error without feature flag (elastic#80945)
  [DOCS] Relocate `index.mapping.dimension_fields.limit` setting docs (elastic#80964)
  Explicit write methods for always-missing values (elastic#80958)
  TSDB: move TimeSeriesModeIT to yaml tests (elastic#80933)
  [ML] Removing temporary debug (elastic#80956)
  Remove unused ConnectTransportException#node (elastic#80944)
  Reinterpret dots in field names as object structure (elastic#79922)
  Remove obsolete typed legacy index templates (elastic#80937)
  Remove unnecessary shuffle in unassigned shards allocation. (elastic#65172)
  TSDB: Tests for nanosecond timeprecision timestamp just beyond the limit (elastic#80932)
  Cleanup SLM History Item .equals (elastic#80938)
  Rework breaking changes for new structure (elastic#80907)
  [DOCS] Fix elasticsearch-reset-password typo (elastic#80919)
  [ML] No need to use parent task client when internal infer delegates (elastic#80905)
  Fix shadowed vars pt6 (elastic#80899)
  add ignore info (elastic#80924)
  Fix several potential circuit breaker leaks in Aggregators (elastic#79676)
  Extract more standard metadata from binary files (elastic#78754)
  ...
weizijun added a commit to weizijun/elasticsearch that referenced this pull request Nov 24, 2021
@weizijun
Merge branch 'upstream/master' into update-filter-improve
8507e30 
* upstream/master: (319 commits)
  Fix typo (elastic#80925)
  Increase docker compose timeouts for CI builds
  TSDB: fix error without feature flag (elastic#80945)
  [DOCS] Relocate `index.mapping.dimension_fields.limit` setting docs (elastic#80964)
  Explicit write methods for always-missing values (elastic#80958)
  TSDB: move TimeSeriesModeIT to yaml tests (elastic#80933)
  [ML] Removing temporary debug (elastic#80956)
  Remove unused ConnectTransportException#node (elastic#80944)
  Reinterpret dots in field names as object structure (elastic#79922)
  Remove obsolete typed legacy index templates (elastic#80937)
  Remove unnecessary shuffle in unassigned shards allocation. (elastic#65172)
  TSDB: Tests for nanosecond timeprecision timestamp just beyond the limit (elastic#80932)
  Cleanup SLM History Item .equals (elastic#80938)
  Rework breaking changes for new structure (elastic#80907)
  [DOCS] Fix elasticsearch-reset-password typo (elastic#80919)
  [ML] No need to use parent task client when internal infer delegates (elastic#80905)
  Fix shadowed vars pt6 (elastic#80899)
  add ignore info (elastic#80924)
  Fix several potential circuit breaker leaks in Aggregators (elastic#79676)
  Extract more standard metadata from binary files (elastic#78754)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@henningandersen henningandersen henningandersen approved these changes

Assignees

@henningandersen henningandersen

Labels

:Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) >non-issue Team:Distributed Meta label for distributed team. v8.1.0

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@howardhuanghua @elasticmachine @matriv @henningandersen @elasticsearchmachine