Skip to content

Document woes between auto-expand-replicas and allocation filtering#30531

Merged
ywelsch merged 2 commits intoelastic:masterfrom
ywelsch:auto-expand-replicas-note
May 14, 2018
Merged

Document woes between auto-expand-replicas and allocation filtering#30531
ywelsch merged 2 commits intoelastic:masterfrom
ywelsch:auto-expand-replicas-note

Conversation

@ywelsch
Copy link
Copy Markdown
Contributor

@ywelsch ywelsch commented May 11, 2018

Relates to #2869 (comment)

@ywelsch ywelsch added >docs General docs changes v7.0.0 :Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) v6.4.0 v6.3.1 labels May 11, 2018
@ywelsch ywelsch requested review from DaveCTurner and inqueue May 11, 2018 08:52
@elasticmachine
Copy link
Copy Markdown
Collaborator

elasticmachine commented May 11, 2018

Pinging @elastic/es-distributed

@ywelsch ywelsch mentioned this pull request May 11, 2018
Closed
DaveCTurner
DaveCTurner approved these changes May 11, 2018
View reviewed changes
Copy link
Copy Markdown
Member

@DaveCTurner DaveCTurner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - I suggested alternative wording too:

  • "available data nodes" might be read as "available to allocate my replicas" which isn't always true.
  • also worth mentioning total_shards_per_node.

docs/reference/index-modules.asciidoc
into account and can lead to a yellow cluster health.
Set to a dash delimited lower and upper bound (e.g. `0-5`) or use `all`
for the upper bound (e.g. `0-all`). Defaults to `false` (i.e. disabled).
for the upper bound (e.g. `0-all`). Defaults to `false` (i.e. disabled).
Copy link
Copy Markdown
Member

@DaveCTurner DaveCTurner May 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about

Automatically choose the number of replicas based on the number of data nodes in the cluster. Set to a dash delimited lower and upper bound (e.g. `0-5`) or use `all` for the upper bound (e.g. `0-all`). Defaults to `false` (i.e. disabled). Note that the number of replicas chosen does not take into account any other allocation rules such as <<shard-allocation-filtering,shard allocation filtering>> or `total_shards_per_node`, and this can lead to the cluster health becoming `YELLOW` if the applicable rules prevent all the replica shards from being allocated.

Copy link
Copy Markdown
Contributor Author

@ywelsch ywelsch May 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I've incorporated your suggestions as well as mentioned shard allocation awareness.

inqueue
inqueue approved these changes May 14, 2018
View reviewed changes
Copy link
Copy Markdown
Member

@inqueue inqueue left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@ywelsch ywelsch merged commit c96f2d7 into elastic:master May 14, 2018
ywelsch added a commit that referenced this pull request May 14, 2018
@ywelsch
Document woes between auto-expand-replicas and allocation filtering (#…
eb3d184 
…30531)

Relates to #2869
ywelsch added a commit that referenced this pull request May 14, 2018
@ywelsch
Document woes between auto-expand-replicas and allocation filtering (#…
3c21393 
…30531)

Relates to #2869
jasontedor added a commit to jasontedor/elasticsearch that referenced this pull request May 14, 2018
@jasontedor
Merge remote-tracking branch 'elastic/master' into one-shard-to-rule-…
6b30d4a 
…them-all

* elastic/master:
  Add missing dependencies on testClasses (elastic#30527)
  [TEST] Mute ML test that needs updating to following ml-cpp changes
  Document woes between auto-expand-replicas and allocation filtering (elastic#30531)
  Moved tokenizers to analysis common module (elastic#30538)
dnhatn added a commit that referenced this pull request May 14, 2018
@dnhatn
Merge branch 'master' into ccr
b971a81 
* master:
  Default to one shard (#30539)
  Unmute IndexUpgradeIT tests
  Forbid expensive query parts in ranking evaluation (#30151)
  Docs: Update HighLevelRestClient migration docs (#30544)
  Clients: Switch to new performRequest (#30543)
  [TEST] Fix typo in MovAvgIT test
  Add missing dependencies on testClasses (#30527)
  [TEST] Mute ML test that needs updating to following ml-cpp changes
  Document woes between auto-expand-replicas and allocation filtering (#30531)
  Moved tokenizers to analysis common module (#30538)
  Adjust copy settings versions
  Mute ShrinkIndexIT suite
  SQL: SYS TABLES ordered according to *DBC specs (#30530)
  Deprecate not copy settings and explicitly disallow (#30404)
  [ML] Improve state persistence log message
  Build: Add mavenPlugin cluster configuration method (#30541)
  Re-enable FlushIT tests
  Bump Gradle heap to 2 GB (#30535)
  SQL: Use request flavored methods in tests (#30345)
  Suppress hdfsFixture if there are spaces in the path (#30302)
  Delete temporary blobs before creating index file (#30528)
  Watcher: Remove TriggerEngine.getJobCount() (#30395)
  [ML] Fix wire BWC for JobUpdate (#30512)
  Use simpler write-once semantics for FS repository (#30435)
  Derive max composite buffers from max content len
  Use simpler write-once semantics for HDFS repository (#30439)
  SQL: Improve correctness of SYS COLUMNS & TYPES (#30418)
  Mute two tests in FlushIT with @AwaitsFix.
  Fix incorrect template name in test case
  Build: Remove legacy bwc files from xpack (#30485)
  Mute UnicastZenPingTests#testSimplePings with @AwaitsFix.
  Security: cleanup code in file stores (#30348)
  Security: fix TokenMetaData equals and hashcode (#30347)
  Mute two tests from SmokeTestWatcherWithSecurityClientYamlTestSuiteIT.
  Mute SharedClusterSnapshotRestoreIT#testSnapshotSucceedsAfterSnapshotFailure with @AwaitsFix.
  SQL: Improve compatibility with MS query (#30516)
  SQL: Fix parsing of dates with milliseconds (#30419)
dnhatn added a commit that referenced this pull request May 14, 2018
@dnhatn
Merge branch '6.x' into ccr-6.x
1325c25 
* 6.x:
  Unmute IndexUpgradeIT tests
  Forbid expensive query parts in ranking evaluation (#30151)
  Docs: Update HighLevelRestClient migration docs (#30544)
  Clients: Switch to new performRequest (#30543)
  [TEST] Fix typo in MovAvgIT test
  [TEST] Mute ML test that needs updating to following ml-cpp changes
  Moved tokenizers to analysis common module (#30538)
  Document woes between auto-expand-replicas and allocation filtering (#30531)
  [ML] Hide internal Job update options from the REST API (#30537)
  Deprecate not copy settings and explicitly disallow (#30404)
  Mute ShrinkIndexIT suite
  SQL: SYS TABLES ordered according to *DBC specs (#30530)
  [ML] Improve state persistence log message
  Build: Add mavenPlugin cluster configuration method (#30541)
  Re-enable FlushIT tests
  Bump Gradle heap to 2 GB (#30535)
  Bump Gradle heap to 1792m (#30484)
  SQL: Use request flavored methods in tests (#30345)
  Suppress hdfsFixture if there are spaces in the path (#30302)
  Delete temporary blobs before creating index file (#30528)
  Watcher: Remove TriggerEngine.getJobCount() (#30395)
  Use simpler write-once semantics for FS repository (#30435)
  Use simpler write-once semantics for HDFS repository (#30439)
  SQL: Improve correctness of SYS COLUMNS & TYPES (#30418)
  Mute two tests in FlushIT with @AwaitsFix.
  Fix incorrect template name in test case
  Build: Remove legacy bwc files from xpack (#30485)
  Security: Simplify security index listeners (#30466)
  Mute SharedClusterSnapshotRestoreIT#testSnapshotSucceedsAfterSnapshotFailure with @AwaitsFix.
  Add proper longitude validation in geo_polygon_query (#30497)
  Mute UnicastZenPingTests#testSimplePings with @AwaitsFix.
  Security: cleanup code in file stores (#30348)
  Security: fix TokenMetaData equals and hashcode (#30347)
  Mute two tests from SmokeTestWatcherWithSecurityClientYamlTestSuiteIT.
  Fix incorrect merged entry in changelog
  SQL: Improve compatibility with MS query (#30516)
  SQL: Fix parsing of dates with milliseconds (#30419)
martijnvg added a commit that referenced this pull request May 15, 2018
@martijnvg
Merge remote-tracking branch 'es/ccr' into ccr_security
e083f5b 
* es/ccr: (37 commits)
  Default to one shard (#30539)
  Unmute IndexUpgradeIT tests
  Forbid expensive query parts in ranking evaluation (#30151)
  Docs: Update HighLevelRestClient migration docs (#30544)
  Clients: Switch to new performRequest (#30543)
  [TEST] Fix typo in MovAvgIT test
  Add missing dependencies on testClasses (#30527)
  [TEST] Mute ML test that needs updating to following ml-cpp changes
  Document woes between auto-expand-replicas and allocation filtering (#30531)
  Moved tokenizers to analysis common module (#30538)
  Adjust copy settings versions
  Mute ShrinkIndexIT suite
  SQL: SYS TABLES ordered according to *DBC specs (#30530)
  Deprecate not copy settings and explicitly disallow (#30404)
  [ML] Improve state persistence log message
  Build: Add mavenPlugin cluster configuration method (#30541)
  Re-enable FlushIT tests
  Bump Gradle heap to 2 GB (#30535)
  SQL: Use request flavored methods in tests (#30345)
  Suppress hdfsFixture if there are spaces in the path (#30302)
  ...
martijnvg added a commit that referenced this pull request May 15, 2018
@martijnvg
Merge remote-tracking branch 'es/ccr' into sync_mapping
dc8e79f 
* es/ccr: (37 commits)
  Default to one shard (#30539)
  Unmute IndexUpgradeIT tests
  Forbid expensive query parts in ranking evaluation (#30151)
  Docs: Update HighLevelRestClient migration docs (#30544)
  Clients: Switch to new performRequest (#30543)
  [TEST] Fix typo in MovAvgIT test
  Add missing dependencies on testClasses (#30527)
  [TEST] Mute ML test that needs updating to following ml-cpp changes
  Document woes between auto-expand-replicas and allocation filtering (#30531)
  Moved tokenizers to analysis common module (#30538)
  Adjust copy settings versions
  Mute ShrinkIndexIT suite
  SQL: SYS TABLES ordered according to *DBC specs (#30530)
  Deprecate not copy settings and explicitly disallow (#30404)
  [ML] Improve state persistence log message
  Build: Add mavenPlugin cluster configuration method (#30541)
  Re-enable FlushIT tests
  Bump Gradle heap to 2 GB (#30535)
  SQL: Use request flavored methods in tests (#30345)
  Suppress hdfsFixture if there are spaces in the path (#30302)
  ...
martijnvg added a commit to martijnvg/elasticsearch that referenced this pull request May 15, 2018
@martijnvg
Merge remote-tracking branch 'es/ccr' into ccr_more_validation
56e584a 
* es/ccr: (37 commits)
  Default to one shard (elastic#30539)
  Unmute IndexUpgradeIT tests
  Forbid expensive query parts in ranking evaluation (elastic#30151)
  Docs: Update HighLevelRestClient migration docs (elastic#30544)
  Clients: Switch to new performRequest (elastic#30543)
  [TEST] Fix typo in MovAvgIT test
  Add missing dependencies on testClasses (elastic#30527)
  [TEST] Mute ML test that needs updating to following ml-cpp changes
  Document woes between auto-expand-replicas and allocation filtering (elastic#30531)
  Moved tokenizers to analysis common module (elastic#30538)
  Adjust copy settings versions
  Mute ShrinkIndexIT suite
  SQL: SYS TABLES ordered according to *DBC specs (elastic#30530)
  Deprecate not copy settings and explicitly disallow (elastic#30404)
  [ML] Improve state persistence log message
  Build: Add mavenPlugin cluster configuration method (elastic#30541)
  Re-enable FlushIT tests
  Bump Gradle heap to 2 GB (elastic#30535)
  SQL: Use request flavored methods in tests (elastic#30345)
  Suppress hdfsFixture if there are spaces in the path (elastic#30302)
  ...
martijnvg added a commit to martijnvg/elasticsearch that referenced this pull request May 15, 2018
@martijnvg
Merge remote-tracking branch 'es/ccr' into ccr_create_and_follow
cdb1062 
* es/ccr: (37 commits)
  Default to one shard (elastic#30539)
  Unmute IndexUpgradeIT tests
  Forbid expensive query parts in ranking evaluation (elastic#30151)
  Docs: Update HighLevelRestClient migration docs (elastic#30544)
  Clients: Switch to new performRequest (elastic#30543)
  [TEST] Fix typo in MovAvgIT test
  Add missing dependencies on testClasses (elastic#30527)
  [TEST] Mute ML test that needs updating to following ml-cpp changes
  Document woes between auto-expand-replicas and allocation filtering (elastic#30531)
  Moved tokenizers to analysis common module (elastic#30538)
  Adjust copy settings versions
  Mute ShrinkIndexIT suite
  SQL: SYS TABLES ordered according to *DBC specs (elastic#30530)
  Deprecate not copy settings and explicitly disallow (elastic#30404)
  [ML] Improve state persistence log message
  Build: Add mavenPlugin cluster configuration method (elastic#30541)
  Re-enable FlushIT tests
  Bump Gradle heap to 2 GB (elastic#30535)
  SQL: Use request flavored methods in tests (elastic#30345)
  Suppress hdfsFixture if there are spaces in the path (elastic#30302)
  ...
@bleskes bleskes added v6.3.0 and removed v6.3.1 labels May 16, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DaveCTurner DaveCTurner DaveCTurner approved these changes

@inqueue inqueue inqueue approved these changes

Assignees

No one assigned

Labels

:Distributed/Allocation All issues relating to the decision making around placing a shard (both master logic & on the nodes) >docs General docs changes v6.3.0 v6.4.0 v7.0.0-beta1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@ywelsch @elasticmachine @DaveCTurner @inqueue @colings86 @bleskes