Skip to content

Log [initial_master_nodes] on formation failure#36466

Merged
DaveCTurner merged 1 commit intoelastic:masterfrom
DaveCTurner:2018-12-11-better-cluster-formation-failure-log
Dec 11, 2018
Merged

Log [initial_master_nodes] on formation failure#36466
DaveCTurner merged 1 commit intoelastic:masterfrom
DaveCTurner:2018-12-11-better-cluster-formation-failure-log

Conversation

@DaveCTurner
Copy link
Copy Markdown
Member

@DaveCTurner DaveCTurner commented Dec 11, 2018

Today we log a slightly cryptic "cluster bootstrapping is disabled on this
node" message if bootstrapping hasn't been configured. Since there is today
only one way to bootstrap the cluster it seems preferable to spell out exactly
which setting is missing.

@DaveCTurner DaveCTurner added >non-issue v7.0.0 :Distributed/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. labels Dec 11, 2018
@DaveCTurner DaveCTurner requested a review from bleskes December 11, 2018 08:50
@elasticmachine
Copy link
Copy Markdown
Collaborator

elasticmachine commented Dec 11, 2018

Pinging @elastic/es-distributed

bleskes
bleskes approved these changes Dec 11, 2018
View reviewed changes
Copy link
Copy Markdown
Contributor

@bleskes bleskes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks. Left an optional suggestion

server/src/main/java/org/elasticsearch/cluster/coordination/ClusterFormationFailureHelper.java Outdated
Copy link
Copy Markdown
Contributor

@bleskes bleskes Dec 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think technically we check for is not set on this node, shall we just use that?

Copy link
Copy Markdown
Member Author

@DaveCTurner DaveCTurner Dec 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We check if it's set here:

elasticsearch/server/src/main/java/org/elasticsearch/cluster/coordination/ClusterBootstrapService.java

Lines 84 to 85 in 6e6e63d

return Stream.of(DISCOVERY_HOSTS_PROVIDER_SETTING, DISCOVERY_ZEN_PING_UNICAST_HOSTS_SETTING,
INITIAL_MASTER_NODE_COUNT_SETTING, INITIAL_MASTER_NODES_SETTING).anyMatch(s -> s.exists(settings));

If it's not set, and nor are any of the other settings listed here, then we auto-bootstrap after a few seconds which leads to a different log message that starts master not discovered or elected yet and doesn't mention bootstrapping.

However if bootstrapping hasn't occurred by the time we write this message then this can't have happened, so the problem is actually that it's empty:

elasticsearch/server/src/main/java/org/elasticsearch/cluster/coordination/ClusterBootstrapService.java

Line 142 in 6e6e63d

} else if (initialMasterNodeCount > 0 || initialMasterNodes.isEmpty() == false) {

Copy link
Copy Markdown
Contributor

@bleskes bleskes Dec 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My comment was just targeted at the if clause:

INITIAL_MASTER_NODES_SETTING.get(Settings.EMPTY).equals(INITIAL_MASTER_NODES_SETTING.get(settings))```

Copy link
Copy Markdown
Member Author

@DaveCTurner DaveCTurner Dec 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also checks it's not empty, since the default for this setting is empty.

Copy link
Copy Markdown
Contributor

@bleskes bleskes Dec 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It technically checks it's not set or set to the default (which is empty). As I said this was just a suggestion. I'm good anyway.

@DaveCTurner
Log [initial_master_nodes] on formation failure
a330fac 
Today we log a slightly cryptic "cluster bootstrapping is disabled on this
node" message if bootstrapping hasn't been configured. Since there is today
only one way to bootstrap the cluster it seems preferable to spell out exactly
which setting is missing.
@DaveCTurner DaveCTurner force-pushed the 2018-12-11-better-cluster-formation-failure-log branch from c41fa2e to a330fac Compare December 11, 2018 10:20
@DaveCTurner
Copy link
Copy Markdown
Member Author

DaveCTurner commented Dec 11, 2018

Transient failure:

 [6.5.3] * What went wrong:
:x-pack:plugin:security:testJar (Thread[Execution worker for ':' Thread 7,5,main]) completed. Took 0.585 secs.
 [6.5.3] Could not resolve all files for configuration ':buildSrc:runtimeClasspath'.
 [6.5.3] > Could not download jna.jar (org.elasticsearch:jna:4.5.1)
 [6.5.3]    > Could not get resource 'https://jcenter.bintray.com/org/elasticsearch/jna/4.5.1/jna-4.5.1.jar'.
 [6.5.3]       > Could not HEAD 'https://jcenter.bintray.com/org/elasticsearch/jna/4.5.1/jna-4.5.1.jar'.
 [6.5.3]          > Connection reset

@elasticmachine please run the Gradle build tests 1

@DaveCTurner DaveCTurner merged commit c3a6d19 into elastic:master Dec 11, 2018
@DaveCTurner DaveCTurner deleted the 2018-12-11-better-cluster-formation-failure-log branch December 11, 2018 12:53
jasontedor added a commit to dnhatn/elasticsearch that referenced this pull request Dec 11, 2018
@jasontedor
Merge remote-tracking branch 'elastic/master' into pr/36141
7fc6dc2 
* elastic/master: (36 commits)
  Add check for minimum required Docker version (elastic#36497)
  Minor search controller changes (elastic#36479)
  Add default methods to DocValueFormat (elastic#36480)
  Fix the mixed cluster REST test explain/11_basic_with_types.
  Modify `BigArrays` to take name of circuit breaker (elastic#36461)
  Move LoggedExec to minimumRuntime source set (elastic#36453)
  added 6.5.4 version
  Add test logging for elastic#35644
  Tests- added helper methods to ESRestTestCase for checking warnings (elastic#36443)
  SQL: move requests' parameters to requests JSON body (elastic#36149)
  [Zen2] Respect the no_master_block setting (elastic#36478)
  Require soft-deletes when access changes snapshot (elastic#36446)
  Switch more tests to zen2 (elastic#36367)
  [Painless] Add extensive tests for def to primitive casts (elastic#36455)
  Add setting to bypass Rollover action (elastic#36235)
  Try running CI against Zulu (elastic#36391)
  [DOCS] Reworked the shard allocation filtering info.  (elastic#36456)
  Log [initial_master_nodes] on formation failure (elastic#36466)
  converting ForbiddenPatternsTask to .java (elastic#36194)
  fixed typo
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@bleskes bleskes bleskes approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

:Distributed/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. >non-issue v7.0.0-beta1

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@DaveCTurner @elasticmachine @bleskes @colings86