Add doc on master elections in DistributedArchitectureGuide by inespot · Pull Request #142435 · elastic/elasticsearch

inespot · 2026-02-13T00:02:53Z

Details the master election flow.

ES-14214

Details master eligibility, node roles, the election flow and failure cases.

github-actions · 2026-02-13T00:04:44Z

🔍 Preview links for changed docs

docs/internal/DistributedArchitectureGuide.md

github-actions · 2026-02-13T00:04:45Z

ℹ️ Important: Docs version tagging

👋 Thanks for updating the docs! Just a friendly reminder that our docs are now cumulative. This means all 9.x versions are documented on the same page and published off of the main branch, instead of creating separate pages for each minor version.

We use applies_to tags to mark version-specific features and changes.

Expand for a quick overview

When to use applies_to tags:

✅ At the page level to indicate which products/deployments the content applies to (mandatory)
✅ When features change state (e.g. preview, ga) in a specific version
✅ When availability differs across deployments and environments

What NOT to do:

❌ Don't remove or replace information that applies to an older version
❌ Don't add new information that applies to a specific version without an applies_to tag
❌ Don't forget that applies_to tags can be used at the page, section, and inline level

🤔 Need help?

Check out the cumulative docs guidelines
Reach out in the #docs Slack channel

elasticsearchmachine · 2026-02-17T03:26:20Z

Pinging @elastic/es-distributed (Team:Distributed)

elasticsearchmachine · 2026-02-17T03:26:21Z

Pinging @elastic/core-docs (Team:Docs)

inespot · 2026-02-17T03:33:30Z

docs/internal/DistributedArchitectureGuide.md


 (A node can coordinate a search across several other nodes, when the node itself does not have the data, and then return a result to the caller. Explain this coordinating role)

+### Cluster State


Outlined some additional subsections outside of Master Elections, to tackle in subsequent PRs.

inespot · 2026-02-17T04:29:38Z

docs/internal/DistributedArchitectureGuide.md

+
+[CoordinationMetadata]:https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/cluster/coordination/CoordinationMetadata.java
+
+[VotingConfiguration]: https://github.com/elastic/elasticsearch/blob/v9.3.0/server/src/main/java/org/elasticsearch/cluster/coordination/CoordinationMetadata.java#L326


This PR uses the v9.3.0 tag for all links not pointing to top-level classes to make sure the lines stay consistent. The existing documentation is a bit varied. Some sections use specific commits (like Snapshot Repository), and others (like HTTP Server) don't use links at all, just plain function names. If people have strong opinions on which is best, happy to adjust

+1 to using a release tag like v9.3.0 because it's immutable (in practice) but please don't refer to a branch like main as these things change over time.

Sounds good, will adjust for top level classes as well!

DaveCTurner

Great stuff, thanks for this.

DaveCTurner · 2026-02-17T08:26:20Z

docs/internal/DistributedArchitectureGuide.md

+
+[CoordinationMetadata]:https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/cluster/coordination/CoordinationMetadata.java
+
+[VotingConfiguration]: https://github.com/elastic/elasticsearch/blob/v9.3.0/server/src/main/java/org/elasticsearch/cluster/coordination/CoordinationMetadata.java#L326


+1 to using a release tag like v9.3.0 because it's immutable (in practice) but please don't refer to a branch like main as these things change over time.

DaveCTurner · 2026-02-17T08:33:28Z

docs/internal/DistributedArchitectureGuide.md

+
+[VotingConfiguration]: https://github.com/elastic/elasticsearch/blob/v9.3.0/server/src/main/java/org/elasticsearch/cluster/coordination/CoordinationMetadata.java#L326
+
+The cluster maintains at most a single master at all times. If no master is


This is the conceptual goal but it's surprisingly tricksy to even define what it even means for two nodes to be master at the same time. You can definitely have two nodes which each believe they are the master (and e.g. will service TransportMasterNodeAction requests) for a while, the key point is that all but at most one of them will not be able to update the cluster state.

Maybe too early to get into this level of detail? But it is worth saying somewhere, to avoid confusion about the exact invariants on which we can rely? You mention it below that we guarantee there will be at most one master in each term, and that the terms of committed cluster state updates are nondecreasing, so in a sense the term acts as a logical clock, but perhaps say there in the Terms section that different nodes may be at different logical times (i.e. terms) at the same physical time?

Suggested change

The cluster maintains at most a single master at all times. If no master is

The cluster maintains (conceptually at least) at most a single master at all times. If no master is

Also maybe worth highlighting at the top that the point of electing a master (and everything else here) is purely to update the cluster state. The elected master also does other things too but the cluster-state-updating bit is the only essential bit.

Added a sentence about the main point of having a master in 941c7 and then clarified the "two masters" case in 4186d

DaveCTurner · 2026-02-17T08:37:56Z

docs/internal/DistributedArchitectureGuide.md

+any [ClusterState] changes until a new master is elected.
+
+To elect a master, Elasticsearch uses a consensus algorithm derived
+from [Paxos](https://lamport.azurewebsites.net/pubs/lamport-paxos.pdf). This algorithm is formally defined in a TLA+


That's the original paper but maybe worth also linking these for a gentler introduction too:

https://lamport.azurewebsites.net/pubs/paxos-simple.pdf

https://raft.github.io/

DaveCTurner · 2026-02-17T08:40:41Z

docs/internal/DistributedArchitectureGuide.md

+To elect a master, Elasticsearch uses a consensus algorithm derived
+from [Paxos](https://lamport.azurewebsites.net/pubs/lamport-paxos.pdf). This algorithm is formally defined in a TLA+
+specification referenced from the [CoordinationState] class. The [Coordinator] class handles the core logic of the
+election, and manages how nodes transition between `CANDIDATE`, `LEADER`, and


Maybe worth documenting somewhere that CoordinationState is about safety whereas the Coordinator is more about liveness, managing timeouts and other behaviours that guarantee progress. It does a certain amount of admin work too, e.g. preparing all the data structures ahead of a publication.

docs/internal/DistributedArchitectureGuide.md

DaveCTurner · 2026-02-17T08:47:08Z

docs/internal/DistributedArchitectureGuide.md

+
+#### Election Flow
+
+The overall election flow looks like this:


I think I'd rather this was more prose-like (e.g. so you can copy-paste the sentences elsewhere) - I'm not sure the boxes and arrows really add much to this straight-line flow, and they are a royal pain to maintain in future edits.

Something like this perhaps?

Leader failure detected.

Follower detects current master failure

See:

LeaderChecker Coordinator.onLeaderFailure()

Node becomes CANDIDATE

Follower transitions to CANDIDATE mode which triggers the discovery process.

See:

Coordinator.becomeCandidate() Mode.CANDIDATE PeerFinder.activate(...)

etc.

Works for me, I'll switch this to be pure prose

DaveCTurner · 2026-02-17T08:49:39Z

docs/internal/DistributedArchitectureGuide.md

+    └───────────────────────────────────────────────┴──────────────────────────────────┘
+```
+
+#### Failure Detection


You asked a good question in the onboarding session about the reasons for having checks in both directions - would you cover that point here?

DaveCTurner · 2026-02-17T08:53:28Z

docs/internal/DistributedArchitectureGuide.md

+the next `handleWakeUp` iteration
+
+When [receiving](https://github.com/elastic/elasticsearch/blob/v9.3.0/server/src/main/java/org/elasticsearch/discovery/PeerFinder.java#L534)
+a [PeersResponse], [PeerFinder] will reach out to all peers specified in the response, including a potential master. If


Nit but maybe worth mentioning that we also reach back out to nodes that send us requests for peers.

DaveCTurner · 2026-02-17T09:00:43Z

docs/internal/DistributedArchitectureGuide.md

+
+[DiscoveryPlugin]: https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/plugins/DiscoveryPlugin.java
+
+Discovery is a fast "gossip-like" protocol by which a node in `CANDIDATE` mode locates master-eligible nodes in the


Not sure if you want to mention how fast we mean by "fast" but FWIW it will discover every other master-eligible node in at most something like ⌈log₂(D)+1⌉ steps where D is the diameter of the graph of seed host configurations.

DaveCTurner

LGTM

…on-sliced-reindex * upstream/main: Activity logging improvements (elastic#142901) Fix serialization of NodeGpuStatsResponse when no GPU is present (elastic#142937) Add doc on master elections in DistributedArchitectureGuide (elastic#142435) ESQL: Account for missing StubRelation due to SurrogateExpressions replacement (elastic#142882) Add BulkByScrollTask Serialization Tests (elastic#142697) Rebalance CI test partitions to reduce Part3 bottleneck (elastic#142930) Mute org.elasticsearch.xpack.esql.qa.multi_node.EsqlClientYamlIT test {p0=esql/40_tsdb/to_aggregate_metric_double with multi_values} elastic#142964 Bump OpenTelemetry dependencies (elastic#142323) SQL: add support for API key to JDBC and CLI (elastic#142021) Ensure requested capability exists (elastic#142695) Warn and fall back to local branches.json (elastic#142606) [CI] Mute testWithFetchFailures, testAddCompletionListenerScheduleErr… (elastic#142926) ESQL: Add support for ORC file format (elastic#142900) Update wolfi (versioned) (elastic#142948) Add BulkByScrollResponse Serialization Tests (elastic#142688) Run 25_id_generation with and without synthetic id (elastic#142770)

Add doc on master elections.

e9c27a4

Details master eligibility, node roles, the election flow and failure cases.

elasticsearchmachine added the v9.4.0 label Feb 13, 2026

inespot added 2 commits February 14, 2026 17:33

Detail the election flow + flush out other sections

afba645

Discovery section

cf25b4b

inespot force-pushed the ip/master-election-doc branch from 70f489d to cf25b4b Compare February 14, 2026 23:35

inespot added 4 commits February 15, 2026 20:12

Failure detection and first draft of serverless

8b6f148

Join section + small tweaks

800a146

Pre-Vote phase

d5d4c06

Typos and grammar

dfacf34

inespot marked this pull request as ready for review February 17, 2026 03:24

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Feb 17, 2026

inespot added :Distributed/Distributed A catch all label for anything in the Distributed Area. Please avoid if you can. >non-issue >docs General docs changes and removed needs:triage Requires assignment of a team area label labels Feb 17, 2026

elasticsearchmachine added the Team:Distributed Meta label for distributed team. label Feb 17, 2026

elasticsearchmachine added the Team:Docs Meta label for docs team label Feb 17, 2026

inespot requested a review from DaveCTurner February 17, 2026 03:26

Add todo to cluster state

f7bf056

inespot commented Feb 17, 2026

View reviewed changes

DaveCTurner reviewed Feb 17, 2026

View reviewed changes

inespot added 2 commits February 17, 2026 14:10

Review suggestions

941c7dc

Clarify master invariant

4186d37

inespot requested a review from DaveCTurner February 17, 2026 19:46

inespot mentioned this pull request Feb 22, 2026

Add doc on ClusterState in DistributedArchitectureGuide #142776

Merged

DaveCTurner approved these changes Feb 24, 2026

View reviewed changes

inespot merged commit c7f0870 into elastic:main Feb 24, 2026
12 checks passed

inespot mentioned this pull request Feb 25, 2026

Node roles doc in DistributedArchitectureGuide #143014

Merged


		(A node can coordinate a search across several other nodes, when the node itself does not have the data, and then return a result to the caller. Explain this coordinating role)

		### Cluster State


		[CoordinationMetadata]:https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/cluster/coordination/CoordinationMetadata.java

		[VotingConfiguration]: https://github.com/elastic/elasticsearch/blob/v9.3.0/server/src/main/java/org/elasticsearch/cluster/coordination/CoordinationMetadata.java#L326


		[VotingConfiguration]: https://github.com/elastic/elasticsearch/blob/v9.3.0/server/src/main/java/org/elasticsearch/cluster/coordination/CoordinationMetadata.java#L326

		The cluster maintains at most a single master at all times. If no master is

	The cluster maintains at most a single master at all times. If no master is
	The cluster maintains (conceptually at least) at most a single master at all times. If no master is


		#### Election Flow

		The overall election flow looks like this:


		[DiscoveryPlugin]: https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/plugins/DiscoveryPlugin.java

		Discovery is a fast "gossip-like" protocol by which a node in `CANDIDATE` mode locates master-eligible nodes in the

Conversation

inespot commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Feb 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔍 Preview links for changed docs

Uh oh!

github-actions bot commented Feb 13, 2026

ℹ️ Important: Docs version tagging

When to use applies_to tags:

What NOT to do:

🤔 Need help?

Uh oh!

elasticsearchmachine commented Feb 17, 2026

Uh oh!

elasticsearchmachine commented Feb 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

inespot Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DaveCTurner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

inespot Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DaveCTurner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

inespot commented Feb 13, 2026 •

edited

Loading

github-actions bot commented Feb 13, 2026 •

edited

Loading

inespot Feb 17, 2026 •

edited

Loading

inespot Feb 17, 2026 •

edited

Loading