Shadow replicas on shared filesystems by dakrone · Pull Request #9727 · elastic/elasticsearch

dakrone · 2015-02-17T16:07:34Z

These commits add the shadow replicas feature for use on shared filesystems
(it does not include segment replication for non-shared filesystems yet).

If we assume that the data in the index path will already be shared across
multiple nodes, we can create and index with shadow replicas, where each replica
shard simply contains an IndexReader that periodically refreshes to pick up
new segments.

All indexing operations will be executed on the primary shard, and will not be
replicated to each replica, since the data will be replicated in a different
way.

During this phase, creating an index with index.shadow_replicas: true and
number_of_replicas greater than 0 will cause operations not to undergo
replication to replica shards. An index can have either regular replicas or
shadow replicas; they are mutually exclusive for an index. The
index.shadow_replicas setting is set at index creation time and cannot be
changed dynamically.

The Elasticsearch cluster will still detect the loss of a primary shard, and
transform the replica into a primary in this situation. This transformation will
take slightly longer, since no IndexWriter will be maintained for each shadow
replica.

In order to ensure the data is being synchronized in a fast enough manner, The
user will need to tune the flush threshold for the index to a desired number. A
flush is needed to fsync segment files to disk, so they will be visible to all
other replica nodes. Users should test what flush threshold levels they are
comfortable with, as increased flushing can impact indexing performance. This
testing can be performed at any time, there is no need to wait for this feature
to be available first.

Once segments are available on the filesystem where the shadow replica resides,
a regular refresh (governed by the index.refresh_interval) can be used to make
the new data searchable.

See #8976 for the overall shadow replica plan

Conflicts: src/main/java/org/elasticsearch/gateway/GatewayMetaState.java

… etc.

…themselves

Also adds a nocommit

make flush to a refresh factor our ShadowIndexShard to have IndexShard be idential to the master and least intrusive cleanup abstractions

Conflicts: src/main/java/org/elasticsearch/index/engine/Engine.java

…dler that skip most phases and enforces shard closing on the soruce before the target opens it's engine

kimchy · 2015-02-18T16:18:13Z

...a/org/elasticsearch/action/support/replication/TransportShardReplicationOperationAction.java

I would clarify this statement? cause mapping updates do get replicated, it just takes longer since it needs to head to the master and then published to the replicas, so there is a delay in mapping introduction.

I will clarify this comment

kimchy · 2015-02-18T16:53:53Z

src/main/java/org/elasticsearch/cluster/metadata/IndexMetaData.java

can we use the same method structure between this method and the following? I like isIndex...

Yep, I'll rename.

dakrone · 2015-02-18T17:07:21Z

@s1monw I added a MockShadowEngine and addressed your other comments

kimchy · 2015-02-18T17:43:30Z

src/main/java/org/elasticsearch/index/shard/IndexShardModule.java

should we use the IndexMetaData#isIndexUsingShadowReplicas help method here?

Yeah, I will change this to use the helper

kimchy · 2015-02-18T17:47:04Z

left really minor comments, it looks great. One note, should we mention in the docs the second phase tasks, like do primary promotion without failing an engine? If so, I would also add a task that on get, we automatically set the "go to primary" flag if shadow replica is used and realtime get is used?

s1monw · 2015-02-18T19:28:53Z

left two more comments other than that LGTM

dakrone · 2015-02-18T21:42:00Z

Pushed more commits hooking up the MockShadowEngine, moving the engine creation into the ShadowIndexShard and automatically using ?preference=_primary when doing a realtime GET.

s1monw · 2015-02-18T22:24:06Z

src/main/java/org/elasticsearch/action/get/TransportGetAction.java

can we use org.elasticsearch.cluster.routing.operation.plain.Preference.PRIMARY.type() here instead?

s1monw · 2015-02-18T22:24:32Z

left on minor comment! LGTM feel free to push!

jpountz · 2015-02-19T17:07:46Z

docs/reference/indices/shadow-replicas.asciidoc

Thanks for this!

dakrone · 2015-02-19T21:50:42Z

pushed to 1.x and master!

dakrone and others added 30 commits February 10, 2015 13:10

Add ShadowEngine

05975af

make tests pass

38135af

make test more evil

7fcb373

Add test that restarts nodes to ensure shadow replicas recover

be02cab

long adder is not available in java7

343dc0b

Merge branch 'master' into shadow-replicas

2a2eed1

Conflicts: src/main/java/org/elasticsearch/gateway/GatewayMetaState.java

utilize the new delete code

24d36c9

shortcut recovery if we are on a shared FS - no need to compare files…

2d42736

… etc.

Merge branch 'master' into shadow-replicas

ca9beb2

Add start of ShadowEngine unit tests

67d7df4

Add testShadowEngineIgnoresWriteOperations and testSearchResultRelease

1896fed

Remove tests that don't apply to ShadowEngine

a95adbe

Add a test for replica -> primary promotion

52e9cd1

Fix missing import

2378fbb

Remove overly-complex test

5e33eea

Remove nocommit in ShadowEngineTests#testFailStart()

80cf0e8

Fix segment info for ShadowEngine, remove test nocommit

e4dbfb0

Add a test checking that indices with shadow replicas clean up after …

06e2eb4

…themselves

Use check for shared filesystem in primary -> primary relocation

fdbe413

Also adds a nocommit

Add testShadowReplicaNaturalRelocation

5689b7d

Make assertPathHasBeenCleared recursive

cf2fb80

fix primary promotion

4a367c0

first cut at catchup from primary

f229719

make flush to a refresh factor our ShadowIndexShard to have IndexShard be idential to the master and least intrusive cleanup abstractions

Merge branch 'master' into shadow-replicas

abda780

Conflicts: src/main/java/org/elasticsearch/index/engine/Engine.java

fix compile error after upstream changes

a62b9a7

Simplify shared filesystem recovery by using a dedicated recovery han…

a7eb53c

…dler that skip most phases and enforces shard closing on the soruce before the target opens it's engine

Refactor more shared methods into the abstract Engine

d8d59db

Remove nocommit, document canDeleteIndexContents

28a9d18

Add documentation to ShadowEngine

4f71c8d

Add documentation to ShadowIndexShard, remove nocommit

ea4e3e5

dakrone added 2 commits February 18, 2015 08:35

Rename ownsShard to canDeleteShardContent

d90d698

Revert changes to RecoveryTarget.java

7346f9f

kimchy reviewed Feb 18, 2015
View reviewed changes

dakrone added 2 commits February 18, 2015 09:18

Add a test for shadow replicas that uses field data

60a4d53

Clarify comment about pre-defined mappings

c8e8db4

kimchy reviewed Feb 18, 2015
View reviewed changes

dakrone added 2 commits February 18, 2015 09:58

Add MockShadowEngine and hook it up to be used

73c62df

Rename usesSharedFilesystem -> isOnSharedFilesystem

1a0d456

kimchy reviewed Feb 18, 2015
View reviewed changes

Use IndexMetaData.isIndexUsingShadowReplicas helper

62b0c28

dakrone added 3 commits February 18, 2015 13:14

Factor out AssertingSearcher so it can be used by mock Engines

67a797a

Move engine creation into protected createNewEngine method

edd4943

Use ?preference=_primary automatically for realtime GET operations

325acbe

s1monw reviewed Feb 18, 2015
View reviewed changes

Use Enum for "_primary" preference

2083503

jpountz reviewed Feb 19, 2015
View reviewed changes

docs/reference/indices/shadow-replicas.asciidoc

Copy link
Copy Markdown

Contributor

jpountz Feb 19, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this!

dakrone closed this Feb 19, 2015

clintongormley added :Shadow Replicas and removed :Engine labels Jun 6, 2015

colings86 mentioned this pull request Aug 4, 2016

Should we remove/modify some of the experiment tags in the documentation #19798

Closed

Conversation

dakrone commented Feb 17, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dakrone commented Feb 18, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kimchy commented Feb 18, 2015

Uh oh!

s1monw commented Feb 18, 2015

Uh oh!

dakrone commented Feb 18, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

s1monw commented Feb 18, 2015

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dakrone commented Feb 19, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants