Skip to content

chore: loadShards changes to more cleanly support 2.x feature#25528

Merged
gwossum merged 5 commits intomain-2.xfrom
gw_loadShards_cleanup_2.x
Nov 12, 2024
Merged

chore: loadShards changes to more cleanly support 2.x feature#25528
gwossum merged 5 commits intomain-2.xfrom
gw_loadShards_cleanup_2.x

Conversation

@gwossum
Copy link
Copy Markdown
Member

@gwossum gwossum commented Nov 8, 2024

  • chore: move shardID parsing and shard filtering into walkShardsAndProcess

  • chore: make it impossible to miss sending shardResponse or marking shard as complete

  • chore: always count number of shards (preparation for 2.x related feature)

  • chore: explicitly load series files and create indices serially

Explicitly load series files and create indices serially. Also avoid passing them to work functions that don't need them.

  • chore: rework loadShards for changes necessary to cancel loading process

  • chore: comment improvements

  • fix: fix race conditions in TestStore_StartupShardProgress and TestStore_BadShardLoading

  • chore: avoid logging nil error

  • chore: refactor shard loading and shard walking

Refactor loadShards and CreateShard to use a common shardLoader class that makes thread-safety easier. Refactor walkShardsAndProcess into findShards.

  • chore: improve comment

  • chore: rename OpenShard to ReopenShard and implement with shardLoader

Rename Store.OpenShard to Store.ReopenShard and implement using a shardLoader object. Changes to tests as necessary.

  • chore: avoid resetting shard options and locking on Reopen

Avoid resetting shard options when reopening a shard. Proper mutex locker in Shard.ReopenShard.

  • chore: fix formatting issue

  • chore: warn on mixed index types in Store.CreateShard

  • chore: change from info to warn when invalid shard IDs found in path

  • chore: use coarser locking in Store.ReopenShard

  • chore: fix typo in comment

  • chore: code simplification

(cherry picked from commit 0bc167b)

Closes #25527

* chore: move shardID parsing and shard filtering into walkShardsAndProcess

* chore: make it impossible to miss sending shardResponse or marking shard as complete

* chore: always count number of shards (preparation for 2.x related feature)

* chore: explicitly load series files and create indices serially

Explicitly load series files and create indices serially. Also
avoid passing them to work functions that don't need them.

* chore: rework loadShards for changes necessary to cancel loading process

* chore: comment improvements

* fix: fix race conditions in TestStore_StartupShardProgress and TestStore_BadShardLoading

* chore: avoid logging nil error

* chore: refactor shard loading and shard walking

Refactor loadShards and CreateShard to use a common shardLoader class that
makes thread-safety easier. Refactor walkShardsAndProcess into findShards.

* chore: improve comment

* chore: rename OpenShard to ReopenShard and implement with shardLoader

Rename Store.OpenShard to Store.ReopenShard and implement using a
shardLoader object. Changes to tests as necessary.

* chore: avoid resetting shard options and locking on Reopen

Avoid resetting shard options when reopening a shard.
Proper mutex locker in Shard.ReopenShard.

* chore: fix formatting issue

* chore: warn on mixed index types in Store.CreateShard

* chore: change from info to warn when invalid shard IDs found in path

* chore: use coarser locking in Store.ReopenShard

* chore: fix typo in comment

* chore: code simplification

(cherry picked from commit 0bc167b)
Copy link
Copy Markdown
Contributor

@davidby-influx davidby-influx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Fix reporting shards not opening correctly when they actually did.
Fix race condition with logging in loadShards.

(cherry picked from commit 65683bf)
Remove unnecessary fmt.Sprintf calls for static code checks in main-2.x.

(cherry picked from commit 8497fbf)
@gwossum gwossum marked this pull request as ready for review November 12, 2024 19:58
@gwossum gwossum merged commit 5c7479e into main-2.x Nov 12, 2024
@gwossum gwossum deleted the gw_loadShards_cleanup_2.x branch November 12, 2024 20:12
@gwossum gwossum mentioned this pull request Nov 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants