This repository was archived by the owner on Feb 18, 2025. It is now read-only.
ConsulTxnStore: batch KV updates by key-prefix to avoid ops limit#1311
Merged
shlomi-noach merged 16 commits intoopenark:masterfrom Apr 4, 2021
Merged
ConsulTxnStore: batch KV updates by key-prefix to avoid ops limit#1311shlomi-noach merged 16 commits intoopenark:masterfrom
shlomi-noach merged 16 commits intoopenark:masterfrom
Conversation
…to consul-txn-batch
Contributor
Author
|
@shlomi-noach I've confirmed this fixes the Consul Transaction support for environments that exceed the 64 max txn ops I think this is safe to merge 👍 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves issue: #1302
Description
This PR batches KVPair updates by key-prefix in to resolve errors for updates with more than 64 KVPairs. KVPairs are batched by key-prefix to ensure keys for a single "cluster" are updated in a single transaction atomically as only 5 x KVPair updates are necessary (< 64)
Details:
ConsulMaxKVsPerTransactionconfig-param for increasing transaction size, default to55causes 1 x transaction per cluster as there are 5 x KVs each5(# of KVs/cluster) and64(a Consul API limit).DistributePairs()mysql/master/cluster1,mysql/master/cluster1/hostname,mysql/master/cluster1/ipv4,mysql/master/cluster1/ipv6andmysql/master/cluster1/portis grouped into a single transaction at the defaultConsulMaxKVsPerTransaction=5sync.WaitGroupTransaction contains too many operations...to mock Consul server in unit test.DistributePairs()docs/kv.mdupdateI'm not 100% sure if my changes to
go/config/config.gofit with the theme. I also don't like that I made a constant for the number of KVPairs per "cluster". Any suggestions appreciated!Example debug-logging from
.DistributePairs():cc @shlomi-noach