feat: Dynamic flow control p2 #670
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov Report
@@ Coverage Diff @@
## dynamic_flow_control #670 +/- ##
=======================================================
Coverage ? 59.20%
Complexity ? 19
=======================================================
Files ? 2
Lines ? 125
Branches ? 18
=======================================================
Hits ? 74
Misses ? 34
Partials ? 17 Continue to review full report at Codecov.
|
ffd9de9 to
657c652
Compare
657c652 to
7a4ff43
Compare
mutianf
commented
Mar 10, 2021
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java
Show resolved
Hide resolved
igorbernstein2
requested changes
Mar 24, 2021
| flowControlEvents.getLastFlowControlEvent() == null | ||
| ? false | ||
| : (now - flowControlEvents.getLastFlowControlEvent().getTimestampMs() | ||
| <= TimeUnit.MINUTES.toMillis(5)); |
Contributor
There was a problem hiding this comment.
Can you extract the thresholds to named constants?
ie. MIN_THROTTLE_INTERVAL = Duration.ofMinutes(5)
CONCURRENCY_DECREASE_STEP = 0.3;
CONCURRENCY_INCREASE_STEP = ...
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java
Outdated
Show resolved
Hide resolved
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java
Outdated
Show resolved
Hide resolved
Comment on lines
34
to
35
| private final FlowControlEventStats flowControlEvents; | ||
| private final DynamicFlowControlStats dynamicFlowControlStats; |
Contributor
There was a problem hiding this comment.
I think we can improve on class naming here. The names are too similar which makes hard to remember what each does. Unfortunately I dont have good suggestions :(
Contributor
Author
There was a problem hiding this comment.
Yeah, I noticed this too but also don't have any better idea 😢
But after flowControlEvents is moved to flowController I think it's less confusing now?
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java
Outdated
Show resolved
Hide resolved
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java
Show resolved
Hide resolved
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java
Outdated
Show resolved
Hide resolved
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java
Outdated
Show resolved
Hide resolved
...loud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStub.java
Outdated
Show resolved
Hide resolved
|
|
||
| UnaryCallable<MutateRowsRequest, Void> flowControlCallable = null; | ||
| if (settings.bulkMutateRowsSettings().isLatencyBasedThrottlingEnabled()) { | ||
| long flowControlAdjustingIntervalMs = TimeUnit.SECONDS.toMillis(20); |
Contributor
There was a problem hiding this comment.
this should be a constant somewhere
igorbernstein2
approved these changes
Mar 29, 2021
Contributor
igorbernstein2
left a comment
There was a problem hiding this comment.
lgtm with a minor nit
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java
Outdated
Show resolved
Hide resolved
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java
Outdated
Show resolved
Hide resolved
mutianf
added a commit
to mutianf/java-bigtable
that referenced
this pull request
Apr 10, 2021
* feat: dynamic flow control for bulk mutation batcher * improve documents * add integration test * clean up tests and formatting * add more comments * moved FlowControlEventStats into FlowController * updates based on review p1 * updates based on review p2 * updates on review * update constant
mutianf
added a commit
to mutianf/java-bigtable
that referenced
this pull request
Apr 10, 2021
* feat: dynamic flow control for bulk mutation batcher * improve documents * add integration test * clean up tests and formatting * add more comments * moved FlowControlEventStats into FlowController * updates based on review p1 * updates based on review p2 * updates on review * update constant
mutianf
added a commit
to mutianf/java-bigtable
that referenced
this pull request
Apr 10, 2021
* feat: dynamic flow control for bulk mutation batcher * improve documents * add integration test * clean up tests and formatting * add more comments * moved FlowControlEventStats into FlowController * updates based on review p1 * updates based on review p2 * updates on review * update constant
mutianf
added a commit
to mutianf/java-bigtable
that referenced
this pull request
Apr 10, 2021
* feat: dynamic flow control for bulk mutation batcher * improve documents * add integration test * clean up tests and formatting * add more comments * moved FlowControlEventStats into FlowController * updates based on review p1 * updates based on review p2 * updates on review * update constant
mutianf
added a commit
to mutianf/java-bigtable
that referenced
this pull request
Apr 12, 2021
* feat: dynamic flow control for bulk mutation batcher * improve documents * add integration test * clean up tests and formatting * add more comments * moved FlowControlEventStats into FlowController * updates based on review p1 * updates based on review p2 * updates on review * update constant
igorbernstein2
pushed a commit
that referenced
this pull request
Apr 13, 2021
* feat: dynamic flow control part 1 (#620) * feat: dynamic flow control p1, add settings * update documents * move flow controller instantiation to stub.create * fix test * add some tests * split out settings to BatchCallSettings * add getters for target rpc latency * fix doc * more changes in doc * update document and checks on parameters * add more docs on flow control configs * fix test * split up enable/disable settings * feat: Dynamic flow control p2 (#670) * feat: dynamic flow control for bulk mutation batcher * improve documents * add integration test * clean up tests and formatting * add more comments * moved FlowControlEventStats into FlowController * updates based on review p1 * updates based on review p2 * updates on review * update constant * fix tests * make row key prefix random * updates based on review
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Part 2 of go/veneer-dynamic-flow-control.
Adding dynamic flow control callable to bulk mutation
Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
Fixes #<issue_number_goes_here> ☕️