feat: defer closing connections until async operations complete #37

prawilny · 2021-09-29T15:15:10Z

This change is

dopiera

Reference counting is only an implementation detail, please adjust the commit message to reflect what you're changing and why. E.g.

feat: defer closing connections until async operations complete

Mirroring client schedules asynchronous operations - to mirror the mutations and to verify reads. Before this PR, closing the MirroringClient would result in closing the underlying clients immediately. This made the pending asynchronous operations fail. This PR defers closing the underlying clients until all pending operations complete. It is achieved by reference counting the operations.

Reviewable status: 0 of 5 files reviewed, 2 unresolved discussions (waiting on @mwalkiewicz and @prawilny)

bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncConnection.java, line 119 at r1 (raw file):

  @Override
  public boolean isClosed() {

I think hbase connections should be thread safe. This is not.

bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncTable.java, line 88 at r1 (raw file):

  }

  private CompletableFuture<Void> createReferenceHoldingFuture() {

I don't understand why it needs to be a future. We are using it as if it was a simple callback.

I thought the idea to use the future was to simply add one continuation to all verifications and be done with it. Correct me if I'm wrong, but the way it's written we may just as well call referenceCounter.incrementReferenceCount()everywhere, where you call createReferenceHoldingFuture and call referenceCounter.decrementReferenceCount() everywhere, where you call complete(null). If that's the case, it would be much simpler, no?

prawilny

Will use the provided title and message.
Applied the suggestions.

Reviewable status: 0 of 5 files reviewed, 2 unresolved discussions (waiting on @dopiera and @mwalkiewicz)

bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncConnection.java, line 119 at r1 (raw file):

Previously, dopiera (Marek Dopiera) wrote…

I think hbase connections should be thread safe. This is not.

Used synchronized in close() signature.

bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncTable.java, line 88 at r1 (raw file):

Previously, dopiera (Marek Dopiera) wrote…

I don't understand why it needs to be a future. We are using it as if it was a simple callback.

I thought the idea to use the future was to simply add one continuation to all verifications and be done with it. Correct me if I'm wrong, but the way it's written we may just as well call referenceCounter.incrementReferenceCount()everywhere, where you call createReferenceHoldingFuture and call referenceCounter.decrementReferenceCount() everywhere, where you call complete(null). If that's the case, it would be much simpler, no?

Changed to ReferenceCounter

dopiera

Reviewable status: 0 of 5 files reviewed, 2 unresolved discussions (waiting on @dopiera, @mwalkiewicz, and @prawilny)

bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncConnection.java, line 119 at r1 (raw file):

Previously, prawilny (Adam Czajkowski) wrote…

Used synchronized in close() signature.

That, unfortunately is not enough. isClosed() may return stale data.

mwalkiewicz

Reviewed all commit messages.
Reviewable status: 0 of 5 files reviewed, 2 unresolved discussions (waiting on @dopiera, @mwalkiewicz, and @prawilny)

bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncTable.java, line 88 at r1 (raw file):

Previously, prawilny (Adam Czajkowski) wrote…

Changed to ReferenceCounter

I think that the code now is more complicated, because reference counting logic leaks into handling operations and it is easy now to accidentally skip one increase/decrease.

If we placed reference counting logic into a callback with a pleasant name (pleaseCallMeWhenAllOperationsAreDone :) ) wouldn't have to worry about "reference counting", but only about "calling operation done callback", which is explicit, simple and it is obvious when to call it. This will make the code easier to maintain IMO.

The base branch was changed.

dopiera

Reviewable status: 0 of 5 files reviewed, 1 unresolved discussion (waiting on @dopiera, @mwalkiewicz, and @prawilny)

bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncTable.java, line 88 at r1 (raw file):

Previously, mwalkiewicz (Mateusz Walkiewicz) wrote…

I think that the code now is more complicated, because reference counting logic leaks into handling operations and it is easy now to accidentally skip one increase/decrease.

If we placed reference counting logic into a callback with a pleasant name (pleaseCallMeWhenAllOperationsAreDone :) ) wouldn't have to worry about "reference counting", but only about "calling operation done callback", which is explicit, simple and it is obvious when to call it. This will make the code easier to maintain IMO.

I'd argue if it would be more readable. IMO, initially, that looks great, but after a while, you forget what an "operation" is (think batch) and what "done" is (successful, failed, or maybe reported to the user), so you need to see what it does anyway, which means more things to comprehend rather than fewer. Perhaps it's just a taste thing, though.

Either way, I think we can at least make it a method rather a callback/future passed around, which I'd strongly argue is more readable (smaller method signatures and most importantly no IoC; when you have IoC you need to know all the callers to reason about what is going on there).

I don't have a strong preference for a method or the decrement directly, but I think this isn't a case for IoC.

prawilny

Reviewable status: 0 of 5 files reviewed, 1 unresolved discussion (waiting on @dopiera and @mwalkiewicz)

bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncConnection.java, line 119 at r1 (raw file):

Previously, dopiera (Marek Dopiera) wrote…

That, unfortunately is not enough. isClosed() may return stale data.

Done.

bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncTable.java, line 88 at r1 (raw file):

Previously, dopiera (Marek Dopiera) wrote…

I'd argue if it would be more readable. IMO, initially, that looks great, but after a while, you forget what an "operation" is (think batch) and what "done" is (successful, failed, or maybe reported to the user), so you need to see what it does anyway, which means more things to comprehend rather than fewer. Perhaps it's just a taste thing, though.

Either way, I think we can at least make it a method rather a callback/future passed around, which I'd strongly argue is more readable (smaller method signatures and most importantly no IoC; when you have IoC you need to know all the callers to reason about what is going on there).

I don't have a strong preference for a method or the decrement directly, but I think this isn't a case for IoC.

I rewrote it a bit: there's a method that calls the reference counter and AsyncRequestSchedulingaccepts a Runnable argument instead of ListenableReferenceCounter.

prawilny

The non-test code has been nearly completely rewritten.

Reviewable status: 0 of 5 files reviewed, 1 unresolved discussion (waiting on @dopiera and @mwalkiewicz)

bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncTable.java, line 88 at r1 (raw file):

Previously, prawilny (Adam Czajkowski) wrote…

I rewrote it a bit: there's a method that calls the reference counter and AsyncRequestSchedulingaccepts a Runnable argument instead of ListenableReferenceCounter.

Rewrote it as written above.

prawilny

I'd advise looking at full diff - it's way cleaner.

Reviewable status: 0 of 6 files reviewed, 1 unresolved discussion (waiting on @dopiera and @mwalkiewicz)

prawilny

Note that I'm simultaneously fixing both PRs: this and FutureUtils one. So some changes may be fixed in the other PR.

Reviewable status: 0 of 6 files reviewed, 3 unresolved discussions (waiting on @dopiera and @mwalkiewicz)

bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncTable.java, line 197 at r10 (raw file):

Previously, dopiera (Marek Dopiera) wrote…

I think the change looks very good, however, I don't like this function - it seems to glue to together two completely unrelated things.

How about a different one:
void keepReferenceUntilOperationCompletes(CompletableFuture<Void> future) {
  referenceCounter.increment();
  fut.thenApply(() -> referenceCounter.decrement());
}
So that the code would look like this:
... primaryFuture = ...;
CompletableFuture<...> completionStages = writeWithFlowControl(...);
keepReferenceUntilOperationCompletes(stages.verificationFuture);
return completionStages.primaryFuture;
WDYT?

I feel like not incrementing the reference counter before doing all the work may possibly introduce a race condition.

Refactored the code as suggested but with the incrementation of the reference counter out of the function.

bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/utils/futures/FutureUtils.java, line 51 at r10 (raw file):

Previously, dopiera (Marek Dopiera) wrote…

This method is unused.

And it never will be. Removed

dopiera

Reviewable status: 0 of 6 files reviewed, 3 unresolved discussions (waiting on @dopiera, @mwalkiewicz, and @prawilny)

bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncTable.java, line 197 at r10 (raw file):

Previously, prawilny (Adam Czajkowski) wrote…

I feel like not incrementing the reference counter before doing all the work may possibly introduce a race condition.

Refactored the code as suggested but with the incrementation of the reference counter out of the function.

Why would it create a race condition?

In order for this to fail, the user would have to call close while in the middle of a method call - that is no different than calling the method (e.g. get()) after calling close().

I think putting the increment in the same function makes it slightly safer - we don't need to ensure that there is an increment for every keepReferenceUntilOperationCompletes() call.

prawilny

Dismissed @dopiera from a discussion.
Reviewable status: 0 of 7 files reviewed, 2 unresolved discussions (waiting on @dopiera and @mwalkiewicz)

bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncTable.java, line 197 at r10 (raw file):

Previously, dopiera (Marek Dopiera) wrote…

Why would it create a race condition?

In order for this to fail, the user would have to call close while in the middle of a method call - that is no different than calling the method (e.g. get()) after calling close().

I think putting the increment in the same function makes it slightly safer - we don't need to ensure that there is an increment for every keepReferenceUntilOperationCompletes() call.

Done (incrementation moved into the function).
The reason I brought that up was that I thought that calling close() in another thread wasn't forbidden just because "The implementation is required to be thread safe".

dopiera

Reviewable status: 0 of 7 files reviewed, 1 unresolved discussion (waiting on @dopiera and @mwalkiewicz)

The base branch was changed.

prawilny · 2021-10-15T13:45:38Z

https://github.com/Unoperate/java-bigtable-hbase-1/actions/runs/1346154433

mwalkiewicz

Reviewed 2 of 4 files at r9, 2 of 5 files at r12, 2 of 3 files at r14, 2 of 2 files at r15, all commit messages.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on @dopiera)

Mirroring client schedules asynchronous operations - to mirror the mutations and to verify reads. Before this PR, closing the MirroringAsyncConnection would result in closing the underlying connections immediately. This made the pending asynchronous operations fail. This PR defers closing the underlying connections until all pending operations complete. It is achieved by reference counting the operations.

* Initial implementation of hbase1.x Mirroring Client (#1) * Initial implementation of hbase1.x Mirroring Client * MirroringTable and AsyncTableWrapper (#4) * MirroringTable and AsyncTableWrapper - Initial implementation of MirroringTable. - Wrapper around HBase 1.x Table enabling scheduling asynchronous operations. - Asynchronous verification of get/exists results. * Add missing licence header to TestResultComparator (#6) * Setup running tests in GitHub Actions (#7) * Asynchronous ResultScanner (#5) * Asynchronous ResultScanner * Implement faillog.Appender (#2) `faillog.Appender` is the lowermost component of the infrastructure for dumping failed mutations in the `MirroringClient`. `faillog/README.md` explains the design decisions in a bit more detail. `faillog.Appender` essentially logs arbitrary data asynchronously, through a separate thread reading from a bounded buffer. * ListenableReferenceCounter (#8) * Count references in asynchronous tasks before closing Table/Scanner. * Flow controller (#10) * Flow Controller * Flow Control strategy based on single requests queue * Mirroring table: writes (#12) * Add missing condition in result comparator (#15) * Add more mirroring config opitons (#16) * MismatchDetector implementation. * FlowControllerStrategy implementation. * Maximal number of outstanding requests used by FlowControllerStrategy. * Primary/Secondary connection implementation option now accepts "default" value which can be used when default HBase Connection implementation should be used by MirroringConnection. * Make AsyncTableWrapper ListenableCloseable (#20) Implements our standard interface for objects that can run callbacks after asynchronous close to simplify reference counting of MirroringTable. * MirroringConnection: use new config options (#17) * MirroringResultScanner improvements (#18) * Count references to MirroringResultScanner Current implementation of MirroringResultScanner doesn't count verificaiton requests that it has scheduled and allows to close instances while verificaiton requests are in-flight. This causes lost verifications. This PR fixes this issue by counting references to MirroringResultScanner instances when scheduling verification requests. Moreover, ListenableClosable interface is implemented for consistency with other classes that use this scheme, because now the MirroringResultScanner instances will be closed asynchronously, when all scheduled requests have finished. * MirroringTable: count references to scanners and secondary wrapper (#21) Current MirroringTable implementaion does not count its references held by MirroringResultScanners and SecondaryAsyncWrapper, thus MirroringConnection consideres it closed before we are sure that all asynchronous operations have completed. This PR adds correct reference counting of MirroringTable based on work done in previously merged PRs. * Result Scanner - ensure verification ordering (#22) Current implementation assumes that next() operations on primary and secondary scanners are called in the same order and uses this assumption to match results for verification. However, next()s on secondary database are called asynchronusly and their order is not defined, which causes invalid mismatch reports. This PR fixes this problem by placing data to be verified - results of next()s called on primary scanner - and details of next() call - number of requested elements - call on a queue. Each asynchronous call to next() is synchonized and pops a single element from that queue. Appropriate next is called based on number of requested elements. Then results of that request and results from the queue are verified. This ensures that results of next()s passed to verification are correctly matched and ordered. * Integration tests (#23) HBase 1.x integration tests * Add trace logging. (#24) * Estimate memory overhead in RequestResourceDescription (#25) * Tests: extract executor service to TestRule (#26) Extract executor service utilities into TestRule to facilitate code reuse in other test classes. * Integration tests: read configuration from xml files * MirroringBufferedMutator * MirroringBufferedMutator: integration tests (#9) * Fix error introduced in rebase (#11) * Obtain resource reservation before scheduling secondary calls (#4) Fixes a bug when secondary database request Future was created before obtaining resources from FlowController. * Integration tests - MirroringTable operations (#10) * MirroringAsyncConfiguration (#5) Add configuration class to be used by MirroringAsyncConnection. * SecondaryWriteErrorConsumer in MirroringTable (#15) Use SecondaryWriteErrorConsumer to handle write errors in secondary database in MirroringTable's writes. * Use Put to implement Increment and Append (#16) * refactor: extract functions using reflection into package utils.reflection * refactor: extract BatchHelpers into utils Extract common part of batch() helpers into a class and add Predicate argument to nested classes' constructors making it possible to reuse the code in 2.x client. * feat: Initial implementation of a subset of asynchronous HBase 2.x Mirroring Client Contains basic implementation of MirroringAsyncConnection and MirroringAsyncTable. * refactor: extract FlowController's request cancellation into a method * fix: Increment ITs fail with Bigtable as primary (#21) We were setting timerange on Increment objects used in integration tests without any reason and Bigtable doesn't support this operation. Setting timerange in ITs was removed. * fix: RequestScheduling should handle rejected resource reservations (#24) Custom FlowControlerStrategy implementations might, contrary to the default implementation, resolve reservation requests with exception, what we should handle by not performing the action that had to acquire the resources. * feat: Add OpenCensus tracing and metrics. (#14) * fix: make BatchHelpers skip verification of empty read results BatchHelpers provides error handling of batch() when there may be some partial results. Before the commit, matching successful reads were redundantly verified if there were none of them. This commit brings back the behaviour from up to 5a29253: when there are no successful matching reads, a MismatchDetector isn't called on empty arrays. * refactor: make MirroringAsync{Connection,Table} use SecondaryWriteErrorConsumerWithMetrics BatchHelpers require using SecondaryWriteErrorConsumerWithMetrics API. * refactor: make AsyncRequestScheduling accept CompletableFuture<ResourceReservation> instead of ResourceReservation This change is split off from commit introducing MirroringAsyncTable#batch() * feat: implement batch() in MirroringAsyncTable Implementation of MirroringAsyncTable's batch() and MirroringAsyncTable's methods such as get(List<Get) and put(List<Put>) using it. * feat: implement failed mutations log (#19) Failed secondary mutations are written to disk in JSON format, which the user can parse programmatically or inspect visually. Each failure is logged as a separate line, which makes it compatible with solutions like logstash. * refactor: split SplitBatchResponse (#40) SplitBatchResponse was refactored into two parts: splitting into reads/writes and failed/successful. This makes the code simpler and easier to maintain. * refactor: extract helper methods in tests (#48) * refactor: remove redundant writeWithControlFlow argument * feat: copy HBase operations' input lists (#57) * refactor: remove redundant field from MirroringConnection (#55) * feat: verification with sampling (#28) * fix: mirror Increment/Append in batch() using Put. (#47) * refactor: Move HBaseOperation into WriteOperationInfo (#68) * refactor: remove redundant parameter from scheduleWriteWithControlFlow (#69) * fix: integration tests - fix build (#70) * fix: count references to batch operations (#63) * fix: close underlying connections when MirroringConnection is closed (#49) * refactor: fix IDE warnings in MirroringAsyncTable test (#64) * fix: integration tests - check if write errors were reported (#71) * feat: make SecondaryWriteErrorConsumer accept error cause and operation (#65) * fix: do not call callbacks with lock held (#53) * refactor: use AccumulatedExceptions where appropriate (#54) * fix: fix key used in verification sampling ITs (#77) * feat: use faillog for handling write errors (#66) * refactor: add utilities for Futures (FutureUtils) * feat: defer closing connections until async operations complete (#37) Mirroring client schedules asynchronous operations - to mirror the mutations and to verify reads. Before this PR, closing the MirroringAsyncConnection would result in closing the underlying connections immediately. This made the pending asynchronous operations fail. This PR defers closing the underlying connections until all pending operations complete. It is achieved by reference counting the operations. * feat: implement AsyncTableBuilder (#42) * feat: implement MirroringAsyncTable#checkAndMutate (#43) * fix: Implement single Append and Increment as Put (#38) * refactor: simplify SecondaryWriteErrorConsumer API (#78) * feat: concurrent writes in MirroringTable (#79) * test: fix failing concurrent write test (#120) * refactor: renames and moves in RequestScheduling (#87) * wip: handover session comments Co-authored-by: Mateusz Walkiewicz <mwalkiewicz@unoperate.com> Co-authored-by: Adam Czajkowski <prawilny@unoperate.com>

prawilny requested review from dopiera and mwalkiewicz September 29, 2021 15:15

prawilny changed the base branch from master to ac/v2-template-pr2 September 29, 2021 15:15

dopiera requested changes Sep 29, 2021

View reviewed changes

prawilny force-pushed the ac/v2-template-pr2 branch 3 times, most recently from 2ac7e27 to 7a39dc0 Compare September 30, 2021 11:05

prawilny force-pushed the ac/refcounting branch from 39cec5c to bf4d2da Compare September 30, 2021 11:16

prawilny changed the title ~~feat: reference counting~~ feat: defer closing connections until async operations complete Sep 30, 2021

prawilny force-pushed the ac/refcounting branch from bf4d2da to 9d34524 Compare September 30, 2021 11:52

prawilny commented Sep 30, 2021

View reviewed changes

prawilny force-pushed the ac/refcounting branch from 9d34524 to 8712fb3 Compare September 30, 2021 14:12

dopiera previously approved these changes Sep 30, 2021

View reviewed changes

prawilny force-pushed the ac/v2-template-pr2 branch from ee241f0 to b1f3845 Compare September 30, 2021 14:57

mwalkiewicz reviewed Oct 1, 2021

View reviewed changes

prawilny force-pushed the ac/refcounting branch from cf6873a to e080271 Compare October 1, 2021 14:21

prawilny changed the base branch from ac/v2-template-pr2 to master October 1, 2021 14:22

dopiera requested changes Oct 4, 2021

View reviewed changes

prawilny commented Oct 5, 2021

View reviewed changes

prawilny mentioned this pull request Oct 5, 2021

refactor: utilities for Futures (FutureUtils) #56

Merged

prawilny force-pushed the ac/refcounting branch from 2ddedd8 to 24cf598 Compare October 5, 2021 16:20

prawilny changed the base branch from master to ac/future-utils/1 October 5, 2021 16:21

prawilny commented Oct 5, 2021

View reviewed changes

prawilny requested review from dopiera and mwalkiewicz October 5, 2021 16:29

prawilny force-pushed the ac/future-utils/1 branch from 909c785 to 740daaa Compare October 6, 2021 10:32

prawilny force-pushed the ac/refcounting branch from 006de90 to a2d1d80 Compare October 6, 2021 10:51

prawilny commented Oct 6, 2021

View reviewed changes

prawilny force-pushed the ac/future-utils/1 branch from 740daaa to a5c4115 Compare October 6, 2021 12:38

prawilny force-pushed the ac/future-utils/1 branch from a5c4115 to 369f9cf Compare October 8, 2021 09:41

prawilny commented Oct 8, 2021

View reviewed changes

prawilny force-pushed the ac/refcounting branch from 1def25c to 2374ee6 Compare October 8, 2021 09:41

prawilny requested a review from dopiera October 8, 2021 09:44

dopiera requested changes Oct 11, 2021

View reviewed changes

prawilny force-pushed the ac/future-utils/1 branch from 369f9cf to d5f8276 Compare October 12, 2021 15:38

prawilny force-pushed the ac/refcounting branch from 2374ee6 to ccf17d0 Compare October 12, 2021 15:38

prawilny commented Oct 12, 2021

View reviewed changes

dopiera previously approved these changes Oct 12, 2021

View reviewed changes

prawilny force-pushed the ac/future-utils/1 branch from d5f8276 to 25e9276 Compare October 13, 2021 16:09

prawilny force-pushed the ac/refcounting branch from ccf17d0 to 372fc07 Compare October 13, 2021 16:22

prawilny force-pushed the ac/future-utils/1 branch from 25e9276 to 7c4176d Compare October 13, 2021 16:33

prawilny force-pushed the ac/refcounting branch from 372fc07 to 1f6d43a Compare October 13, 2021 16:33

prawilny force-pushed the ac/future-utils/1 branch 2 times, most recently from 7b4ab2e to 68ea45b Compare October 15, 2021 09:23

prawilny force-pushed the ac/refcounting branch from 1f6d43a to 0baf926 Compare October 15, 2021 13:37

prawilny changed the base branch from ac/future-utils/1 to master October 15, 2021 13:37

prawilny force-pushed the ac/refcounting branch 3 times, most recently from 3205f32 to aa04629 Compare October 15, 2021 14:03

refcounting

a6a619f

prawilny force-pushed the ac/refcounting branch from aa04629 to a6a619f Compare October 15, 2021 14:25

mwalkiewicz approved these changes Oct 15, 2021

View reviewed changes

prawilny merged commit 6f7ab6c into master Oct 15, 2021

prawilny deleted the ac/refcounting branch November 2, 2021 14:00

feat: defer closing connections until async operations complete #37

feat: defer closing connections until async operations complete #37

Uh oh!

Conversation

prawilny commented Sep 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dopiera left a comment

Choose a reason for hiding this comment

Uh oh!

prawilny left a comment

Choose a reason for hiding this comment

Uh oh!

dopiera left a comment

Choose a reason for hiding this comment

Uh oh!

mwalkiewicz left a comment

Choose a reason for hiding this comment

Uh oh!

dopiera left a comment

Choose a reason for hiding this comment

Uh oh!

prawilny left a comment

Choose a reason for hiding this comment

Uh oh!

prawilny left a comment

Choose a reason for hiding this comment

Uh oh!

prawilny left a comment

Choose a reason for hiding this comment

Uh oh!

prawilny left a comment

Choose a reason for hiding this comment

Uh oh!

dopiera left a comment

Choose a reason for hiding this comment

Uh oh!

prawilny left a comment

Choose a reason for hiding this comment

Uh oh!

dopiera left a comment

Choose a reason for hiding this comment

Uh oh!

prawilny commented Oct 15, 2021

Uh oh!

mwalkiewicz left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

prawilny commented Sep 29, 2021 •

edited

Loading