Support for callback when records in hybrid log become read-only#133
Merged
Support for callback when records in hybrid log become read-only#133
Conversation
… be used to perform data write-through to external store.
Collaborator
Author
|
We have also added an extension method to convert an Usage: var s = fht.Log.ToRecordObservable.Subscribe(...);
...
s.Dispose(); |
…e) to convert to more easily consumable record-oriented IObservable.
Collaborator
Author
|
Currently, we support only one subscriber to the log (easy to extend in future). Also, note that the subscriber only receives new log updates from the time of subscription onwards. In other words, no I/O is associated with this operation. If you need to scan the historical part of the log, use the Scan(...) method instead (see #90). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR allows users to register a user-defined callback, which is invoked when records in the hybrid log become read-only. It can be used for various reasons:
(1) to perform data write-through to an external store
(2) to perform stream processing on the change log created by the key-value store
How it Works
The FASTER log accessor now supports the
IObservable<IFasterScanIterator<Key, Value>>interface. This means you canSubscribeto FASTER log, in order to receive (as push calls) updates whenever records enter the read-only region of the hybrid log:The parameter
LogObserveris an instance of typeIObserver<IFasterScanIterator<Key, Value>>. User gets push-based callbacks via theOnNextmethod of the observer. Each callback provides an iterator for a batch of records that have just become read-only, in bulk. The user can then perform operations such as stream processing or write-through to the remote database.Here is an example of the callback: