Skip to content

Support for callback when records in hybrid log become read-only#133

Merged
badrishc merged 3 commits intomasterfrom
evict-callback
Jun 12, 2019
Merged

Support for callback when records in hybrid log become read-only#133
badrishc merged 3 commits intomasterfrom
evict-callback

Conversation

@badrishc
Copy link
Collaborator

@badrishc badrishc commented Jun 11, 2019

This PR allows users to register a user-defined callback, which is invoked when records in the hybrid log become read-only. It can be used for various reasons:

(1) to perform data write-through to an external store
(2) to perform stream processing on the change log created by the key-value store

How it Works

The FASTER log accessor now supports the IObservable<IFasterScanIterator<Key, Value>> interface. This means you can Subscribe to FASTER log, in order to receive (as push calls) updates whenever records enter the read-only region of the hybrid log:

        var s = fht.Log.Subscribe(new LogObserver());
        ...
        s.Dispose();

The parameter LogObserver is an instance of type IObserver<IFasterScanIterator<Key, Value>>. User gets push-based callbacks via the OnNext method of the observer. Each callback provides an iterator for a batch of records that have just become read-only, in bulk. The user can then perform operations such as stream processing or write-through to the remote database.

Here is an example of the callback:

        class LogObserver : IObserver<IFasterScanIterator<Key, Value>>
        {
            public void OnCompleted()
            {
            }

            public void OnError(Exception error)
            {
            }

            public void OnNext(IFasterScanIterator<Key, Value> iter)
            {
                while (iter.GetNext(out RecordInfo info, out Key key, out Value value))
                {
                    // Process key and value
                }
                iter.Dispose();
            }
        }

… be used to perform data write-through to external store.
@badrishc
Copy link
Collaborator Author

badrishc commented Jun 11, 2019

We have also added an extension method to convert an IObservable<IFasterScanIterator<Key, Value>> to an easier-to-use IObservable<Record<Key, Value>>. This converter copies the key, value, and header info from the log to the Record<Key, Value> struct, so it will not directly work with special inline variable-length structs (see #120).

Usage:

        var s = fht.Log.ToRecordObservable.Subscribe(...);
        ...
        s.Dispose();

badrishc added 2 commits June 11, 2019 10:49
…e) to convert to more easily consumable record-oriented IObservable.
@badrishc
Copy link
Collaborator Author

Currently, we support only one subscriber to the log (easy to extend in future). Also, note that the subscriber only receives new log updates from the time of subscription onwards. In other words, no I/O is associated with this operation. If you need to scan the historical part of the log, use the Scan(...) method instead (see #90).

Copy link

@peterfreiling-zz peterfreiling-zz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@badrishc badrishc merged commit e05f967 into master Jun 12, 2019
@badrishc badrishc deleted the evict-callback branch September 7, 2020 02:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants