Skip to content

Fasterlog async#180

Merged
badrishc merged 18 commits intofasterlogfrom
fasterlog-async
Oct 3, 2019
Merged

Fasterlog async#180
badrishc merged 18 commits intofasterlogfrom
fasterlog-async

Conversation

@badrishc
Copy link
Collaborator

Prototype task async version of FasterLog.

return true;
}

private TaskCompletionSource<long> commitTask = new TaskCompletionSource<long>();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use TaskCreationOptions.RunContinuationsAsynchronously to avoid potential deadlocks in consuming code?


var _commitTask = commitTask;
commitTask = new TaskCompletionSource<long>();
_commitTask.SetResult(flushAddress);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this SetResult guaranteed to happen only once since commit's are done on a single background thread?
If not, a second (accidental) call to SetResult will result in an InvalidOperationException.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since SetResult happens under a lock (per commit), we should be fine.

badrishc and others added 7 commits October 1, 2019 12:08
… commit - 1GB/sec speed with batch size of 100.
1) Allocations can handle thousands of parallel tasks
2) Removed concept of negative address - allocations are always over available pages
3) Improved scan interface to allow user memory pooling
4) Exposed commit task
5) Cleaned up sample
@badrishc badrishc merged commit b06d112 into fasterlog Oct 3, 2019
badrishc added a commit that referenced this pull request Oct 30, 2019
* Initial checkin
* Cleaned up epochs, improved fine grain scalability.
* Added commit and recovery support.
* Added TryAppend so users can implement log throttling.
* Fasterlog lowmem (#178)
Adding support for low memory footprint (4 pages)
Added support for odd-sized payloads in presence of holes in log
Fixed concurrency issue that occurs with low num of pages
Improved max throughput by eliminating a 10ms sleep in BlockAllocate
Misc cleanup of logic to track flush and close addresses in log
* Fasterlog TryAppend (#179)
Adding truly non-blocking TryAppend functionality. See sample for how this is used.
* Fasterlog async (#180)
* Added support for TryAppend. Removed List-based batch support.
* Added non-blocking TryAppend
* Added span variant
* Fix definition of SecondChanceFraction for read cache, to be 1 - MutableFraction of the log.
* Added async FlushAndCommit
* Added batched version by separating out in-memory append and wait for commit - gives better perf as the first operation is usually sync
* Tweak async sample to get back to 2GB/sec
* Other updates:
1) Allocations can handle thousands of parallel tasks
2) Removed concept of negative address - allocations are always over available pages
3) Improved scan interface to allow user memory pooling
4) Exposed commit task
5) Cleaned up sample
* Added check for entry fitting on single page
* Added batch interface (sync and async) to log append.
* Added tailing iterator WaitAsync to wait for iteration to proceed.
* Convert Span to ReadOnlySpan for appends
* Added MemoryPool/IMemoryOwner variant of iterator
* Updated way to pin pooled memory
* Update azure-pipelines.yml
* Support minimum buffer size of just 1 page!
* Actually checking in support for 1 page in memory, added initial draft of disposing task
* Added a test
* Improved sample, changed GetMemory to use byte[] instead of Span<byte>
* Update next address of iterator if GetNext fails early.
* Added random read functionality (ReadAsync) for FasterLog. Moved GetMemory to FasterLogSettings instead of Scan. Speed up TruncateUntil. Updated nuspec.
* Ensure begin addresses commit if needed, even when tail addresses do not change. Added CommittedBeginAddress metric.
* changed test project target
* Updated random read example
* Use TrySetResult instead of SetResult, since log closure moves the task to completed state.
* Added simple version/checksum to commit info.
* Added opt-in support for per-entry 8-byte checksum (xor) in header of entry.
* Fixing issue with async enqueue.
* Fixed testcase since thread abort not supported on some platforms.
* Fixing concurrency issue with contiguous partial flush requests. Removed spin-wait for adjacent flush completion.
* Fasterlog exceptions (#189)
* Added storage exception handling, connecting to tasks.
* Cleanup of error handling, control when exception is bubbled up to user.
* Added yield in NeedToWait
* Improved iterator support in case of exception
* Added async iterator support
* Added support for persistent/recoverable named iterators.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants