Merged
Conversation
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Contributor
|
Hey @dgtony, looking good! I started looking though it. Will take another look this evening. Happy to see the importer / exporter functionality. That was actually on my to-do list 🎉 |
sanderpick
approved these changes
Nov 4, 2020
Contributor
sanderpick
left a comment
There was a problem hiding this comment.
Good idea! I didn't dive super deep into the dump/restore logic for all the books, but will be playing around with that functionality soon and will look closer.
Any ideas about the test failures?
| ) | ||
|
|
||
| // Finalizer collects resources for convenient cleanup. | ||
| func NewFinalizer() *Finalizer { |
common/common.go
Outdated
| return nil | ||
| } | ||
|
|
||
| type LogstoreKind string |
Signed-off-by: Anton Dort-Golts <dortgolts@gmail.com>
Contributor
Author
|
Regarding test failures. This one and some other random failures start periodically emerge on the master. Looks like a timing issue, e.g. introducing the 20 milliseconds sleep before checking test results (db/collection_test.go:931) seems to fix it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently logstore requests are skewed towards the reads, while some retrievals (notably
GetThread) are pretty expensive, requiring multiple subrequests of logstore components that goes through the series of mutexes preserving data consistency.So we implemented a new hybrid model of logstore, which is both persistent and enables fast reads. Basically it's just a write-through in-memory cache backed by a datastore for persistency. We combined two existing implementations (lstoreds + lstoremem), such that all reads goes into in-memory store only, and writes are performed in both.
Resulting hybrid model makes request processing order of magnitude faster with a tradeoff of slower writes and increased memory consumption as we're keeping all the data in memory. Given that such a setting is not universal but would be useful for higher-rank nodes, it can be enabled with an option and lstoreds remains the default one.
As a useful side effect, now we also get export/import capabilities for all the different logstore implementations. It makes implementation of components and data exchange easier, enables dumping data for analysis etc.