Add stats for time spent fetching data while searching snapshots#51866
Conversation
This commit builds on elastic#51637, adding tracking of the total time spent fetching data from the blob store and a linear regression model for these fetches.
|
Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore) |
|
I'm not sure the linear regression is actually useful here. Since we're tracking stats on a file-by-file basis, essentially every fetch will be the same size, which stops us from building a meaningful model of the fetch time as a function of size. Maybe we just want to track the total took time? Possibly min and max too? |
|
I tend to agree with you, unless we implement different range sizes per files (which we could do easily) I'm not sure the linear regression is very useful. Maybe we could reuse the existing counters and track the total/min/max took times as you suggested. |
…e-snapshots-track-time-spent-fetching-from-blob-store
DaveCTurner
left a comment
There was a problem hiding this comment.
Updated following discussions on other channels; this is ready for a proper review now.
| public Map<String, DirectoryFactory> getDirectoryFactories() { | ||
| return Map.of(SearchableSnapshotRepository.SNAPSHOT_DIRECTORY_FACTORY_KEY, | ||
| SearchableSnapshotRepository.newDirectoryFactory(repositoriesService::get, cacheService::get)); | ||
| SearchableSnapshotRepository.newDirectoryFactory(repositoriesService::get, cacheService::get, System::nanoTime)); |
There was a problem hiding this comment.
Using System::nanoTime since we need finer resolution than ThreadPool::relativeTimeInNanos offers.
This commit builds on #51637, adding tracking of the total time spent fetching
data from the blob store.
Relates #50999.