New data APIs 11: port all range-only views (plots, logs…)#5992
New data APIs 11: port all range-only views (plots, logs…)#5992
Conversation
9da77dc to
e8fe7e9
Compare
ad387f5 to
909038b
Compare
e8fe7e9 to
1aac73b
Compare
909038b to
b7f669d
Compare
1aac73b to
27ce535
Compare
eff4db7 to
6291c69
Compare
emilk
left a comment
There was a problem hiding this comment.
I can't help to notice that it's a lot more verbose. Obviously it does more (handling promise failures), but I wonder what we can do to reduce the boilerplate
It's much more than resolving promises, the view is now in charge of:
As I've mentioned before, I'm fairly confident that we're going to find out that the extra flexibility outweighs the extra verbosity. That being said, what makes it very hard to abstract some of the boilerplate away is the deserialization cache in particular (because of lifetimes, and mutex guards, and typed values, etc). |
b02426b to
c8150d3
Compare
6291c69 to
b35a330
Compare
Static-aware, key-less, component-based, cached range APIs.
```rust
let caches = re_query_cache2::Caches::new(&store);
// First, get the raw results for this query.
//
// They might or might not already be cached. We won't know for sure until we try to access
// each individual component's data below.
let results: CachedRangeResults = caches.range(
&store,
&query,
&entity_path.into(),
MyPoints::all_components().iter().copied(), // no generics!
);
// Then, grab the results for each individual components.
// * `get_required` returns an error if the component batch is missing
// * `get_or_empty` returns an empty set of results if the component if missing
// * `get` returns an option
//
// At this point we still don't know whether they are cached or not. That's the next step.
let all_points: &CachedRangeComponentResults = results.get_required(MyPoint::name())?;
let all_colors: &CachedRangeComponentResults = results.get_or_empty(MyColor::name());
let all_labels: &CachedRangeComponentResults = results.get_or_empty(MyLabel::name());
// Then comes the time to resolve/convert and deserialize the data.
// These steps have to be done together for efficiency reasons.
//
// That's when caching comes into play.
// If the data has already been accessed in the past, then this will just grab the
// pre-deserialized, pre-resolved/pre-converted result from the cache.
// Otherwise, this will trigger a deserialization and cache the result for next time.
let all_points = all_points.to_dense::<MyPoint>(&resolver);
let all_colors = all_colors.to_dense::<MyColor>(&resolver);
let all_labels = all_labels.to_dense::<MyLabel>(&resolver);
// The cache might not have been able to resolve and deserialize the entire dataset across all
// available timestamps.
//
// We can use the following APIs to check the status of the front and back sides of the data range.
//
// E.g. it is possible that the front-side of the range is still waiting for pending data while
// the back-side has been fully loaded.
assert!(matches!(
all_points.status(),
(PromiseResult::Ready(()), PromiseResult::Ready(()))
));
// Zip the results together using a stateful time-based join.
let all_frames = range_zip_1x2(
all_points.range_indexed(),
all_colors.range_indexed(),
all_labels.range_indexed(),
);
// Then comes the time to resolve/convert and deserialize the data, _for each timestamp_.
// These steps have to be done together for efficiency reasons.
//
// Both the resolution and deserialization steps might fail, which is why this returns a `Result<Result<T>>`.
// Use `PromiseResult::flatten` to simplify it down to a single result.
eprintln!("results:");
for ((data_time, row_id), points, colors, labels) in all_frames {
let colors = colors.unwrap_or(&[]);
let color_default_fn = || {
static DEFAULT: MyColor = MyColor(0xFF00FFFF);
&DEFAULT
};
let labels = labels.unwrap_or(&[]).iter().cloned().map(Some);
let label_default_fn = || None;
// With the data now fully resolved/converted and deserialized, the joining logic can be
// applied.
//
// In most cases this will be either a clamped zip, or no joining at all.
let results = clamped_zip_1x2(points, colors, color_default_fn, labels, label_default_fn)
.collect_vec();
eprintln!("{data_time:?} @ {row_id}:\n {results:?}");
}
```
---
Part of a PR series to completely revamp the data APIs in preparation
for the removal of instance keys and the introduction of promises:
- #5573
- #5574
- #5581
- #5605
- #5606
- #5633
- #5673
- #5679
- #5687
- #5755
- #5990
- #5992
- #5993
- #5994
- #6035
- #6036
- #6037
Builds on top of the static data PR series:
- #5534
76c92ea to
4c5f37a
Compare
Title. The new cache being natively component-based makes things much smoothier than before. --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534
b35a330 to
ddb0441
Compare
Migrate all spatial views that were using the old cache APIs to the new ones. Instance keys are not queried at all anymore. All views are now range-aware by default. Also took the opportunity to somewhat streamline everything. The 10min air-traffic example with full visible range is about 2-2.5x faster than before. I'm sure I broke a few things here and there, I'll run a full check suite once everything's said and done. --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534
`re_query_cache` is gone, `re_query_cache2` takes its place -- simple as that. --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534
Migrate every little thing that didn't use to go through the cached APIs. `Image` and `Mesh3D` are temporarily cached even though they shouldn't be, that's taken care of in a follow-up PR. Once again, I probably broke a million edge cases -- I want to get as fast as possible to removing instance keys before doing an in-depth quality pass. --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534
There is now only one way to query data: `re_query` (well you can still query the datastore directly if you're a monster, but that's for another PR). All queries go through both the query cache and the deserialization cache. There will be a follow-up PR to disable the deserialization cache for specific components. Most of this is just (re)moving stuff around except for the last two commits which take care of porting the cached test suites since they cannot depend on uncached APIs to do comparisons anymore. - Closes #6018 - Closes #3320 --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534
Make it possible to not cache some components, all while pretending really hard that they've been cached. - Related: #5974 --- Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises: - #5573 - #5574 - #5581 - #5605 - #5606 - #5633 - #5673 - #5679 - #5687 - #5755 - #5990 - #5992 - #5993 - #5994 - #6035 - #6036 - #6037 Builds on top of the static data PR series: - #5534

Text logs, line plots and scatter plots.
A bit faster than
main, with a bit less memory overhead.Part of a PR series to completely revamp the data APIs in preparation for the removal of instance keys and the introduction of promises:
ClampedZipiterator machinery #5573Component, once and for all #5605RangeZipiterator machinery #5679Builds on top of the static data PR series:
TimeInt#5534Checklist
mainbuild: app.rerun.ionightlybuild: app.rerun.io