Skip to content

C++ flexible ComponentBatch, AsComponents, logging unification#3791

Merged
Wumpf merged 41 commits intomainfrom
andreas/cpp/enhanced-component-batches
Oct 11, 2023
Merged

C++ flexible ComponentBatch, AsComponents, logging unification#3791
Wumpf merged 41 commits intomainfrom
andreas/cpp/enhanced-component-batches

Conversation

@Wumpf
Copy link
Copy Markdown
Member

@Wumpf Wumpf commented Oct 11, 2023

What

This PR introduces two new concepts to our logging/serialization pipeline that make it much more extensible and make us able to avoid allocations for (simple) user types.
On top, it simplifies our logging pipelines and - despite template juggling - should lead to better error messages!

The whole thing works with two new trait classes:

  • ComponentBatch<TComponent>
    • this is used wherever we previously used std::vector<TComponent> in archetypes
    • can be constructed for every TInput for which ComponentBatchAdapater<TComponent, TInput> exists
      • the generic implementations of this trait take over std::vector/array/etc. support which actually opens more ways of feeding an archetype
      • in order to keep it simple (it's free of SFINAE!) there's some slight usability compromises, check adjustments to examples and tests to learn more. In particular some type conversions from constructor arguments of TComponent no longer work (you have to construct your component before passing now).
      • it's super easy for users to create their own non-generic adapters, check the unit test!
    • data is either owned (via an std::vector) or borrowed. Borrowed means it holds a pointer which is comes with a safety tradeoff. Behavior can be controlled explicitly (borrow(ptr, len) / take_ownership(std::vector&&)) but adapters typically decide
  • AsComponents<T>
    • similar to the trait with the same name in Rust!
    • everything that is implemented for it can be logged -> There's only log and try_log now 🎉
    • default implemented for ComponentBatch, vector/array/c-array of components and generated for all archetypes (-> serialization is no longer part of the archetype struct, but pushed to the trait impl)

Unlike typically with traits, the default of both traits IS implemented, containing a static_assert (thanks @AnkeAnke for the idea!!) which gives a (somewhat readable) error message when there's not batch adapter or component:

AsComponents failure:

struct Foo {};
stream.log("test", Foo{});

leads to:

/Users/andreas/dev/rerun-io/rerun/rerun_cpp/src/rerun/as_components.hpp:18:9: error: static_assert failed due to requirement 'NoAsComponentsFor<Foo>::value' "AsComponents is not implemented for this type. It is implemented for all built-in archetypes as well as std::vector, std::array, and c-arrays of components. You can add your own implementation by specializing AsComponents<T> for your type T."
        static_assert(
        ^
/Users/andreas/dev/rerun-io/rerun/rerun_cpp/src/rerun/recording_stream.hpp:160:25: note: in instantiation of template class 'rerun::AsComponents<Foo>' requested here
                        AsComponents<Ts>().serialize(archetypes_or_component_batches);
                        ^
/Users/andreas/dev/rerun-io/rerun/rerun_cpp/src/rerun/recording_stream.hpp:137:13: note: in instantiation of function template specialization 'rerun::RecordingStream::try_log<Foo>' requested here
            try_log(entity_path, archetypes_or_component_batches...).log_on_failure();
            ^
/Users/andreas/dev/rerun-io/rerun/rerun_cpp/tests/recording_stream.cpp:527:16: note: in instantiation of function template specialization 'rerun::RecordingStream::log<Foo>' requested here
        stream.log("test", Foo{});
               ^

ComponentBatchAdapter failure:

struct Foo {
} foo;
rerun::archetypes::Points2D test(foo);
/Users/andreas/dev/rerun-io/rerun/rerun_cpp/src/rerun/archetypes/../component_batch.hpp:32:9: error: static_assert failed due to requirement 'NoAdapterFor<rerun::components::Position2D, Foo>::value' "ComponentBatchAdapter is not implemented for this type. It is implemented for for single components as well as std::vector, std::array, and c-arrays of components. You can add your own implementation by specializing ComponentBatchAdapter<TComponent, T> for a given component and your input type T."
        static_assert(
        ^
/Users/andreas/dev/rerun-io/rerun/rerun_cpp/src/rerun/archetypes/../component_batch.hpp:95:52: note: in instantiation of template class 'rerun::ComponentBatchAdapter<rerun::components::Position2D, Foo>' requested here
        ComponentBatch(T&& input) : ComponentBatch(TAdapter<T>()(std::forward<T>(input))) {}
                                                   ^
/Users/andreas/dev/rerun-io/rerun/rerun_cpp/tests/recording_stream.cpp:528:42: note: in instantiation of function template specialization 'rerun::ComponentBatch<rerun::components::Position2D>::ComponentBatch<Foo &>' requested here
        rerun::archetypes::Points2D test(foo);
                                         ^

Future work:

  • MonoComponentBatch for mono components - right now mono components (like transform, but also like MeshProperties which has all triangle indices 😞) are not adaptable and always full-copy
  • ComponentBatch optimization for owned single components
  • ComponentBatch stride adapter
  • iterate on to_data_cell arrow serialization methods to make them fit better into the system. Since ComponentBatch should have strides etc. in the future, they will need to consume ComponentBatches which in turn need the ability to iterate.
  • example demonstrating the new extensibility

Checklist

  • I have read and agree to Contributor Guide and the Code of Conduct
  • I've included a screenshot or gif (if applicable)
  • I have tested demo.rerun.io (if applicable)
  • The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG

@Wumpf Wumpf added sdk-cpp C/C++ API specific exclude from changelog PRs with this won't show up in CHANGELOG.md labels Oct 11, 2023
Copy link
Copy Markdown
Member

@emilk emilk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks very promising! I just had a quick look so far, will dive deeper later

};
rec.log("strip", rr::LineStrips3D(points));
});
rec.log("strip", rr::LineStrips3D(linestrip));
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Am I reading this right: a single strip is passed to a constructor that can also take multiple strips?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

correct!

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this implemented with an overloaded constructor to ComponentBatch? There's so many layers to this 😓 implicit constructor calling in C++ is a bit 😬

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is relatively straight forward :)

    /// Adapter for a single component, temporary or reference.
    template <typename TComponent>
    struct ComponentBatchAdapter<TComponent, TComponent> {
        ComponentBatch<TComponent> operator()(const TComponent& one_and_only) {
            return ComponentBatch<TComponent>::borrow(&one_and_only, 1);
        }

        ComponentBatch<TComponent> operator()(TComponent&& one_and_only) {
            return ComponentBatch<TComponent>::take_ownership(std::move(one_and_only));
        }
    };

@Wumpf Wumpf mentioned this pull request Oct 11, 2023
Copy link
Copy Markdown
Member

@emilk emilk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent work!

@Wumpf Wumpf merged commit 268f140 into main Oct 11, 2023
@Wumpf Wumpf deleted the andreas/cpp/enhanced-component-batches branch October 11, 2023 16:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

exclude from changelog PRs with this won't show up in CHANGELOG.md sdk-cpp C/C++ API specific

Projects

None yet

Development

Successfully merging this pull request may close these issues.

C++ Component arrays (in Archetypes and log_components) need to be optional non-owning (& mappable)

2 participants