`Mesh3D` archetype by teh-cmc · Pull Request #3363 · rerun-io/rerun

teh-cmc · 2023-09-19T10:43:39Z

Implements the Mesh3D archetype:

table Mesh3D {
  vertex_positions: [rerun.components.Position3D] ("attr.rerun.component_required", order: 1000);
  vertex_normals: [rerun.components.Vector3D] ("attr.rerun.component_optional", nullable, order: 3000);
  vertex_colors: [rerun.components.Color] ("attr.rerun.component_optional", nullable, order: 3100);

  mesh_material: rerun.components.Material ("attr.rerun.component_optional", nullable, order: 3200);
  mesh_properties: rerun.components.MeshProperties ("attr.rerun.component_recommended", nullable, order: 2000);

  class_ids: [rerun.components.ClassId] ("attr.rerun.component_optional", nullable, order: 3300);
  instance_keys: [rerun.components.InstanceKey] ("attr.rerun.component_optional", nullable, order: 3400);
}

An interesting facet of the Mesh3D archetype is that it's composed of two kinds of data:

Per-vertex data, which uses our usual query system with joins and everything, allowing the mesh's components to be viewed in different ways, partially updated, etc.
Per-mesh data, which is essentially global data that applies to the mesh as a whole and is kind of a splat but not really.

Because the vertices becomes the instances, meshes cannot be batched.

There's a bit of a mess here and there because the ancestors of Mesh3D & Asset3D used to be tightly coupled.
This will be fixed in the next PR that implements the next-gen Asset3D archetype.

No dedicated roundtrip tests: API examples already cover everything.

Fixes #2788

Here's an example of partial updates:

rr.init("rerun_example_mesh3d_partial_updates", spawn=True)

vertex_positions = np.array([[-1.0, 0.0, 0.0], [1.0, 0.0, 0.0], [0.0, 1.0, 0.0]], dtype=np.float32)

# Log the initial state of our triangle
rr.set_time_sequence("frame", 0)
rr2.log(
    "triangle",
    rr2.Mesh3D(
        vertex_positions,
        vertex_normals=[0.0, 0.0, 1.0],
        vertex_colors=[[255, 0, 0], [0, 255, 0], [0, 0, 255]],
    ),
)

# Only update its vertices' positions each frame
factors = np.abs(np.sin(np.arange(1, 300, dtype=np.float32) * 0.04))
for i, factor in enumerate(factors):
    rr.set_time_sequence("frame", i)
    rr2.log_components("triangle", [rr2.cmp.Position3DArray.from_similar(vertex_positions * factor)])

23-09-19_15.45.08.patched.mp4

Checklist

I have read and agree to Contributor Guide and the Code of Conduct
I've included a screenshot or gif (if applicable)
I have tested demo.rerun.io (if applicable)

crates/re_types/definitions/rerun/archetypes/mesh3d.fbs

teh-cmc · 2023-09-19T14:16:12Z

~~Python colors are broken now that I've fixed the nullability crash... Looking...~~ fixed

emilk · 2023-09-19T14:21:25Z

crates/re_types/definitions/rerun/archetypes/mesh3d.fbs

+  /// Optional properties for the mesh as a whole (including indexed drawing).
+  //
+  // NOTE: We cannot have triangle indices here as that would break our instance key rules (either 0, 1 or N).
+  mesh_properties: rerun.components.MeshProperties ("attr.rerun.component_recommended", nullable, order: 2000);


Why not call this mesh_indices: rerun.components.MeshIndices instead?

Feels likely like we're gonna want to stash a few more things in there sooner than later (winding order, index format, ...) 🤷‍♂️

MeshTriangles maybe? "properties" is very vague. I would expect there to be color stuff in there, not fundamental stuff like indices.

Im not necessarily a fan of MeshProperties but I like MeshTriangles even less tbh.

This really has nothing to do with triangles at this point (it's now vertex_indices)

Indices are not "fundamental", quite the contrary they're optional :/

Anyway, time to ship this; feel free to change it in another PR.

teh-cmc · 2023-09-19T14:52:50Z

Added vertex colors support to raw_mesh.py while I'm here fixing colors... 🌈

teh-cmc · 2023-09-19T15:06:37Z

One thing to keep in mind: this will make large meshes (like hundreds of thousands of vertices and more) quite slower than before, since meshes are now more akin to point clouds: all vertex properties are joined etc according to their instance keys.

We could easily bypass that join (this PR actually used to do that) to regain that performance, at the cost of violating our query model and therefore user expectations.
Considering the kind of slowdown we're talking about (e.g. handful of milliseconds for ARKit scene), and the fact that #3107 would remove the join cost entirely in 99.9% use cases, I vote to keep this as-is.

For reference, arkit before:

and arkit after:

emilk

Amazing!

crates/re_types/definitions/rerun/datatypes/mesh_properties.fbs

crates/re_query/src/archetype_view.rs

crates/re_renderer/src/importer/gltf.rs

crates/re_renderer/src/importer/obj.rs

crates/re_space_view_spatial/src/mesh_loader.rs

emilk · 2023-09-20T08:11:23Z

crates/re_types/src/archetypes/mesh3d.rs

+    #[inline]
+    fn try_to_arrow(
+        &self,
+    ) -> crate::SerializationResult<
+        Vec<(::arrow2::datatypes::Field, Box<dyn ::arrow2::array::Array>)>,
+    > {


Now that we have as_component_batches, can't try_to_arrow just call self.as_component_batches and then call try_to_arrow() on each component batch? A lot less code needs generating. I guess we get an additional dyn call per component, but that seems worth it

Reimplement Archetype::try_to_arrow in terms of as_component_batches #3386

crates/re_types/src/archetypes/mesh3d_ext.rs

crates/re_types_builder/src/lib.rs

examples/rust/raw_mesh/Cargo.toml

examples/rust/raw_mesh/src/main.rs

teh-cmc · 2023-09-20T10:12:24Z

One thing to keep in mind: this will make large meshes (like hundreds of thousands of vertices and more) quite slower than before, since meshes are now more akin to point clouds: all vertex properties are joined etc according to their instance keys.

We could easily bypass that join (this PR actually used to do that) to regain that performance, at the cost of violating our query model and therefore user expectations. Considering the kind of slowdown we're talking about (e.g. handful of milliseconds for ARKit scene), and the fact that #3107 would remove the join cost entirely in 99.9% use cases, I vote to keep this as-is.

For reference, arkit before: and arkit after:

#3377 wins us back a millisecond:

The only way to do better at this point would be to not deserialize at all when we take the optimized path, instead working with raw ArrowBuffers in that case (which is effectively what the legacy view did):

Expose ArchetypeView API to deal with raw arrow buffers when join is bypassed #3379

emilk · 2023-09-20T11:41:50Z

There's almost 2ms not covered by puffin scopes

teh-cmc · 2023-09-20T16:14:35Z

There's almost 2ms not covered by puffin scopes

(Yes, this is much faster than previous screenshots... because this is a debug build (???!!!!))

It makes no sense why we're spending so much time where we're spending it but whatever, that's a problem for another day.

teh-cmc added sdk-python Python logging API sdk-rust Rust logging API 🍏 primitives Relating to Rerun primitives sdk-cpp C/C++ API specific labels Sep 19, 2023

teh-cmc force-pushed the cmc/mesh_archetype branch from a78524c to a7910bc Compare September 19, 2023 13:37

teh-cmc marked this pull request as ready for review September 19, 2023 13:50

emilk reviewed Sep 19, 2023

View reviewed changes

crates/re_types/definitions/rerun/archetypes/mesh3d.fbs Outdated Show resolved Hide resolved

emilk reviewed Sep 19, 2023

View reviewed changes

teh-cmc force-pushed the cmc/mesh_archetype branch from f43cc9f to e57b1a8 Compare September 19, 2023 16:09

emilk mentioned this pull request Sep 20, 2023

Optimize out unnecessary joins across autogenerated instance keys #3107

Closed

emilk approved these changes Sep 20, 2023

View reviewed changes

teh-cmc added 14 commits September 20, 2023 15:51

all IDL definitions for Mesh3D and related

4d137f1

Rust extensions

0a579d4

Python extensions

85acc41

C++ extensions

cdfe828

Rust unit tests

1d9150d

Python unit tests

6624f69

C++ unit tests

f1b3cb3

crosslang API examples/roundtrips

55566df

better UI for Material & MeshProperties

b6f4219

various fixes to codegen

5b16790

switch all the mesh rendering stuff to index triplets

1b1bb45

rewrite space view for new Mesh3D and prepare for Asset3D

6ea97d1

remove old RawMesh3D stuff, rewrite legacy API, update legacy examples

56542e5

FromIterator for ArrowBuffer<T>

f2e0eb2

teh-cmc added 10 commits September 20, 2023 16:03

codegen

0b49f83

demonstrate partial mesh updates

57a6559

bring back Point3D doc stuff to please link checker

17024d4

dealing with python/arrow nullability insanities

cde5d6b

lints

c427727

no need for dedicated roundtrips

6e3b0be

stale comments

0ac3b31

fix python colors

dfbf501

add support for vertex_colors to raw_mesh while we're at it

b72fe49

python lints............................................

964fd7f

This was referenced Sep 20, 2023

Reimplement Archetype::try_to_arrow in terms of as_component_batches #3386

Closed

Space views should return errors rather than logging them #3387

Open

everything, and more

ccad17e

teh-cmc force-pushed the cmc/mesh_archetype branch from 42a22b5 to ccad17e Compare September 20, 2023 16:22

missed one

3acd2db

teh-cmc merged commit f73a137 into main Sep 20, 2023

teh-cmc deleted the cmc/mesh_archetype branch September 20, 2023 16:51

Conversation

teh-cmc commented Sep 19, 2023 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Uh oh!

teh-cmc commented Sep 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emilk Sep 19, 2023

Choose a reason for hiding this comment

Uh oh!

teh-cmc Sep 19, 2023

Choose a reason for hiding this comment

Uh oh!

emilk Sep 20, 2023

Choose a reason for hiding this comment

Uh oh!

teh-cmc Sep 20, 2023

Choose a reason for hiding this comment

Uh oh!

teh-cmc commented Sep 19, 2023

Uh oh!

teh-cmc commented Sep 19, 2023

Uh oh!

emilk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emilk Sep 20, 2023

Choose a reason for hiding this comment

Uh oh!

teh-cmc Sep 20, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

teh-cmc commented Sep 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

emilk commented Sep 20, 2023

Uh oh!

teh-cmc commented Sep 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

teh-cmc commented Sep 19, 2023 •

edited by github-actions bot

Loading

teh-cmc commented Sep 19, 2023 •

edited

Loading

teh-cmc commented Sep 20, 2023 •

edited

Loading

teh-cmc commented Sep 20, 2023 •

edited

Loading