Writing a project that uses napari to create animation having only slice_ready will be much harder than for the current state as it will require a switch from some form of only loop to object keeping state to pass around waiting for the event. It may be nice to have fully sync mode.

@alisterburt I think that it may touch your work.

napari animation is a great call out, we'd have to make that respond to some event after rendering is done to make sure we captured the screenshot at the right time. we might need an additional event then or a force sync mode

Great point @Czaki !

I think you mentioned this verbally, but thanks for writing it down. I think it would be a useful exercise to try to see how hard it is to make napari-animation work with the current prototype in: andy-sweet#21

I'm planning to do that, but won't complain if someone else gets to it before me!

Great point @Czaki !

Note that this is getting moved to the vispy side in #4846.

Can you clarify what you mean byt this? That a LayerSliceResponse should know where it lives in the world coordinates?

Yes, I think that's right. I believe this originally came up in a discussion with @alisterburt.

The response knows where it lives in the scene (i.e. the subspace of the world that is being visualized), but slicing is a lossy operation that means the output loses the info to get back to ND world coordinate system.

I guess if the response also stores at least some parts of the request (e.g. the bounding box that defines which region which are slicing), then that might be enough because the non-visualized dimensions are the sliced dimensions (either at an exact coordinate or spanning a range like thick slices).

I think this requires some extra thought; I'm trying to imagine how this all would work out in a possible future where we have DataSources that maintain data loaded onto the GPU so that we can for example share visualizations between canvases, or between layers, or allow faster slicing in the shader. Indeed, I'm working towards this in PRs such as vispy/vispy#2350 and vispy/vispy#2351. Is there a way we can take advantage of this async machinery, while also allowing "pre-loading" data onto the GPU and do the slicing in the shader?

Slicing in the shader is super interesting. I suspect that will need to be in addition and complementary to this PR, so you could say have a 4D timeseries dataset that you sliced to get a smaller single 3D timepoint that you "pre-loaded" (as you say above) and then sliced in the GPU to get a specific 2D plane. Does it sound like we're on the same page here @brisvag?

Yup :) But I think the biggest advantage would be for multicanvas in order to display the same data multiple times using the same texture; imagine having a sizeable 3D volume on which you performed feature picking in 10 different ways, and you'd like to see them side by side. You could have a grid of 10 canvases with linked cameras and zoom in onto details and compare, but without sharing texture you'd be decuplicating the vram usage.

This might be an oversimplification, but as long as the thing that's rendering the slice (e.g. vispy) has enough to be able to do that "quickly", then I think the job of slicing as described here is done. In the case described in the comments above (i.e. 4D data, same underlying 3D slice with different features presented in multiple canvases), I'd imagine that slicing just wouldn't do much after the 3D sliced data has been read for the first time. Not quite sure what data would be in that case, which is one motivation to keep the typing of that as vague as possible for now (e.g. Any) and maybe keep things temporarily private.

I had another comment about this, but I see now you addressed it already. I think this would be a big improvement, especially because some slicing operations are CPU-intensive (Points) rather than I/O. But if it makes it easier to keep this for a later data, that's ok!

Points slicing could be seedup (by the cost of memory), outside this NAP as we use suboptimal way of storage

Yes, definitely! some thought already went into this from @kevinyamauchi :)

I think it should be the last request. Maybe a way of preventing issues is to have a temporary NotLoaded sentinel value in places where things are out of date and users might access them. Or introduce a lock so that if one tries to access anything out-of-sync, they'll have to wait.

+1 to last request

There seems to be some general consensus around last request. Some other related comments about this are that we consider storing the last response equivalent of this and other similar state, though maybe not publicly.

This would prevent us from easily re-using this logic with another backend in the future.

+1 vispy independent

This would prevent us from easily re-using this logic with another backend in the future.

Not necessarily. Most of the slicing logic proposed here is defined in the layers (in particular Layer._get_slice) and in LayerSlicer and this idea wouldn't change that. I haven't done a great job of explaining what the alternative is, so I'll probably update this text with some pseudocode and list it as an alternative considered.

I wonder if we should create a dedicated Slice object that includes all the includes all the encapsulated dims and camera info to determine what part of the data to show. Slice could then be an additional argument to a slicing function. Thoughts @jni - I know you don't want a proliferation of classes, but to me that seems like a helpful one

Yeah, I like this idea and I think @nclack has a similar one: #4892 (comment)

Would be great to fold a lot of base layer's request state into a world bounding box.

Could we not have one LayerSlice object instead of both LayerSliceRequest and LayerSliceResponse? Especially if we have encapsulated all the dims/ camera info into a single Slice object then I think all the other attributes of the two objects are the same - though I guess there could be times when they are not.

I'm trying to think from the perspective of someone adding a new layer and trying to make sure that the number of new objects added is as small as possible - ideally two, one for the full layer and then one for the slice data.

I tried exploring something like that here with the datasource idea from @d-v-b and quite liked it's feel https://github.com/napari/napari/pull/1908/files#diff-22adcfbe4884971c3fdb13eae4a24a4683d3365fb437b02df0e9f147f2c854dbR8-R29 I called it PointsData and had a "full" version of it actually live on the layer too.

Curious what @jni @tlambert03 @nclack think

Something just feels a bit off to me with having the request & response in a single object. I'd like both objects to be immutable, but I don't have a strong case for it.

Separately, what would it look like to to send LayerSliceResponse to something other than vispy visual, e.g., a web-based widget?

Separately, what would it look like to to send LayerSliceResponse to something other than vispy visual, e.g., a web-based widget?

That's a great question! Would be awesome to try as a proof of concept

Something just feels a bit off to me with having the request & response in a single object. I'd like both objects to be immutable, but I don't have a strong case for it.

That's fair - I just want to minimize burden on folks contributing new layers. I could be they just have to add one LayerDataSource object and we turn it into request and response

Definitely want to avoid any kind of class explosion, but I also don't love the idea of defining one type for two different purposes. I wonder if frozen dataclasses are less of a concern than more complex classes (i.e. with methods that have side effects). I'd also be curious to know if unpacking the request into keyword arguments for Layer.get_slice is another option that might make things easier to understand.

I'd also be curious to know if unpacking the request into keyword arguments for Layer.get_slice is another option that might make things easier to understand.

i actually liked this less, to me it became too many kwargs

I also don't love the idea of defining one type for two different purposes

ok all this is fair - we can start with two classes - one request & one response. always possible to refactor later and at least being explicit has advantage of being clear

I still like the idea of a dedicated Slice class that contains all the Camera & Dims info needed to make a slice.

I like bbox much better. point could just get encoded in the (affine) transform, and the extent information from bbox would be very valuable.

I guess the proposed design is there because that fits the current use, hence "in the future". But imo the future is now :)

I would love to collapse a few of the existing request fields into something bounding_box_world. The main reason I didn't do that yet is laziness - partly because some of the existing slicing code uses point by name and type. But I agree with you that it is probably the right time to make the switch, so I think this would be a great idea when we start writing code to be merged into napari.

One thing that makes me hesitant is around dimension correspondence. Ideally, the bounding box would have named dimensions and we could do the correspondence between its and a layer's dimensions by name. I think we should still be able to work with the numpy broadcasting correspondence we have now, but I haven't thought that through yet.