Skip to content

Add UI event support for Brain class#11891

Merged
larsoner merged 108 commits intomne-tools:mainfrom
wmvanvliet:brain
Sep 1, 2023
Merged

Add UI event support for Brain class#11891
larsoner merged 108 commits intomne-tools:mainfrom
wmvanvliet:brain

Conversation

@wmvanvliet
Copy link
Copy Markdown
Contributor

@wmvanvliet wmvanvliet commented Aug 17, 2023

Now that we have a UI event system, we can start work on implementing it in places where we want extra interop between figures. This PR tackles the Brain object.

UI Events implemented:

  • TimeChange (scroll through time, playback)
  • ColormapRange (when you adjust the colormap with the sliders and buttons)
  • VectexSelect (when you click on a source point)
  • CameraMove (when you move the camera) (difficult to implement, likely YAGNI)

The idea is to completely replace the Brain.callbacks dictionary, mne/viz/_brain/callback.py, and mne.viz._brain/_LInkViewer with the UI events system.

Along the way, two bugs are fixed:

  1. Glitching behavior when using the mouse to scroll backwards through time when using time interpolation. There was an incorrect int(time) conversion where it should have remained a float.
  2. The sliders didn't work on data with only very small numbers. For example, the plain MNE source estimate has values around 1E-10 and the sliders would effectively have a range of [0, 0]. This PR adds a scaling factor for the sliders to make them useful in these cases.

The event-handing codepath for updating something about the plot goes as follows:

  • Public method (e.g. Brain.set_time) publishes a UI event (TimeChange)
  • GUI element has a QT or matplotlib callback that publishes a UI event (TimeChange), possibly by calling the public method, but could also publish the UI event directly if the public function has too much overhead.
  • Brain subscribes to its own UI events and calls a private function when receiving it (Brain._on_time_change)
  • Prive function (Brain._on_time_change) updates the plot and GUI elements to match the new value.

@wmvanvliet
Copy link
Copy Markdown
Contributor Author

For the linking of camera's, here are my thoughts about it:

While it is certainly something you may want to do in certain cases, it is also something you may not want to do. Calling link(brain1, brain2) will link all UI events, including the camera. Excluding the camera is in the current version of link a pain, because you would need to specify a custom set of UI events to link. Perhaps the function signature should be changed to link(fig1, fig2, include_events=None, exclude_events=None). So one could do link(brain1, brain2, exclude_events="camera_move").

There's a performance consideration. The straightforward implementation of CameraMove is to call show_view(...). However, that function is slightly expensive due to some matrix transformations being performed. Camera movement might stutter if we implement it like this. The current _LinkViewer class solves this cleverly by assigning the same Camera object to both figures, making it super efficient. However, I don't see how that could be implemented through UI events. Perhaps a good solution is to have the CameraMove event take a pyvista Camera object as parameter and the implementation just copies over the parameters to the camera in the new figure. That would likely be efficient enough, but it breaks the semantics of UI events a little, in which I try to maintain high-level semantics, rather than such a low-level, implementation-specific object. WDYT?

@larsoner
Copy link
Copy Markdown
Member

However, that function is slightly expensive due to some matrix transformations being performed.

I'm surprised by this. I'd expect most to be 4x4 and very fast to compute or combine, and VTK should do a lot of it in c++ or in the GPU. Can you give an example bit if code that shows the slowdown? But in the end even if it's a bit slow during interactive rotation it's perhaps not necessarily a deal breaker.

@wmvanvliet
Copy link
Copy Markdown
Contributor Author

This is ready for review now!

@wmvanvliet
Copy link
Copy Markdown
Contributor Author

I haven’t benchmarked show_view yet, I’ll give it a try. But it feels inefficient seeing how many events camera movement will spawn

@larsoner
Copy link
Copy Markdown
Member

Even done in Python it seems quite fast to do some multiplies:

$ python -m timeit -s "import numpy as np; x = np.ones((4, 4))" "x @ x @ x @ x.T @ x"
50000 loops, best of 5: 4.03 usec per loop

And at the renderer level VTK should be smart enough not to render faster than 60 Hz (I think it is) and do all this stuff in C++ or (more likely) at the GPU level just at render time. So I'm still optimistic...

The first figure whose event channel will be linked to the second.
fig2 : matplotlib.figure.Figure | Figure3D
The second figure whose event channel will be linked to the first.
figs : matplotlib.figure.Figure | Figure3D
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be something like

Suggested change
figs : matplotlib.figure.Figure | Figure3D
figs : tuple of matplotlib.figure.Figure, shape (n,) | tuple of Figure3D, shape (n,)

@wmvanvliet
Copy link
Copy Markdown
Contributor Author

wmvanvliet commented Aug 31, 2023

I've looked into building a CameraMove event. In principle doable and it does seem performant. Sorry for ever doubting it :)

However, I've run into a problem: if I get the camera parameters from brain.get_view() and assign them back into the proper parameters of brain.show_view(), the camera angle has changed. Same for going one level deeper with brain._renderer.get_camera() and brain._renderer.set_camera(). It seems that whatever the set_camera() is doing is not lining up with what get_camera() returns (other than having a different order of the parameters)

@wmvanvliet
Copy link
Copy Markdown
Contributor Author

So for this PR, I'm going to leave it alone I think.

@larsoner
Copy link
Copy Markdown
Member

larsoner commented Sep 1, 2023

Works beautifully @wmvanvliet, in it goes!

@larsoner larsoner merged commit f3fac15 into mne-tools:main Sep 1, 2023
@wmvanvliet wmvanvliet deleted the brain branch September 4, 2023 07:23
snwnde pushed a commit to snwnde/mne-python that referenced this pull request Mar 20, 2024
Co-authored-by: Eric Larson <larson.eric.d@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants