ENH: Add a playback button to the notebook 3d backend by GuillaumeFavelier · Pull Request #8741 · mne-tools/mne-python

GuillaumeFavelier · 2021-01-13T10:30:15Z

This PR adds a set of ipywidgets buttons for playback. Here how it looks like:

output.mp4

Known issue

Does not depend on playback_speed but on the widget's interval value.

I also tried backgroundjobsfrom ENH: Notebook 3d backend improvements (ipyvtklink) #8704 (comment) without success so far. I'll give it a try again later this afternoon.

It's an item of #8704.

larsoner · 2021-01-13T12:21:07Z

Does not depend on playback_speed but on the widget's interval value.

It seems like you should only set "interval" to get the right refresh rate (e.g., 1/60 or 1/30), and the callback should be responsible for picking the correct time points. It seems like this should be the same for the Qt and IPywidgets backends actually -- and then as long as the callback tracks/knows 1) when the last frame was rendered (or if it's the first frame), 2) what the desired time dilation is, and 3) what the current time is (time.time()) and from this can calculate what time index/value it needs to change to, no? Such code would also adapt properly if we tried to get a framerate of 60 Hz but it renders at something slower or variable, which is nice.

GuillaumeFavelier · 2021-01-13T12:43:12Z

Oh you're right! All this code is in _play() and _advance():

mne-python/mne/viz/_brain/_brain.py

Lines 778 to 802 in fb22fa6

    
           @safe_event 
        
           def _play(self): 
        
               if self.playback: 
        
                   try: 
        
                       self._advance() 
        
                   except Exception: 
        
                       self.toggle_playback(value=False) 
        
                       raise 
        
           def _advance(self): 
        
               this_time = time.time() 
        
               delta = this_time - self._last_tick 
        
               self._last_tick = time.time() 
        
               time_data = self._data['time'] 
        
               times = np.arange(self._n_times) 
        
               time_shift = delta * self.playback_speed 
        
               max_time = np.max(time_data) 
        
               time_point = min(self._current_time + time_shift, max_time) 
        
               # always use linear here -- this does not determine the data 
        
               # interpolation mode, it just finds where we are (in time) in 
        
               # terms of the time indices 
        
               idx = np.interp(time_point, time_data, times) 
        
               self.callbacks["time"](idx, update_widget=True) 
        
               if time_point == max_time: 
        
                   self.toggle_playback(value=False)

GuillaumeFavelier · 2021-01-13T14:34:02Z

The Play widget is limited to a range of int values. I increased the number of frames depending on the playback_speed but it won't be as smooth as the qt version since it does not reuse _advance() to synchronize the frame rate. Instead, it's a fixed to self.refresh_rate_ms.

GuillaumeFavelier · 2021-01-14T14:48:49Z

It's ready to go on my end 🚀

larsoner · 2021-01-14T16:30:31Z

Works great! FWIW on my local Ubuntu-based 3.9 after pip installing the Jupyter widgets I needed to enable them:

jupyter-nbextension enable ipycanvas --py --user
jupyter nbextension enable ipyevents --py --user

Onto the minor issues:

when clicking a time point there is a quick hop back to the old one before it continues on from the old one (and this does not happen with the standard Python plotting):

Not a dealbreaker but if it's easy to fix we might as well.
Changing the playback speed slider changes the time index even when no playback is ocurring
Playback speed does not seem correct. The data is 0.25 sec so with speed == 0.1 playback should happen in 2.5 sec. Top is the HTML which goes maybe half as fast as it should, and bottom is the Qt/Python version which goes at the right speed:

I suspect this has to do with relying somehow on the refresh rate (which probably does not actually get met by this backend) as opposed to making use of time.time()-based index selection.

drammock · 2021-01-14T16:49:04Z

One nitpick from the peanut gallery: "toggle visibility" I think would be more clear if it said "toggle controls". Otherwise this is looking really nice.

GuillaumeFavelier · 2021-04-23T15:48:37Z

This PR requires quite the update. I think the challenge now changed to finding a GUI-agnostic way (Qt/Ipywidget) to playback.

larsoner · 2021-04-23T17:37:51Z

I think the challenge now changed to finding a GUI-agnostic way (Qt/Ipywidget) to playback.

Instead of making it agnostic I'd use our GUI abstraction layer to deal with the differences, and use a Qt timer and ipywidget object (Play?) as necessary in those backends

GuillaumeFavelier · 2021-04-27T11:58:49Z

The test fails because Play is stored as an action but is technically not a Button.

GuillaumeFavelier · 2021-04-27T12:00:50Z

Since I return it as a _Widget anyway, I think it's reasonable to remove it from the list of actions.

GuillaumeFavelier · 2021-04-27T12:06:24Z

And this is how it looks when integrated in the toolbar:

GuillaumeFavelier · 2021-04-27T12:14:04Z

I chose the simplest approach which is to connect the playback buttons to the time slider bidirectionally.

The known issues:

When the slider reaches the right end, the layout does not fit the dock (but I don't think it's related to this PR)

From:

To:

The timeout is fixed (interval=500) so it's not realtime like the Qt solution
The timesteps are fixed too (step=1). For this one, I tried implementing something similar to QFloatSlider and use a similar precision of 10000 but I could not manage to connect the slider in this case (or any callback actually).

GuillaumeFavelier · 2021-04-27T12:41:41Z

In my experiments, the bidirectional connection was not reliable. For example, using the playback to start, changing the slider and coming back to the playback, it will start from the old position:

I can mitigate by duplicating jsdlink():

        jsdlink((play, 'value'), (slider, 'value'))
        jsdlink((slider, 'value'), (play, 'value'))

But it brings lag when the slider is dragged (it snaps the value of the slider).

output.mp4

This reverts commit be61d64.

GuillaumeFavelier · 2021-04-30T10:49:01Z

TL;DR: The realtime approach may not be possible considering the current limitations so I think the approach based on jsdlink is good enough. What do you think @larsoner, @agramfort ?

I took the time to fix 3) and the solution explained in jupyter-widgets/ipywidgets#1775 really helped. I shared the prototype in be61d64 for reference but I ended up reverting because I am not satisfied with the overall experience. The number of calls to set_time_point increases but the ViewInteractiveWidget just cannot keep up with the updates anyway causing a visible 'lag' (and I tested locally so it might be unusable with a remote connection).

larsoner · 2021-05-03T13:00:39Z

Some ability to play back is better than none, even if it's effectively a slideshow and the speed slider doesn't really reflect an accurate time dilation. We can always work on improving it later I guess. So if you're happy with how it works now @GuillaumeFavelier let me know and I'll look, test, and merge

GuillaumeFavelier · 2021-05-03T13:06:13Z

You can have a look @larsoner, I'm ready for reviews.

larsoner

Otherwise LGTM

larsoner · 2021-05-03T13:19:18Z

mne/viz/backends/_notebook.py

            # pyvista>=0.30.0
            viewer = self.plotter.show(
-                jupyter_backend="ipyvtk_simple", return_viewer=True)
+                jupyter_backend="ipyvtklink", return_viewer=True)


Should this be a try/except as well?

Since 0.30.0 pyvista does not use ipyvtk_simple anymore

Yes but I'm thinking that it should also work for people with 0.29 if possible

I plan to bump/update those requirements in #9341

If it's only a line or two to keep it working, it would be nice not to force people to upgrade

But if they have ipyvtk_simple and pyvista < 0.30 it should work, right? For example:

kwargs = dict() if LooseVersion(pyvista.__version__) < LooseVersion('0.30'): kwargs['jupyter_backend' ] = 'ipyvtk_simple' else: kwargs['jupyter_backend'] = 'ipyvtklink' viewer = self.plotter.show(return_viewer=True, **kwargs)

it would be nice not to force people to upgrade

I understand

I'll rework this part to use LooseVersion instead.

The API changed too, the following won't work:

self.plotter.show(jupyter_backend="ipyvtk_simple", return_viewer=True)

larsoner · 2021-05-03T14:28:36Z

mne/viz/_brain/_brain.py

+            try:
+                # pyvista<0.30.0
+                self.picked_renderer = \
+                    self.plotter.iren.FindPokedRenderer(x, y)
+            except AttributeError:
+                # pyvista>=0.30.0
+                self.picked_renderer = \
+                    self.plotter.iren.interactor.FindPokedRenderer(x, y)


Like this block works for >= 0.30 and < 0.30

But this one does not depend on another package.

If the user does not have ipyvtklink but has pyvista 0.30.0, it just won't work.

That's why I think bumping is the least painful solution

larsoner · 2021-05-03T16:01:12Z

Thanks @GuillaumeFavelier !

* upstream/main: MRG, ENH: Make _get_hpi_info public (mne-tools#9369) ENH: Add a playback button to the notebook 3d backend (mne-tools#8741) better docs for permutation_cluster_test (mne-tools#9365) MRG: Add fNIRS to html output (mne-tools#9367) When plotting GFP comparison in Report, don't show sensor layout by default (mne-tools#9366) DOC: Update Mayavi troubleshooting section (mne-tools#9362) more tutorial tweaks (mne-tools#9359)

* upstream/main: FIX: make epoch cropping idempotent (mne-tools#9378) MRG, ENH: Add NIRSport support (mne-tools#9348) MRG, ENH: Make _get_hpi_info public (mne-tools#9369) ENH: Add a playback button to the notebook 3d backend (mne-tools#8741) better docs for permutation_cluster_test (mne-tools#9365) MRG: Add fNIRS to html output (mne-tools#9367) When plotting GFP comparison in Report, don't show sensor layout by default (mne-tools#9366) DOC: Update Mayavi troubleshooting section (mne-tools#9362) more tutorial tweaks (mne-tools#9359) MRG, MAINT: Use native GitHub Actions skip (mne-tools#9361) MAINT: Clean up crufty code [circle front] (mne-tools#9358) API: Complete deprecations (mne-tools#9356) Add qdarkstyle, darkdetect to environment.yml [circle full] (mne-tools#9357) FIX: Fix FIX: Add

Add Play widget

fb22fa6

GuillaumeFavelier added the ENH label Jan 13, 2021

GuillaumeFavelier self-assigned this Jan 13, 2021

GuillaumeFavelier mentioned this pull request Jan 13, 2021

ENH: Notebook 3d backend improvements (ipyvtklink) #8704

Closed

11 tasks

Depends on playback_speed and refresh_rate_ms

b4c1d4b

Improve interactions

5faa659

GuillaumeFavelier changed the title ~~WIP: Add a playback button to the notebook 3d backend~~ MRG: Add a playback button to the notebook 3d backend Jan 13, 2021

Rename to 'Toggle Controls'

af11121

GuillaumeFavelier changed the title ~~MRG: Add a playback button to the notebook 3d backend~~ WIP: Add a playback button to the notebook 3d backend Jan 15, 2021

Base automatically changed from master to main January 23, 2021 18:27

Merge branch 'main' into enh/notebook_playback

d99a10b

GuillaumeFavelier added 3 commits April 27, 2021 11:32

Revert

bda63fa

Merge branch 'main' into enh/notebook_playback

22ce2d6

Upload new version

0a2caf9

Fix

e7d2258

GuillaumeFavelier added 2 commits April 27, 2021 14:46

Mitigate

5fd78ed

Fix

4353499

GuillaumeFavelier added 5 commits April 27, 2021 15:53

Refactor

66ea733

Merge branch 'main' into enh/notebook_playback

29e395c

Fix

84269cd

Publish prototype with observe and precision

be61d64

Revert "Publish prototype with observe and precision"

1eab1dd

This reverts commit be61d64.

GuillaumeFavelier changed the title ~~WIP: Add a playback button to the notebook 3d backend~~ ENH: Add a playback button to the notebook 3d backend Apr 30, 2021

GuillaumeFavelier mentioned this pull request Apr 30, 2021

MNT: Migrate to ipyvtklink #9341

Merged

larsoner reviewed May 3, 2021

View reviewed changes

Better handle the transition

f4b92b7

larsoner merged commit 203d6fe into mne-tools:main May 3, 2021

GuillaumeFavelier deleted the enh/notebook_playback branch May 3, 2021 16:01

Uh oh!

Conversation

GuillaumeFavelier commented Jan 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

larsoner commented Jan 13, 2021

Uh oh!

GuillaumeFavelier commented Jan 13, 2021

Uh oh!

GuillaumeFavelier commented Jan 13, 2021

Uh oh!

GuillaumeFavelier commented Jan 14, 2021

Uh oh!

larsoner commented Jan 14, 2021

Uh oh!

drammock commented Jan 14, 2021

Uh oh!

GuillaumeFavelier commented Apr 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

larsoner commented Apr 23, 2021

Uh oh!

GuillaumeFavelier commented Apr 27, 2021

Uh oh!

GuillaumeFavelier commented Apr 27, 2021

Uh oh!

GuillaumeFavelier commented Apr 27, 2021

Uh oh!

GuillaumeFavelier commented Apr 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GuillaumeFavelier commented Apr 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GuillaumeFavelier commented Apr 30, 2021

Uh oh!

larsoner commented May 3, 2021

Uh oh!

GuillaumeFavelier commented May 3, 2021

Uh oh!

larsoner left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GuillaumeFavelier May 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GuillaumeFavelier May 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

larsoner commented May 3, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

GuillaumeFavelier commented Jan 13, 2021 •

edited

Loading

GuillaumeFavelier commented Apr 23, 2021 •

edited

Loading

GuillaumeFavelier commented Apr 27, 2021 •

edited

Loading

GuillaumeFavelier commented Apr 27, 2021 •

edited

Loading

GuillaumeFavelier May 3, 2021 •

edited

Loading

GuillaumeFavelier May 3, 2021 •

edited

Loading