MRG: Brain silhouette by GuillaumeFavelier · Pull Request #8771 · mne-tools/mne-python

GuillaumeFavelier · 2021-01-22T10:17:20Z

This PR follows #8749 (comment) and adds a silhouette parameter to Brain.__init__().

master	PR

What would be even more helpful for this kind of mesh is ambient occlusion.

GuillaumeFavelier · 2021-01-22T10:24:42Z

For stc_vec:

GuillaumeFavelier · 2021-01-22T11:14:47Z

Result looks good with SSAO on stc_surf:

PR	PR + ssao

Code snippet

    def _enable_ssao(self, dataset):
        bounds = np.asarray(dataset.GetBounds())
        basic_pass = vtk.vtkRenderStepsPass()
        scene_size = np.linalg.norm(bounds[:3] - bounds[3:])
        ssao = vtk.vtkSSAOPass()
        ssao.SetRadius(0.1 * scene_size)  # comparison radius
        ssao.SetBias(0.001 * scene_size)  # comparison bias
        ssao.SetKernelSize(128)  # number of samples used
        ssao.BlurOff()  # do not blur occlusion
        ssao.SetDelegatePass(basic_pass)
        self.plotter.renderer.SetPass(ssao)

But I ran into visual artifacts on stc_vector:

Reference: https://blog.kitware.com/ssao/

larsoner · 2021-01-22T12:12:12Z

Is it the combination of depth peeling + SSAO?

GuillaumeFavelier · 2021-01-22T13:13:02Z

Is it the combination of depth peeling + SSAO?

I think there is a depth peeling step in vtkRenderStepsPass by default

larsoner · 2021-01-22T13:14:27Z

So maybe you need to disable depth_peeling when using SSO? In other words, maybe you have two depth peelings going on and they don't get along...

larsoner · 2021-01-22T13:15:07Z

(i.e., don't call brain.enable_depth_peeling() / ensure it's off at the PyVista level however you need to do it, and let SSO do the depth peeling it needs to render)

GuillaumeFavelier · 2021-01-22T14:13:29Z

Depth peeling is disabled by default in the _pyvista backend. I confirm that I still have the visual artifacts when the basic vtkRenderStepsPass is used and SSAO disabled. So the bug is related either to the vtkRenderStepsPass, my usage of it or my config. Do you reproduce those?

GuillaumeFavelier · 2021-01-22T14:27:51Z

Avoiding vtkRenderStepsPass altogether and re-creating a full vtkSequencePass as described in discourse.vtk.org fixed it for me!

GuillaumeFavelier · 2021-01-22T14:42:15Z

After f722e74 :

x	master	PR
surf
vec
vol

larsoner · 2021-01-22T14:56:01Z

Wow, that looks great! It makes all the renderings look much better in a realistic sense.

In some cases it makes them less usable, such as the volumetric case -- but I think this is mostly because the brain opacity is too high in that case. Can you see what something like this example looks like? On master it's

And locally on your PR I see:

GuillaumeFavelier · 2021-01-22T15:04:47Z

Oh yeah I reproduce this on multi-renderers as well. It's trivial to solve the silhouette but SSAO gives this weird empty plot 🤔

GuillaumeFavelier · 2021-01-22T15:37:52Z

I have the same behaviour with the standard Plotter on a minimal example. My guess is that it is a limitation of the multi-renderer design of PyVista/VTK since those apperead with EDL too. Maybe better isolation like a multi-plotter system could solve this 🤔

MWE

import numpy as np
import vtk
import pyvista as pv


def _enable_ssao(plotter, dataset=None, idx=None):
    lightsP = vtk.vtkLightsPass()
    opaqueP = vtk.vtkOpaquePass()
    translucentP = vtk.vtkTranslucentPass()
    volumeP = vtk.vtkVolumetricPass()

    collection = vtk.vtkRenderPassCollection()
    collection.AddItem(lightsP)

    # opaque passes
    ssaoCamP = vtk.vtkCameraPass()
    ssaoCamP.SetDelegatePass(opaqueP)

    if dataset is None:
        ssaoP = vtk.vtkSSAOPass()
        ssaoP.SetRadius(30)
        ssaoP.SetBias(0.1)
        ssaoP.SetKernelSize(32)
        ssaoP.SetBlur(True)
    else:
        bounds = np.asarray(dataset.GetBounds())
        scene_size = np.linalg.norm(bounds[:3] - bounds[3:])
        ssaoP = vtk.vtkSSAOPass()
        ssaoP.SetRadius(0.1 * scene_size)  # comparison radius
        ssaoP.SetBias(0.001 * scene_size)  # comparison bias
        ssaoP.SetKernelSize(128)  # number of samples used
        ssaoP.BlurOff()  # do not blur occlusion
    ssaoP.SetDelegatePass(ssaoCamP)
    collection.AddItem(ssaoP)

    # translucent and volumic passes
    ddpP = vtk.vtkDualDepthPeelingPass()
    ddpP.SetTranslucentPass(translucentP)
    ddpP.SetVolumetricPass(volumeP)
    collection.AddItem(ddpP)

    # finally overlays
    overP = vtk.vtkOverlayPass()
    collection.AddItem(overP)

    sequence = vtk.vtkSequencePass()
    sequence.SetPasses(collection)

    camP = vtk.vtkCameraPass()
    camP.SetDelegatePass(sequence)

    if idx is None:
        plotter.renderer.SetPass(camP)
    else:
        plotter.renderers[idx].SetPass(camP)


p = pv.Plotter(shape=(2, 1))
p.subplot(0, 0)
p.add_mesh(pv.Sphere())
_enable_ssao(p)
p.subplot(1, 0)
p.add_mesh(pv.Cone())
_enable_ssao(p)
p.show()

GuillaumeFavelier · 2021-01-22T17:16:30Z

I confirm that better isolation fixes this. I built a custom Qt Plotter that contains separated instances of QtInteractor and now SSAO works:

Code snippet

import numpy as np
import vtk
import pyvista as pv
import pyvistaqt as pvqt


def _enable_ssao(plotter, dataset=None, idx=None):
    lightsP = vtk.vtkLightsPass()
    opaqueP = vtk.vtkOpaquePass()
    translucentP = vtk.vtkTranslucentPass()
    volumeP = vtk.vtkVolumetricPass()

    collection = vtk.vtkRenderPassCollection()
    collection.AddItem(lightsP)

    # opaque passes
    ssaoCamP = vtk.vtkCameraPass()
    ssaoCamP.SetDelegatePass(opaqueP)

    if dataset is None:
        ssaoP = vtk.vtkSSAOPass()
        ssaoP.SetRadius(30)
        ssaoP.SetBias(0.1)
        ssaoP.SetKernelSize(32)
        ssaoP.SetBlur(True)
    else:
        bounds = np.asarray(dataset.GetBounds())
        scene_size = np.linalg.norm(bounds[:3] - bounds[3:])
        ssaoP = vtk.vtkSSAOPass()
        ssaoP.SetRadius(0.1 * scene_size)  # comparison radius
        ssaoP.SetBias(0.001 * scene_size)  # comparison bias
        ssaoP.SetKernelSize(128)  # number of samples used
        ssaoP.BlurOff()  # do not blur occlusion
    ssaoP.SetDelegatePass(ssaoCamP)
    collection.AddItem(ssaoP)

    # translucent and volumic passes
    ddpP = vtk.vtkDualDepthPeelingPass()
    ddpP.SetTranslucentPass(translucentP)
    ddpP.SetVolumetricPass(volumeP)
    collection.AddItem(ddpP)

    # finally overlays
    overP = vtk.vtkOverlayPass()
    collection.AddItem(overP)

    sequence = vtk.vtkSequencePass()
    sequence.SetPasses(collection)

    camP = vtk.vtkCameraPass()
    camP.SetDelegatePass(sequence)

    if idx is None:
        plotter.renderer.SetPass(camP)
    else:
        plotter.renderers[idx].SetPass(camP)


p = pvqt.Plotter(shape=(2, 2))
p.interactors[0].add_mesh(pv.Sphere())
_enable_ssao(p.interactors[0])
p.interactors[1].add_mesh(pv.Cone())
_enable_ssao(p.interactors[1])
p.interactors[2].add_mesh(pv.Box())
_enable_ssao(p.interactors[2])
p.interactors[3].add_mesh(pv.Cylinder())
_enable_ssao(p.interactors[3])
p.show()

GuillaumeFavelier · 2021-01-26T14:22:31Z

I think this can move forward without SSAO. It makes it less impactful/realistic though.

SSAO will need MultiPlotter and this implies heavy refactoring of _pyvista and Brain (plus some work on the pyvistaqt side).

larsoner · 2021-01-26T14:26:37Z

Agreed, let's move forward with silhouette. SSAO is a separable issue.

Can you modify plot_visualize_stc, plot_beamformer_lcmv.py, and plot_mixed_source_space_inverse.py which should all benefit in principle from the silhouette? That way we can compare main and this PR

mne/viz/_3d.py

This reverts commit 5b5e9a8.

larsoner

These all look good to me / better than master:

Other than some minor gripes LGTM +1 for merge

larsoner · 2021-01-27T12:52:51Z

mne/utils/docs.py

 """
+docdict['brain_kwargs'] = """
+brain_kwargs : dict | None
+    Additional arguments to brain.__init__ (e.g.,


Suggested change

Additional arguments to brain.__init__ (e.g.,

Additional arguments to the :class:`mne.viz.Brain` constructor (e.g.,

mne/viz/_brain/_brain.py

larsoner · 2021-01-27T12:57:56Z

And don't forget latest.inc update. Maybe start a bullet-point list for mne.viz.Brain enhancements since I have a feeling there will be a few we will be able to group together for this release?

Add silhouette parameter

5c2dc77

GuillaumeFavelier added the VIZ label Jan 22, 2021

GuillaumeFavelier self-assigned this Jan 22, 2021

GuillaumeFavelier mentioned this pull request Jan 22, 2021

WIP: Prototype of comparative views #8749

Closed

1 task

Fix docstring

b53f7b9

Update alpha

6237905

GuillaumeFavelier added 2 commits January 22, 2021 15:09

Fix silhouette check

bc0a25a

TST: SSAO bug

5860302

Fix ssao

f722e74

GuillaumeFavelier mentioned this pull request Jan 22, 2021

ENH: new MultiPlotter class pyvista/pyvistaqt#78

Merged

Base automatically changed from master to main January 23, 2021 18:27

GuillaumeFavelier added 2 commits January 26, 2021 15:39

Remove SSAO

66e16c4

Touch examples

5b5e9a8

larsoner reviewed Jan 26, 2021

View reviewed changes

mne/viz/_3d.py Outdated Show resolved Hide resolved

Add brain_kwargs

7a6a3e9

GuillaumeFavelier added 5 commits January 26, 2021 15:57

Revert "Touch examples"

5b512e2

This reverts commit 5b5e9a8.

Add brain_kwargs

d54d1f6

Touch examples

dbbd858

Adjust decimation and fix silhouette

3c59f14

Improve coverage

fefa567

GuillaumeFavelier changed the title ~~WIP, ENH: Brain silhouette~~ ENH: Brain silhouette Jan 26, 2021

GuillaumeFavelier marked this pull request as ready for review January 26, 2021 16:41

Improve coverage

9494eaa

GuillaumeFavelier changed the title ~~ENH: Brain silhouette~~ MRG: Brain silhouette Jan 27, 2021

Improve coverage

0eecf65

larsoner reviewed Jan 27, 2021

View reviewed changes

GuillaumeFavelier mentioned this pull request Jan 27, 2021

ENH: New interface for Brain #7162

Closed

86 tasks

GuillaumeFavelier added 2 commits January 27, 2021 14:19

Commit suggestions

ff4e1ea

Merge branch 'main' into enh/brain_silhouette

4921ef6

larsoner approved these changes Jan 27, 2021

View reviewed changes

GuillaumeFavelier merged commit fa779fc into mne-tools:main Jan 27, 2021

GuillaumeFavelier deleted the enh/brain_silhouette branch January 27, 2021 14:37

GuillaumeFavelier mentioned this pull request Mar 5, 2021

WIP: Prototype of MultiPlotter #8997

Closed

10 tasks

	Additional arguments to brain.__init__ (e.g.,
	Additional arguments to the :class:`mne.viz.Brain` constructor (e.g.,

Uh oh!

Conversation

GuillaumeFavelier commented Jan 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GuillaumeFavelier commented Jan 22, 2021

Uh oh!

GuillaumeFavelier commented Jan 22, 2021

Uh oh!

larsoner commented Jan 22, 2021

Uh oh!

GuillaumeFavelier commented Jan 22, 2021

Uh oh!

larsoner commented Jan 22, 2021

Uh oh!

larsoner commented Jan 22, 2021

Uh oh!

GuillaumeFavelier commented Jan 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GuillaumeFavelier commented Jan 22, 2021

Uh oh!

GuillaumeFavelier commented Jan 22, 2021

Uh oh!

larsoner commented Jan 22, 2021

Uh oh!

GuillaumeFavelier commented Jan 22, 2021

Uh oh!

GuillaumeFavelier commented Jan 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GuillaumeFavelier commented Jan 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GuillaumeFavelier commented Jan 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

larsoner commented Jan 26, 2021

Uh oh!

Uh oh!

larsoner left a comment

Choose a reason for hiding this comment

Uh oh!

larsoner Jan 27, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

larsoner commented Jan 27, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

GuillaumeFavelier commented Jan 22, 2021 •

edited

Loading

GuillaumeFavelier commented Jan 22, 2021 •

edited

Loading

GuillaumeFavelier commented Jan 22, 2021 •

edited

Loading

GuillaumeFavelier commented Jan 22, 2021 •

edited

Loading

GuillaumeFavelier commented Jan 26, 2021 •

edited

Loading