Skip to content

MRG, ENH: Minor speedup of permutation tests#7784

Merged
larsoner merged 4 commits intomne-tools:masterfrom
yh-luo:opt
May 22, 2020
Merged

MRG, ENH: Minor speedup of permutation tests#7784
larsoner merged 4 commits intomne-tools:masterfrom
yh-luo:opt

Conversation

@yh-luo
Copy link
Copy Markdown
Contributor

@yh-luo yh-luo commented May 14, 2020

Minor refactoring to speed up permutation tests. This PR brings about 1~2% speedup (best case 5%).

Testing

Setup

import numpy as np
import time
from mne.stats import permutation_cluster_1samp_test

def large_samples_perm(n_perm=100, n_space=100, n_time=100):
    noise_level = 20
    n_time_1 = n_time
    n_time_2 = n_time
    normfactor = np.hanning(20).sum()
    rng = np.random.RandomState(42)
    condition1_1d = rng.randn(n_time_1, n_space) * noise_level
    for c in condition1_1d:
        c[:] = np.convolve(c, np.hanning(20), mode="same") / normfactor
    pseudoekp = 10 * np.hanning(round(n_space / 2))[None, :]
    condition1_1d[:, round(n_space / 2):] += pseudoekp

    thresh_tfce = dict(start=0, step=0.2)
    permutation_cluster_1samp_test(condition1_1d, threshold=thresh_tfce,
                                   n_permutations=n_perm, tail=0, seed=1,
                                   buffer_size=None)

n_perm=100, n_space=100, n_time=100

%timeit large_samples_perm()
# master: 436 ms ± 13.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# opt: 413 ms ± 2.04 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

5.2% speedup

n_perm=1000, n_space=100, n_time=100

%timeit large_samples_perm(n_perm=1000, n_space=100, n_time=100)
# master: 3.85 s ± 13.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# opt: 3.75 s ± 6.31 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

2.6% speedup

n_perm=1000, n_space=300, n_time=300

%timeit large_samples_perm(n_perm=1000, n_space=300, n_time=300)
# master: 5.83 s ± 22.1 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# opt: 5.73 s ± 31 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

1.7% speedup

n_perm=1000, n_space=1000, n_time=1000

t0 = time.time()
large_samples_perm(n_perm=1000, n_space=1000, n_time=1000)
t1 = time.time()
elapsed = t1 - t0
# master: 17.224292039871216
# opt: 16.965471029281616

1.5% speedup

Copy link
Copy Markdown
Member

@agramfort agramfort left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@larsoner ok for you?

Copy link
Copy Markdown
Member

@larsoner larsoner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Speedups are always helpful! Just a couple of minor comments

len_c = len(c)
scores[c] += h * (len_c ** e_power)
# turn sums into ndarray after running
sums = np.concatenate(sums) if sums else np.empty(0)
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sums is 1D so it's simpler just to do sums = np.array(sums) as np.array([]) is the same as np.empty(0)

Copy link
Copy Markdown
Contributor Author

@yh-luo yh-luo May 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because sums is a list of np.array, sums = np.array(sums) breaks other functions.
I would change it to sums = np.concatenate(sums) if sums else np.array([]).

@yh-luo yh-luo changed the title MRG, ENH: Minor speedup of permutation tests WIP, ENH: Minor speedup of permutation tests May 15, 2020
@yh-luo
Copy link
Copy Markdown
Contributor Author

yh-luo commented May 15, 2020

Current results

12-15% speedup

n_perm=100, n_space=100, n_time=100

370 ms ± 4.16 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
15.13% speedup

n_perm=1000, n_space=100, n_time=100

3.39 s ± 39.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
11.95% speedup

n_perm=1000, n_space=300, n_time=300

5.11 s ± 39.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
12.35% speedup

n_perm=1000, n_space=1000, n_time=1000

14.764962434768677
14.28% speedup

The performance is better than I thought 🥳

@yh-luo yh-luo changed the title WIP, ENH: Minor speedup of permutation tests MRG, ENH: Minor speedup of permutation tests May 15, 2020
@yh-luo yh-luo requested a review from larsoner May 22, 2020 00:48
@larsoner larsoner merged commit 292a478 into mne-tools:master May 22, 2020
@larsoner
Copy link
Copy Markdown
Member

Thanks @yh-luo !

@larsoner
Copy link
Copy Markdown
Member

(FYI, we are not notified of title changes, so thanks for the ping to review -- it probably would have taken me a while to notice otherwise)

@yh-luo yh-luo deleted the opt branch August 11, 2020 01:23
sharifhsn added a commit to sharifhsn/mne-python that referenced this pull request Mar 8, 2026
Add benchmark scripts and feasibility documentation for GPU-accelerating
the spatio-temporal cluster-based permutation test, which is the mne-tools#1
computational bottleneck for MNE researchers doing source-space analyses.

The connected-component labeling step in _get_components() consumes ~97%
of permutation test runtime. This adds:

- gpu_accel/benchmark_cluster_cpu.py: CPU baseline benchmark using the
  MNE sample dataset (fsaverage ico-5, ~20K vertices)
- gpu_accel/patch_cupy_poc.py: CuPy proof-of-concept that monkey-patches
  _get_components with GPU connected_components (NVIDIA)
- gpu_accel/FEASIBILITY.md: Full analysis of GPU CCL algorithms, hardware
  requirements, and a three-phase plan (CuPy PoC → wgpu+Rust → fused pipeline)
- CLAUDE.md: Development guide with uv setup instructions and GPU work context

Related: mne-tools#5439, mne-tools#12609, mne-tools#7784, mne-tools#8095, mne-tools#13175

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants