Skip to content

Improve performance by using numpy for trigger detection and closing matplotlib figures to avoid memleaks.#448

Merged
smoia merged 7 commits intophysiopy:masterfrom
courtois-neuromod:fix/huge_memleaks_perfs_issues
May 4, 2023
Merged

Improve performance by using numpy for trigger detection and closing matplotlib figures to avoid memleaks.#448
smoia merged 7 commits intophysiopy:masterfrom
courtois-neuromod:fix/huge_memleaks_perfs_issues

Conversation

@bpinsard
Copy link
Copy Markdown
Contributor

@bpinsard bpinsard commented May 3, 2023

cutting acq files into runs lead to very unreasonably large memory usage, and long runtime.
After memory and time profiling here are the proposed changes.
I ran the test locally and some crashed but do not seem related to my changes. I fixed one of the fail test by adding the missing requirements.

Proposed Changes

  • properly close matplotlib figures after rendering/saving to avoid clogging memory
  • replace loop-based trigger counting with a more (100-fold) efficient numpy-based one.
  • fix requirements to have some failing test passing

Change Type

  • bugfix (+0.0.1)
  • minor (+0.1.0)
  • major (+1.0.0)
  • refactoring (no version update)
  • test (no version update)
  • infrastructure (no version update)
  • documentation (no version update)
  • other

Checklist before review

  • I added everything I wanted to add to this PR.
  • [Code or tests only] I wrote/updated the necessary docstrings.
  • [Code or tests only] I ran and passed tests locally.
  • [Documentation only] I built the docs locally.
  • My contribution is harmonious with the rest of the code: I'm not introducing repetitions.
  • My code respects the adopted style, especially linting conventions.
  • The title of this PR is explanatory on its own, enough to be understood as part of a changelog.
  • I added or indicated the right labels.
  • I added information regarding the timeline of completion for this PR.
  • Please, comment on my PR while it's a draft and give me feedback on the development!

Copy link
Copy Markdown
Member

@smoia smoia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @bpinsard, thank you for this PR!
There's a couple of comments scattered around, but it's quick stuff!

@smoia
Copy link
Copy Markdown
Member

smoia commented May 3, 2023

All of the tests passed remotely (although I'm missing pre-commit, so let me check that), so I'm fairly sure we're good to go once the small changes have been incorporated!

@smoia smoia added the Minormod This PR generally closes an `Enhancement` issue. It increments the minor version (0.+1.0) label May 3, 2023
@smoia smoia changed the title Fix/huge memleaks perfs issues Improve performance by ... May 3, 2023
@smoia
Copy link
Copy Markdown
Member

smoia commented May 3, 2023

pre-commit found ;)
One last thing before merging: we care a lot about PR titles, since they are automatically used to compile the changelog.

Can you edit the title to make it more self-explanatory (yet concise)?
I started a hint, but feel free to change it completely!

bpinsard and others added 2 commits May 3, 2023 21:19
Co-authored-by: Stefano Moia <s.moia.research@gmail.com>
Co-authored-by: Stefano Moia <s.moia.research@gmail.com>
@bpinsard bpinsard changed the title Improve performance by ... Improve performance by using numpy for trigger detection and closing matplotlib figures to avoid memleaks. May 4, 2023
@bpinsard
Copy link
Copy Markdown
Contributor Author

bpinsard commented May 4, 2023

Added testing for the memory issue.

@bpinsard bpinsard requested a review from smoia May 4, 2023 13:05
Copy link
Copy Markdown
Member

@smoia smoia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Thank you @bpinsard !

@smoia smoia merged commit 0651a2a into physiopy:master May 4, 2023
@smoia
Copy link
Copy Markdown
Member

smoia commented May 4, 2023

🚀 PR was released in 2.10.0 🚀

@smoia smoia added the released This issue/pull request has been released. label May 4, 2023
@bpinsard
Copy link
Copy Markdown
Contributor Author

bpinsard commented May 4, 2023

Thanks for your fast processing of the PR!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Minormod This PR generally closes an `Enhancement` issue. It increments the minor version (0.+1.0) released This issue/pull request has been released.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants