Skip to content

[MRG] Turns off memory_profiler in examples to stop circleci from hanging#16629

Merged
rth merged 25 commits intoscikit-learn:masterfrom
thomasjpfan:circleci_doc_build_timeout_fix
Mar 9, 2020
Merged

[MRG] Turns off memory_profiler in examples to stop circleci from hanging#16629
rth merged 25 commits intoscikit-learn:masterfrom
thomasjpfan:circleci_doc_build_timeout_fix

Conversation

@thomasjpfan
Copy link
Copy Markdown
Member

@thomasjpfan thomasjpfan commented Mar 4, 2020 •
edited
Loading

Reference Issues/PRs

Fixes #16628

What does this implement/fix? Explain your changes.

Update sphinx to try to get circleci to not timeout.

(Also removes azure pipeline configuration to be nice to the CI, while I test)

@thomasjpfan
Copy link
Copy Markdown
Member Author

thomasjpfan commented Mar 4, 2020

Hmm removing the memory profiler resolve the issue

@adrinjalali
Copy link
Copy Markdown
Member

adrinjalali commented Mar 4, 2020

So do we just remove the memory profiler, or also upgrade sphinx? (I'm happy with bumping up the scphinx version)

@thomasjpfan
Copy link
Copy Markdown
Member Author

thomasjpfan commented Mar 5, 2020

We may need to disable memory profiler. The gallery generation hangs for Sphinx 2.2-2.4.

@thomasjpfan thomasjpfan mentioned this pull request Mar 5, 2020
Open
@thomasjpfan thomasjpfan changed the title [NOMRG] Update sphinx to 2.3 to fix circleci timeout [NOMRG] Reduces the number of samples in plot_sparse_logistic_regression_mnist to prevent circleci from freezing Mar 6, 2020
@thomasjpfan thomasjpfan changed the title [NOMRG] Reduces the number of samples in plot_sparse_logistic_regression_mnist to prevent circleci from freezing [MRG] Turns off memory_profiler in examples to stop circleci from hanging Mar 8, 2020
@thomasjpfan
Copy link
Copy Markdown
Member Author

thomasjpfan commented Mar 8, 2020

I have tried to remove the high memory usage examples and circleci still hangs. For now, this PR now just turns off the memory_profiler.

adrinjalali
adrinjalali approved these changes Mar 9, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@adrinjalali adrinjalali left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @thomasjpfan

How can we check the memory usage of different tests/documents locally to try and fix them?

rth
rth approved these changes Mar 9, 2020
View reviewed changes
Copy link
Copy Markdown
Member

@rth rth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Thomas!

@rth rth merged commit eb3f5df into scikit-learn:master Mar 9, 2020
@rth
Copy link
Copy Markdown
Member

rth commented Mar 9, 2020

Good to see master CI finally green!

@thomasjpfan
Copy link
Copy Markdown
Member Author

thomasjpfan commented Mar 9, 2020

You can test using memory_profiler:

mprof run python examples/...

I tried completely removing the high memory examples and circleci still stalled. For reference the high memory examples are:

  1. examples/linear_model/plot_sgd_early_stopping.py
  2. examples/linear_model/plot_sparse_logistic_regression_mnist.py
  3. examples/linear_model/plot_poisson_regression_non_normal_loss.py
  4. examples/linear_model/plot_tweedie_regression_insurance_claims.py

In all cases, the original dataset loading lead to a bunch of memory.

As a follow up, I am planning to test if #16084 resolves the issue. (That PR is needs one more review to get merged)

@rth
Copy link
Copy Markdown
Member

rth commented Mar 9, 2020

Thanks for investigating. Yes, the last two examples load a somewhat large dataset and the openml loader needs improvement.

ashutosh1919 pushed a commit to ashutosh1919/scikit-learn that referenced this pull request Mar 13, 2020
@thomasjpfan @ashutosh1919
BLD Turns off memory_profiler in examples to fix CircleCI (scikit-lea…
68d1fef 
…rn#16629)
@thomasjpfan thomasjpfan mentioned this pull request Apr 5, 2020
Merged
gio8tisu pushed a commit to gio8tisu/scikit-learn that referenced this pull request May 15, 2020
@thomasjpfan @gio8tisu
BLD Turns off memory_profiler in examples to fix CircleCI (scikit-lea…
75e1936 
…rn#16629)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adrinjalali adrinjalali adrinjalali approved these changes

+1 more reviewer

@rth rth rth approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Circleci has been timing out when building docs

3 participants

@thomasjpfan @adrinjalali @rth