Skip to content

fix: use pulp internal timelimit, instead of stopit #2938

Merged
johanneskoester merged 19 commits intosnakemake:mainfrom
fgvieira:solver_subsample_jobs
Sep 4, 2024
Merged

fix: use pulp internal timelimit, instead of stopit #2938
johanneskoester merged 19 commits intosnakemake:mainfrom
fgvieira:solver_subsample_jobs

Conversation

@fgvieira
Copy link
Copy Markdown
Contributor

@fgvieira fgvieira commented Jul 3, 2024

There are some issues of the solver stalling (or not behaving as expected) when running very large DAGs (e.g. #871, #1003, #1374, #1620, #2354).

To minimize those issues, I was wondering if it would make sense for the solver/scheduler to evaluate just a subset (randomly sampled) of the jobs waiting to run.

QC

  • The PR contains a test case for the changes or the changes are already covered by an existing test case.
  • The documentation (docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).

Summary by CodeRabbit

  • New Features
    • Enhanced job scheduling with improved timeout handling using asynchronous programming for better performance and responsiveness.
  • Chores
    • Updated project dependencies, replacing stopit with timeout_decorator to support modern asynchronous handling.

@fgvieira fgvieira requested a review from johanneskoester as a code owner July 3, 2024 11:16
Comment on lines +265 to +270
# Subsample jobs to be run (to speedup solver)
n_total_needrun = len(needrun)
solver_max_jobs = int(os.environ.get("SNAKEMAKE_SOLVER_MAX_JOBS", 1000))
if n_total_needrun > solver_max_jobs:
import random
needrun = set(random.sample(tuple(needrun), k=solver_max_jobs))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mhm, interesting idea. We already have a 10s timeout for the ILP solver, which leads to falling back to the greedy solver that should be very fast. The stalling that people observe, does it happen because this fallback does not work, or is the greedy solver too slow as well to solve those instances?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The timeout is checked in line 602 (609 in this PR).

@fgvieira
Copy link
Copy Markdown
Contributor Author

fgvieira commented Jul 4, 2024

Some cases are quite old, but it seems to be mostly that the timeout was not triggered (and changing to greedy fixed the issue), but in some very large DAGs it seems to still persist with the greedy algorithm.

@johanneskoester
Copy link
Copy Markdown
Contributor

Some cases are quite old, but it seems to be mostly that the timeout was not triggered (and changing to greedy fixed the issue), but in some very large DAGs it seems to still persist with the greedy algorithm.

Could be that the timeout was introduced after those issues. I would vote for the following:

  1. keep the current timeout approach, for the greedy fallback though, also have a timeout which then does a subsample if 10s are exceeded.

@fgvieira
Copy link
Copy Markdown
Contributor Author

fgvieira commented Jul 11, 2024

Maybe the issue is with the timeout. I am running a workflow locally now and, even though there are no jobs running, no new jobs are being launched. If I look on top the only thing running is a cbc job (ILP solver) that has been running for 4,5 hours:

 cbc /tmp/206393e6a5dd4ee79cc331f95fc3a77a-pulp.mps -max -timeMode elapsed -branch -printingOptions all -solution /tmp/206393e6a5dd4ee79cc331f95fc3a77a-pulp.sol

If I kill it, it says:

Failed to solve scheduling problem with ILP solver in time (10s). Falling back to greedy solver.

and starts launching jobs again.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Aug 13, 2024

Walkthrough

The recent changes involve a transition in the Snakemake codebase to utilize timeout_decorator instead of stopit for managing timeouts within job scheduling. This update includes modifications to the job selection logic, specifically refactoring the _solve_ilp method to enhance its configuration and error handling. Additionally, the removal of stopit from both the main and testing dependencies streamlines the project’s requirements.

Changes

Files Change Summary
snakemake/scheduler.py Replaced stopit with timeout_decorator for timeout management. Refactored _solve_ilp method to accept additional parameters and return status, with improved exception handling.
setup.cfg Updated install_requires: removed stopit, added timeout_decorator to reflect the new dependency requirements.
test-environment.yml Modified dependencies: removed stopit, added timeout_decorator to align the testing environment with the new handling of timeouts.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

@johanneskoester
Copy link
Copy Markdown
Contributor

Maybe the issue is with the timeout. I am running a workflow locally now and, even though there are no jobs running, no new jobs are being launched. If I look on top the only thing running is a cbc job (ILP solver) that has been running for 4,5 hours:

 cbc /tmp/206393e6a5dd4ee79cc331f95fc3a77a-pulp.mps -max -timeMode elapsed -branch -printingOptions all -solution /tmp/206393e6a5dd4ee79cc331f95fc3a77a-pulp.sol

If I kill it, it says:

Failed to solve scheduling problem with ILP solver in time (10s). Falling back to greedy solver.

and starts launching jobs again.

stopit, the package we use for the timeout cannot time while the GIL is hold (https://pypi.org/project/stopit/#threading-based-resources), so that it might wait for the solving to be completed. This suprises me though because I tested this before and it worked fine.

One option to try is to use a different kind of timeout instead that can handle this without requiring the GIL to be released. Pypi lists a lot of packages.

@fgvieira
Copy link
Copy Markdown
Contributor Author

I can try using another package. I am considering either wrapt_timeout_decorator or async-timeout.

Any preference?

@fgvieira fgvieira force-pushed the solver_subsample_jobs branch from 4f6eb6c to cfea0b6 Compare August 13, 2024 12:33
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE

Commits

Files that changed from the base of the PR and between 7e2d075 and cfea0b6.

Files selected for processing (2)
  • snakemake/path_modifier.py (1 hunks)
  • snakemake/scheduler.py (2 hunks)
Additional context used
Ruff
snakemake/scheduler.py

264-264: Trailing comma missing

Add trailing comma

(COM812)


279-279: Trailing comma missing

Add trailing comma

(COM812)

Additional comments not posted (2)
snakemake/path_modifier.py (1)

Line range hint 13-13:
Consider the impact of removing debug logging.

The removal of the debug logging statement in the modify method reduces verbosity, which can improve performance. However, ensure that this does not hinder debugging efforts, especially in scenarios where path modification issues are common.

Tools
Ruff

36-36: Missing return type annotation for public function modify

(ANN201)


36-36: Missing type annotation for self in method

(ANN101)


36-36: Missing type annotation for function argument path

(ANN001)


36-36: Missing type annotation for function argument property

(ANN001)

snakemake/scheduler.py (1)

261-270: Ensure correctness of subsampling logic.

The subsampling mechanism uses random.sample() to limit the number of jobs. Verify that the environment variable SNAKEMAKE_SOLVER_MAX_JOBS is correctly set and that the subsampling logic does not inadvertently exclude critical jobs.

Tools
Ruff

264-264: Trailing comma missing

Add trailing comma

(COM812)

@fgvieira fgvieira changed the title fix: subsample jobs to speedup solver fix: use async-timeout, instead of stopit Aug 13, 2024
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE

Commits

Files that changed from the base of the PR and between cfea0b6 and f78eb10.

Files selected for processing (3)
  • setup.cfg (1 hunks)
  • snakemake/scheduler.py (3 hunks)
  • test-environment.yml (1 hunks)
Additional context used
Ruff
snakemake/scheduler.py

617-617: Local variable e is assigned to but never used

Remove assignment to unused variable e

(F841)

Additional comments not posted (4)
test-environment.yml (1)

10-10: Dependency update approved.

The replacement of stopit with async-timeout aligns with the transition to asynchronous programming. This change is appropriate for enhancing non-blocking operations.

setup.cfg (1)

58-58: Dependency update approved.

The replacement of stopit with async-timeout is consistent with the shift towards asynchronous programming. This update should enhance timeout handling.

snakemake/scheduler.py (2)

495-495: Update import statements for asynchronous timeout handling.

The import of timeout and TimeoutError from async_timeout replaces the previous stopit imports. This change is necessary for transitioning to asynchronous timeout management.


615-617: Approve asynchronous timeout handling in get_temp_sizes_gb.

The use of async with timeout(10) effectively handles timeouts in an asynchronous context, aligning with the PR's objectives.

Tools
Ruff

617-617: Local variable e is assigned to but never used

Remove assignment to unused variable e

(F841)

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE

Commits

Files that changed from the base of the PR and between f78eb10 and 1fe05b6.

Files selected for processing (1)
  • snakemake/scheduler.py (3 hunks)
Additional context used
Ruff
snakemake/scheduler.py

616-616: Missing return type annotation for private function _solve_ilp_timeout

Add return type annotation: None

(ANN202)


616-616: Missing type annotation for function argument self

(ANN001)


616-616: Missing type annotation for function argument prob

(ANN001)


616-616: Missing type annotation for function argument secs

(ANN001)


624-624: Trailing comma missing

Add trailing comma

(COM812)

Additional comments not posted (1)
snakemake/scheduler.py (1)

495-495: LGTM!

The transition to asynchronous timeout handling using async_timeout is well-implemented.

fgvieira and others added 3 commits August 13, 2024 15:24
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
@fgvieira fgvieira changed the title fix: use async-timeout, instead of stopit fix: use wrapt_timeout_decorator, instead of stopit Aug 14, 2024
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE

Commits

Files that changed from the base of the PR and between 1fe05b6 and c95281a.

Files selected for processing (3)
  • setup.cfg (1 hunks)
  • snakemake/scheduler.py (4 hunks)
  • test-environment.yml (1 hunks)
Additional context used
Path-based instructions (1)
snakemake/scheduler.py (1)

Pattern **/*.py: Do not suggest to add trailing commas.
Do not suggest type annotations for functions that are defined inside of functions or methods.

Ruff
snakemake/scheduler.py

495-495: wrapt_timeout_decorator.timeout imported but unused

Remove unused import: wrapt_timeout_decorator.timeout

(F401)


643-643: Undefined name timeout

(F821)

Additional comments not posted (3)
test-environment.yml (1)

10-10: Dependency change approved.

The replacement of stopit with wrapt_timeout_decorator aligns with the PR objectives to improve timeout management. Ensure that the new package is compatible with the rest of the dependencies.

setup.cfg (1)

58-58: Dependency change approved.

The replacement of stopit with wrapt_timeout_decorator aligns with the PR objectives to improve timeout management. Ensure that the new package is compatible with the rest of the dependencies.

snakemake/scheduler.py (1)

615-622: Exception handling approved.

The transition from TimeoutException to TimeoutError aligns with the new asynchronous timeout strategy. Ensure that all related code paths are tested.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE

Commits

Files that changed from the base of the PR and between c95281a and 38695a5.

Files selected for processing (2)
  • snakemake/scheduler.py (4 hunks)
  • test-environment.yml (2 hunks)
Additional context used
Path-based instructions (1)
snakemake/scheduler.py (1)

Pattern **/*.py: Do not suggest to add trailing commas.
Do not suggest type annotations for functions that are defined inside of functions or methods.

Additional comments not posted (4)
test-environment.yml (1)

71-71: Dependency update approved.

The replacement of stopit with wrapt_timeout_decorator aligns with the PR objectives to improve timeout management.

snakemake/scheduler.py (3)

642-643: Import change approved.

The import of wrapt_timeout_decorator is appropriate for the new asynchronous timeout management strategy.


614-618: Exception handling update approved.

The update to handle TimeoutError is consistent with the use of wrapt_timeout_decorator.


644-644: Decorator usage approved.

The @timeout(10) decorator is correctly applied to enforce a timeout on the _solve_ilp method.

@fgvieira fgvieira marked this pull request as draft August 15, 2024 15:04
@johanneskoester
Copy link
Copy Markdown
Contributor

I can try using another package. I am considering either wrapt_timeout_decorator or async-timeout.

Any preference?

wrapt seems like a good choice, as async-timeout maybe makes things unnecessarily complex because it is async, also it might have the same GIL issues.

@fgvieira
Copy link
Copy Markdown
Contributor Author

fgvieira commented Aug 22, 2024

@johanneskoester I looked a bit more into the pulp binary, and it seems it has an option to limit the time it takes to find the solution. If it cannot find one in the given time limit, it just reports a sub-optimal one.
Any reason why you decided not to use it? I think it would make for a much better solution.

@fgvieira fgvieira marked this pull request as ready for review August 22, 2024 12:34
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Outside diff range, codebase verification and nitpick comments (2)
snakemake/scheduler.py (2)

613-623: Consider parameterizing the timeout value.

The timeout value of 10 seconds is hardcoded in the call to _solve_ilp. Consider making this a configurable parameter to allow for flexibility in different scenarios.

-        status = self._solve_ilp(prob, time_limit=10)
+        status = self._solve_ilp(prob, time_limit=self.workflow.scheduling_settings.ilp_timeout)

Ensure that ilp_timeout is defined in the scheduling settings.


Line range hint 640-663: Improve handling of the PATH environment variable.

The manipulation of the PATH environment variable could be made safer and clearer by using a context manager to ensure it is restored even if an exception occurs.

from contextlib import contextmanager

@contextmanager
def temporary_path_prepend(path):
    old_path = os.environ["PATH"]
    os.environ["PATH"] = f"{path}:{old_path}"
    try:
        yield
    finally:
        os.environ["PATH"] = old_path

# Usage in _solve_ilp
with temporary_path_prepend(self.workflow.scheduling_settings.solver_path):
    solver = (
        pulp.getSolver(self.workflow.scheduling_settings.ilp_solver)
        if self.workflow.scheduling_settings.ilp_solver
        else pulp.apis.LpSolverDefault
    )
Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 8e18cc8 and 220dc92.

Files selected for processing (3)
  • setup.cfg (1 hunks)
  • snakemake/scheduler.py (5 hunks)
  • test-environment.yml (1 hunks)
Files skipped from review due to trivial changes (2)
  • setup.cfg
  • test-environment.yml
Additional context used
Path-based instructions (1)
snakemake/scheduler.py (1)

Pattern **/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of the self argument of methods.
Do not suggest type annotation of the cls argument of classmethods.
Do not suggest return type annotation if a function or method does not contain a return statement.

@sonarqubecloud
Copy link
Copy Markdown

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

Review details

Configuration used: CodeRabbit UI
Review profile: CHILL

Commits

Files that changed from the base of the PR and between 220dc92 and afbd0e6.

Files selected for processing (1)
  • snakemake/scheduler.py (6 hunks)
Files skipped from review as they are similar to previous changes (1)
  • snakemake/scheduler.py

@johanneskoester
Copy link
Copy Markdown
Contributor

@johanneskoester I looked a bit more into the pulp binary, and it seems it has an option to limit the time it takes to find the solution. If it cannot find one in the given time limit, it just reports a sub-optimal one. Any reason why you decided not to use it? I think it would make for a much better solution.

Also an interesting idea indeed. One should evaluate whether the greedy or the suboptimal solution is preferable. For now, let us merge this in order to get the fix out.

Copy link
Copy Markdown
Contributor

@johanneskoester johanneskoester left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@johanneskoester
Copy link
Copy Markdown
Contributor

@johanneskoester I looked a bit more into the pulp binary, and it seems it has an option to limit the time it takes to find the solution. If it cannot find one in the given time limit, it just reports a sub-optimal one. Any reason why you decided not to use it? I think it would make for a much better solution.

Also an interesting idea indeed. One should evaluate whether the greedy or the suboptimal solution is preferable. For now, let us merge this in order to get the fix out.

Ah, now I get what you meant. Indeed, the solution provided here is perfect already. We use the internal timeout functionality of pulp but still use greedy to solve the scheduling problem if the solution is not optimal. Nice!

@johanneskoester johanneskoester merged commit 3b64e41 into snakemake:main Sep 4, 2024
johanneskoester added a commit that referenced this pull request Sep 4, 2024
🤖 I have created a release *beep* *boop*
---


##
[8.19.1](v8.19.0...v8.19.1)
(2024-09-04)


### Bug Fixes

* fix issues with misinterpretation of max-jobs-per-timespan and
max-jobs-per-seconds
([#3067](#3067))
([d82453b](d82453b))
* pip deployment path
([#3062](#3062))
([bf9305b](bf9305b))
* return empty set if rate limiter at max
([#3060](#3060))
([4e59963](4e59963))
* use wrapt_timeout_decorator, instead of stopit
([#2938](#2938))
([3b64e41](3b64e41))
* Wrong linenumbers reported when linting
([#2985](#2985))
([3a8bd36](3a8bd36))


### Documentation

* update `doc-environment.yml` file and Documentation Setup
documentation
([#3058](#3058))
([a540a2e](a540a2e))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Johannes Köster <johannes.koester@tu-dortmund.de>
@fgvieira fgvieira deleted the solver_subsample_jobs branch September 4, 2024 12:20
@hsiehph
Copy link
Copy Markdown

hsiehph commented Sep 6, 2024

I got a question about this... is it recommended to use --scheduler greedy --scheduler-ilp-solver PULP_CBC_CMD for large workflow (>200,000 jobs) with v8.19.3? Using --scheduler greedy alone helped but it became slow again after running through 60% of the pipeline. not sure why that is the case .... Any suggestions are welcome!

@fgvieira
Copy link
Copy Markdown
Contributor Author

fgvieira commented Sep 6, 2024

With this latest patch, I'd say to not use greedy as it will try to use ILP for 10 seconds and, if it cannot find an optimal solution, switch to greedy.

@fgvieira fgvieira changed the title fix: use wrapt_timeout_decorator, instead of stopit fix: use pulp internal timelimit, instead of stopit Sep 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants