Files that changed from the base of the PR and between 2edc2f0 and 32f6277.

• snakemake/cli.py (7 hunks)
• snakemake/jobs.py (1 hunks)
• snakemake/scheduler.py (11 hunks)
• snakemake/settings/types.py (3 hunks)
• tests/testapi.py (2 hunks)
• tests/tests.py (1 hunks)

snakemake/settings/types.py (2)

57-81: Verify regular expression pattern and error handling.

The MaxJobsPerTimespan class uses a regular expression to parse input strings. Ensure that the pattern correctly matches all intended formats and that error messages are clear and informative.

Verification successful

---

Regular expression and error handling are correctly implemented.

The regular expression pattern in the MaxJobsPerTimespan class correctly matches the intended formats, and the error handling provides clear messages for invalid inputs. No issues found.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify if the regular expression in `MaxJobsPerTimespan` correctly matches intended formats.

# Test: Check if the regex matches valid formats and rejects invalid ones.
rg --type python -A 5 $'MaxJobsPerTimespan.arg_re'

Length of output: 80

---

Script:

#!/bin/bash
# Description: Verify if the regular expression in `MaxJobsPerTimespan` correctly matches intended formats.

# Test: Search for the `arg_re` pattern within the `MaxJobsPerTimespan` class definition.
rg 'MaxJobsPerTimespan' -A 10

Length of output: 4314

---

289-295: Ensure backward compatibility and consistency.

The max_jobs_per_second attribute has been changed to optional, and max_jobs_per_timespan has been introduced. Verify that these changes maintain backward compatibility and that the logic in __post_init__ is consistent with existing functionality.

Verification successful

---

Backward Compatibility and Consistency Verified

The changes to make max_jobs_per_second optional and introduce max_jobs_per_timespan maintain backward compatibility and are consistent with existing functionality. The logic in __post_init__ ensures proper initialization when max_jobs_per_second is set.

• Usage in snakemake/scheduler.py and snakemake/cli.py shows that the code handles the optional nature of max_jobs_per_second appropriately.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify backward compatibility and consistency of changes in `SchedulingSettings`.

# Test: Check for usage of `max_jobs_per_second` and ensure it is handled correctly with the new optional type.
rg --type python -A 5 $'max_jobs_per_second'

Length of output: 74

---

Script:

#!/bin/bash
# Description: Verify backward compatibility and consistency of changes in `SchedulingSettings`.

# Test: Check for usage of `max_jobs_per_second` and ensure it is handled correctly with the new optional type.
rg --type py -A 5 'max_jobs_per_second'

Length of output: 1789

snakemake/scheduler.py (1)

72-77: Verify initialization logic for job rate limiter.

Ensure that the job_rate_limiter is correctly initialized based on the scheduling settings, and verify that this logic does not introduce any unintended side effects.

Verification successful

---

Job Rate Limiter Initialization and Usage Verified

The job_rate_limiter is correctly initialized and integrated within the scheduler logic. It checks the scheduling settings and is only instantiated if the conditions are met. The limiter is used to manage job execution, ensuring the number of jobs does not exceed the specified limits. This logic appears to be implemented correctly without introducing unintended side effects.

• Initialization: The job_rate_limiter is initialized based on max_jobs_per_timespan and dryrun conditions.
• Usage: It regulates job execution by checking available slots and logging when limits are reached.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the initialization logic for `job_rate_limiter`.

# Test: Check if `job_rate_limiter` is correctly initialized and used in the scheduler.
rg --type python -A 5 $'job_rate_limiter'

Length of output: 71

---

Script:

#!/bin/bash
# Description: Verify the initialization logic for `job_rate_limiter`.

# Test: Check if `job_rate_limiter` is correctly initialized and used in the scheduler.
rg --type py -A 5 $'job_rate_limiter'

Length of output: 1638

snakemake/jobs.py (1)

108-123: Refactor improves clarity and adds _job_count.

The refactoring of _get_scheduler_resources enhances readability by separating the creation of res_dict from the assignment of _scheduler_resources. The addition of "_job_count": 1 seems intended for tracking job counts, which could be useful for resource management. Ensure this addition aligns with the overall design and resource management strategy.

snakemake/cli.py (6)

874-874: LGTM! Internal argument grouping is clear.

The --scheduler-solver-path argument has been moved to the internal group with a clear help description indicating its internal use.

---

1460-1463: LGTM! Internal argument grouping is clear.

The --local-groupid argument has been moved to the internal group with a clear help description indicating its internal use.

---

1464-1469: LGTM! Internal argument grouping is clear.

The --attempt argument has been moved to the internal group with a clear help description indicating its internal use.

---

1696-1699: LGTM! Internal argument grouping is clear.

The --scheduler-solver-path argument has been added to the internal group with a clear help description indicating its internal use.

---

1700-1708: LGTM! Internal argument grouping is clear.

The --deploy-sources argument has been added to the internal group with a clear help description indicating its internal use.

---

1339-1347: Verify the implementation of --max-jobs-per-timespan.

The new argument --max-jobs-per-timespan is introduced with a clear format specification. The --max-jobs-per-second is marked as deprecated. Ensure that all references to --max-jobs-per-second are updated in the codebase.

Verification successful

---

Deprecation of --max-jobs-per-second is correctly implemented.

The --max-jobs-per-second argument is correctly marked as deprecated in snakemake/cli.py, and its mention in CHANGELOG.md is appropriate for documentation purposes. No further updates are needed.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify that all references to `--max-jobs-per-second` are updated or removed.

# Test: Search for `--max-jobs-per-second` usage. Expect: Only occurrences in deprecated context.
rg --type python -- '--max-jobs-per-second'

Length of output: 74

---

Script:

#!/bin/bash
# Description: Verify that all references to `--max-jobs-per-second` are updated or removed.

# Test: Search for `--max-jobs-per-second` usage. Expect: Only occurrences in deprecated context.
rg '--max-jobs-per-second'

Length of output: 487

---

Script:

#!/bin/bash
# Description: Verify that all references to `--max-jobs-per-second` are updated or removed.

# Test: Search for `--max-jobs-per-second` usage. Expect: Only occurrences in deprecated context.
rg -e '--max-jobs-per-second'

Length of output: 161

Files that changed from the base of the PR and between 32f6277 and b1658eb.

• snakemake/cli.py (8 hunks)
• snakemake/scheduler.py (11 hunks)
• tests/common.py (1 hunks)
• tests/conftest.py (1 hunks)
• tests/testapi.py (2 hunks)
• tests/tests.py (7 hunks)

• tests/testapi.py

snakemake/scheduler.py (3)

809-821: Efficient use of deque for job tracking.

The JobRateLimiter class efficiently uses deque for managing job registration times. The methods are well-implemented and align with the class's purpose.

---

475-487: Effective integration of job rate limiting.

The job_selector method effectively integrates the JobRateLimiter to control job submissions. This enhances the scheduling logic without disrupting existing functionality.

---

72-77: LGTM! Verify job_rate_limiter initialization.

The integration of JobRateLimiter into the JobScheduler class is well-executed. Ensure that the job_rate_limiter is correctly initialized and used throughout the codebase.

Verification successful

---

job_rate_limiter is correctly initialized and used.

The job_rate_limiter is properly initialized and utilized within the snakemake/scheduler.py file. Its integration into the JobScheduler class is effective, and relevant methods are being called. Additionally, there is a test function test_job_rate_limiter indicating that its functionality is being tested.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the initialization and usage of `job_rate_limiter`.

# Test: Search for `job_rate_limiter` initialization and usage.
rg --type py 'job_rate_limiter'

Length of output: 368

tests/tests.py (1)

1683-1685: Ensure run and dpath are defined.

The test function test_max_jobs_per_timespan uses run and dpath, which may be undefined due to star imports. Ensure these are correctly imported or defined to avoid runtime errors.

snakemake/cli.py (5)

58-58: Verify MaxJobsPerTimespan usage.

Ensure that the MaxJobsPerTimespan type is correctly implemented and used in the argument parser.

Verification successful

---

MaxJobsPerTimespan is correctly used in the codebase.

The MaxJobsPerTimespan type is properly imported and utilized in the argument parser in snakemake/cli.py, consistent with its definition and usage elsewhere in the codebase. No issues were found.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Check where `MaxJobsPerTimespan` is used in the codebase.

# Test: Search for the usage of `MaxJobsPerTimespan` in the codebase.
ast-grep --lang python --pattern $'MaxJobsPerTimespan'

Length of output: 939

---

1695-1727: Relocation of internal arguments to group_internal.

The relocation of internal arguments is consistent, and the help text clearly indicates they are for internal use only.

Ensure that the internal use notice is clear and consistent across all relocated arguments.

Verification successful

---

All internal arguments have a clear internal use notice.

The help text for each internal argument in group_internal clearly indicates that they are for internal use only. This is consistent across all relocated arguments.

• The help text for --scheduler-solver-path, --deploy-sources, --target-jobs, and --mode all include the phrase "Internal use only."

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify that all internal arguments have a clear internal use notice.

# Test: Search for the help text of internal arguments to ensure they mention internal use.
rg --type python --context 5 'help=help_internal'

Length of output: 80

---

Script:

#!/bin/bash
# Description: Verify that all internal arguments have a clear internal use notice.

# Test: Search for the help text of internal arguments to ensure they mention internal use.
rg --type py --context 5 'help=help_internal'

Length of output: 1920

---

1695-1696: Function definition for help_internal.

The help_internal function is correctly implemented to format help text for internal arguments.

Verify that help_internal is used consistently for all internal arguments.

Verification successful

---

Usage of help_internal is consistent for internal arguments.

The help_internal function is used consistently for internal arguments throughout the snakemake/cli.py file. This confirms that the function is being utilized as intended for formatting help text for internal options.

• File: snakemake/cli.py
  • Lines: Multiple occurrences where help_internal is used to provide help text for internal arguments.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Check the usage of `help_internal` function for internal arguments.

# Test: Search for the usage of `help_internal` in the argument definitions.
rg --type python --fixed-strings 'help=help_internal'

Length of output: 84

---

Script:

#!/bin/bash
# Description: Check the usage of `help_internal` function for internal arguments.

# Test: Search for the usage of `help_internal` in the argument definitions using the correct file type for Python.
rg --type py --fixed-strings 'help=help_internal'

Length of output: 322

---

1340-1342: Ensure correct implementation of --max-jobs-per-timespan.

Verify that the MaxJobsPerTimespan.parse_choice method correctly handles the expected input format and that the help text is clear and informative.

---

1346-1348: Deprecation notice for --max-jobs-per-second.

The deprecation notice clearly informs users to use --max-jobs-per-timespan instead.

Verify that all references to --max-jobs-per-second are updated or deprecated in the codebase.

Files that changed from the base of the PR and between b1658eb and 312d959.

• snakemake/scheduler.py (11 hunks)
• snakemake/settings/types.py (3 hunks)
• tests/conftest.py (2 hunks)
• tests/tests.py (7 hunks)

• snakemake/settings/types.py
• tests/conftest.py
• tests/tests.py

snakemake/scheduler.py (7)

72-77: Verify job_rate_limiter initialization logic.

Ensure that max_jobs_per_timespan is correctly configured in the workflow settings to avoid potential issues with job rate limiting.

Verification successful

---

max_jobs_per_timespan is properly configured and integrated.

The max_jobs_per_timespan setting is defined and utilized across the codebase, including the scheduler and CLI components. Additionally, its behavior is tested, indicating proper configuration and integration.

• snakemake/scheduler.py: Utilizes max_jobs_per_timespan for job rate limiting.
• snakemake/cli.py: Configures max_jobs_per_timespan from command-line arguments.
• snakemake/settings/types.py: Defines and manages max_jobs_per_timespan.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify that `max_jobs_per_timespan` is configured in the workflow settings.

# Test: Search for `max_jobs_per_timespan` usage in the codebase. Expect: Proper configuration and usage.
rg --type python $'max_jobs_per_timespan'

Length of output: 71

---

Script:

#!/bin/bash
# Description: Verify that `max_jobs_per_timespan` is configured in the workflow settings.

# Test: Search for `max_jobs_per_timespan` usage in the codebase. Expect: Proper configuration and usage.
rg --type py $'max_jobs_per_timespan'

Length of output: 707

---

475-487: Verify integration of job rate limiter in job_selector.

Ensure that the job rate limiter correctly interacts with the job selection logic and that the number of free jobs is accurately calculated.

Verification successful

---

Integration of job rate limiter in job_selector is consistent.

The job_selector method correctly integrates with the job rate limiter, using it to control job submissions based on available slots. The method is part of a strategy pattern that allows different job selection algorithms to be used.

• snakemake/scheduler.py: The job_selector method interacts with _job_selector and the job rate limiter.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the interaction of job rate limiter with job selection logic.

# Test: Search for `job_selector` usage in the codebase. Expect: Proper interaction with job rate limiter.
rg --type python $'job_selector'

Length of output: 62

---

Script:

#!/bin/bash
# Description: Verify the interaction of job rate limiter with job selection logic.

# Test: Search for `job_selector` usage in the codebase. Expect: Proper interaction with job rate limiter.
rg --type py 'job_selector'

Length of output: 874

---

814-817: Verify job registration logic in register_jobs.

Ensure that the current time is correctly recorded for the specified number of jobs to maintain accurate job tracking.

Verification successful

---

Job registration logic in register_jobs is correctly implemented and tested.

The register_jobs method is used in the codebase and has corresponding tests in tests/testapi.py, indicating proper functionality verification.

• snakemake/scheduler.py: Method usage for job registration.
• tests/testapi.py: Multiple test cases for register_jobs.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the job registration logic in `register_jobs`.

# Test: Search for `register_jobs` usage in the codebase. Expect: Correct job tracking.
rg --type python $'register_jobs'

Length of output: 63

---

Script:

#!/bin/bash
# Description: Verify the job registration logic in `register_jobs`.

# Test: Search for `register_jobs` usage in the codebase with the correct file type. Expect: Correct job tracking.
rg --type py 'register_jobs'

Length of output: 332

---

810-813: Verify initialization logic and data structure choice in JobRateLimiter.

Ensure that the deque is the appropriate data structure for tracking job registration times, considering potential performance implications with a large number of jobs.

Verification successful

---

Initialization logic and data structure choice in JobRateLimiter are appropriate.

The deque is efficiently used for managing job registration times, supporting the intended rate-limiting functionality. The initialization logic aligns with the expected usage patterns in both the scheduler and tests.

• File: snakemake/scheduler.py
• Tests: tests/testapi.py

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the initialization logic and data structure choice in `JobRateLimiter`.

# Test: Search for `JobRateLimiter` usage in the codebase. Expect: Appropriate data structure choice.
rg --type python $'JobRateLimiter'

Length of output: 64

---

Script:

#!/bin/bash
# Description: Correctly search for `JobRateLimiter` usage in Python files.

# Test: Search for `JobRateLimiter` usage in the codebase without specifying an incorrect file type.
rg 'JobRateLimiter' --glob '*.py' -A 3

Length of output: 1547

---

742-748: Verify resource update logic in update_available_resources.

Ensure that resources are correctly decremented for each selected job to maintain accurate resource tracking.

Verification successful

---

Resource update logic in update_available_resources is consistent.

The method correctly decrements resources for jobs that are fully scheduled or selected, ensuring accurate resource tracking.

• selected_jobs and solution are both derived from conditions that ensure only fully scheduled jobs are considered.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the resource update logic in `update_available_resources`.

# Test: Search for `update_available_resources` usage in the codebase. Expect: Correct resource tracking.
rg --type python $'update_available_resources'

Length of output: 76

---

Script:

#!/bin/bash
# Description: Verify the resource update logic in `update_available_resources`.

# Test: Search for `update_available_resources` usage in the codebase. Expect: Correct resource tracking.
rg 'update_available_resources'

Length of output: 266

---

Script:

#!/bin/bash
# Description: Find definitions and modifications of `selected_jobs` and `solution` in `snakemake/scheduler.py`.

# Search for `selected_jobs` in `snakemake/scheduler.py`
rg 'selected_jobs' snakemake/scheduler.py -A 5 -B 5

# Search for `solution` in `snakemake/scheduler.py`
rg 'solution' snakemake/scheduler.py -A 5 -B 5

Length of output: 2685

---

641-641: Verify call to update_available_resources in job_selector_ilp.

Ensure that the resources are correctly updated after job selection to prevent resource contention issues.

Verification successful

---

Verified: Call to update_available_resources in job_selector_ilp functions correctly.

The method update_available_resources correctly updates the available resources by deducting the resources used by selected_jobs, ensuring proper resource management. No issues found with the implementation.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the call to `update_available_resources` in `job_selector_ilp`.

# Test: Search for `job_selector_ilp` usage in the codebase. Expect: Correct updating of resources.
rg --type python $'job_selector_ilp'

Length of output: 66

---

Script:

#!/bin/bash
# Description: Verify the call to `update_available_resources` in `job_selector_ilp`.

# Test: Search for `job_selector_ilp` usage in the codebase. Expect: Correct updating of resources.
rg --type py $'job_selector_ilp'

Length of output: 174

---

Script:

#!/bin/bash
# Description: Verify the implementation of `update_available_resources`.

# Test: Search for the definition and usage of `update_available_resources` in the codebase.
rg --type py $'def update_available_resources'

Length of output: 127

---

Script:

#!/bin/bash
# Description: Retrieve the implementation of `update_available_resources`.

# Extract the implementation of `update_available_resources` from `snakemake/scheduler.py`.
ast-grep --lang python --pattern $'def update_available_resources(self, selected_jobs):\n    $$$' snakemake/scheduler.py

Length of output: 666

---

818-824: Verify calculation logic in get_free_jobs.

Ensure that the calculation of free job slots accurately reflects the current job rate limit to prevent over-submission.

Verification successful

---

Calculation logic in get_free_jobs is verified through tests.

The test cases in tests/testapi.py confirm that the get_free_jobs method correctly calculates the number of free job slots, ensuring that the job rate limit is respected. The assertions cover various scenarios, including job registration and time-based slot availability.

• tests/testapi.py: Assertions verify expected behavior of get_free_jobs.

Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify the calculation logic in `get_free_jobs`.

# Test: Search for `get_free_jobs` usage in the codebase. Expect: Accurate calculation of free job slots.
rg --type python $'get_free_jobs'

Length of output: 63

---

Script:

#!/bin/bash
# Corrected: Verify the calculation logic in `get_free_jobs`.

# Test: Search for `get_free_jobs` usage in the codebase. Expect: Accurate calculation of free job slots.
rg --type py get_free_jobs

Length of output: 411

---

Script:

#!/bin/bash
# Extract test cases related to `get_free_jobs` from `tests/testapi.py`.

# Search for the context around `get_free_jobs` assertions in the test file.
rg --type py -A 5 'get_free_jobs' tests/testapi.py

Length of output: 395