refactor: Minor Logging Improvements#3604
Conversation
📝 WalkthroughWalkthroughThe changes refine logging behavior by adding conditional checks to suppress temporary output removal messages when quietness settings include rule-related suppression. The logging handler is updated to skip emitting both "None" and empty string messages. Scheduler progress printing logic is adjusted for finer control based on specific quietness flags. No public API changes are introduced. Changes
Sequence Diagram(s)sequenceDiagram
participant DAG
participant Logger
alt Quiet mode does not suppress RULES logs
DAG->>Logger: Log "Would remove temporary output ..." with extra quietness=RULES
alt Not dry-run
DAG->>Logger: Log "Removing temporary output ..." with extra quietness=RULES
end
else
DAG-->>Logger: (No log output)
end
sequenceDiagram
participant ColorizingTextHandler
participant OutputStream
ColorizingTextHandler->>ColorizingTextHandler: Format log message
alt Message is "None" or ""
ColorizingTextHandler-->>OutputStream: (No output)
else
ColorizingTextHandler->>OutputStream: Write formatted message
end
sequenceDiagram
participant Scheduler
participant Output
alt Not dry-run and (not quiet or quiet excludes PROGRESS)
Scheduler->>Output: Print progress update
else
Scheduler-->>Output: (No progress output)
end
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
|
I took a quick look at the blank lines. It seems they are actually coming from the shellcmd log call. Shellcmd is mapped to quietness.progress, not rules: snakemake/src/snakemake/logging.py Lines 390 to 401 in 86733ee This should be: quietness_map = {
LogEvent.JOB_INFO: Quietness.RULES,
LogEvent.GROUP_INFO: Quietness.RULES,
LogEvent.JOB_ERROR: Quietness.RULES,
LogEvent.GROUP_ERROR: Quietness.RULES,
LogEvent.PROGRESS: Quietness.PROGRESS,
- LogEvent.SHELLCMD: Quietness.PROGRESS,
+ LogEvent.SHELLCMD: Quietness.RULES,
LogEvent.JOB_FINISHED: Quietness.PROGRESS,
LogEvent.RESOURCES_INFO: Quietness.PROGRESS,
LogEvent.RUN_INFO: Quietness.PROGRESS,
} |
|
I guess checking for an empty is not really a direct solution to the issue i observed then. Would it make sense to still check for the log message being empty? EDIT: |
|
This change would in some way also partially solve #2882 since progress would be properly handled |
Yes! I think that would be a great solution. We should just filter the message properly. But I don't think we should add a new quietness level since its already confusing as is with the issue you referenced. I think we should just do something like: class DefaultFilter:
def __init__(self, quiet, debug_dag, dryrun, printshellcmds) -> None:
if quiet is None:
quiet = set()
self.quiet = quiet
self.debug_dag = debug_dag
self.dryrun = dryrun
self.printshellcmds = printshellcmds
def filter(self, record):
from snakemake.settings.enums import Quietness
event, level = get_event_level(record)
...
# handle shell cmd
if event == LogEvent.SHELLCMD and not self.printshellcmds:
return False |
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (1)
src/snakemake/logging.py (1)
509-509: Good defensive check, but verify the formatting condition logic.The check for both
"None"and empty string provides good defensive programming. However, the space in the conditionformatted_message==""lacks proper spacing.Apply this diff to fix the formatting:
- if formatted_message == "None" or formatted_message=="": + if formatted_message == "None" or formatted_message == "":Additionally, consider using a more comprehensive check for falsy values:
- if formatted_message == "None" or formatted_message == "": + if not formatted_message or formatted_message == "None":
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
src/snakemake/logging.py(5 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest type annotations for functions that are defined inside of functions or methods. Do not suggest type annotation of the `s...
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
src/snakemake/logging.py
🧬 Code Graph Analysis (1)
src/snakemake/logging.py (1)
src/snakemake/settings/enums.py (1)
Quietness(24-29)
🔇 Additional comments (5)
src/snakemake/logging.py (5)
246-246: Excellent simplification of shell command formatting.The removal of conditional logic from
format_shellcmdaligns perfectly with the architectural improvement of moving shell command filtering to theDefaultFilterclass. This ensures cleaner separation of concerns.
369-369: Good architectural improvement by adding shell command control to the filter.Adding the
printshellcmdsparameter toDefaultFilteris the correct approach for controlling shell command visibility at the filtering stage rather than in formatting.Also applies to: 375-375
399-402: Excellent implementation of shell command filtering logic.This is the core fix that addresses the blank line issue. By filtering out shell command events when
printshellcmdsis False, the formatter never receives empty messages to format, eliminating the blank lines.The logic is clear and follows the filtering pattern used elsewhere in the class.
618-622: Proper integration of shell command filtering in logger setup.The addition of
self.settings.printshellcmdsto theDefaultFilterconstructor completes the refactoring by ensuring the filter receives the necessary configuration. This maintains the existing API while enabling the new filtering behavior.
393-393:Details
❓ Verification inconclusive
Verify that changing shell command quietness level aligns with user expectations.
Changing
LogEvent.SHELLCMDfromQuietness.PROGRESStoQuietness.RULESmeans shell commands will now be suppressed when using--quiet rulesinstead of--quiet progress. This addresses the blank line issue mentioned in the PR objectives.However, this is a behavioral change that might affect existing user workflows.
Please verify that this change aligns with user expectations by checking existing documentation or usage patterns:
🏁 Script executed:
#!/bin/bash # Description: Check for documentation or examples that mention shell command quietness behavior # Search for documentation about shell command logging and quietness levels rg -A 3 -B 3 "shell.*command.*quiet|quiet.*shell.*command" --type md --type rst --type txt # Search for test cases that might verify shell command quietness behavior rg -A 5 -B 5 "SHELLCMD.*quiet|quiet.*SHELLCMD" --type py # Search for any mentions of the old PROGRESS mapping for shell commands rg -A 3 -B 3 "SHELLCMD.*PROGRESS|PROGRESS.*SHELLCMD" --type pyLength of output: 216
🏁 Script executed:
#!/bin/bash # Inspect Quietness enum definition rg -n "class Quietness" -A 10 -B 0 src/snakemake/logging.py # Inspect CLI parser references to --quiet flag rg -n "\-\-quiet" -A 5 -B 5 src/snakemake # Inspect use of PROGRESS and RULES in logging.py rg -n "PROGRESS" -A 5 -B 5 src/snakemake/logging.py rg -n "RULES" -A 5 -B 5 src/snakemake/logging.pyLength of output: 8120
Confirm shell-command quietness mapping
We changed LogEvent.SHELLCMD from Quietness.PROGRESS to Quietness.RULES (src/snakemake/logging.py:393), so shell commands will now be suppressed by--quiet rulesinstead of--quiet progress.• No user documentation or examples mention shell-command quietness behavior.
• No existing unit tests cover how--quiet progressvs.--quiet rulesaffects shell-command output.Please verify that this new mapping matches intended workflows, update any user-facing docs or CLI help accordingly, and add a unit test to cover shell-command suppression under both
--quiet progressand--quiet rules.
cademirch
left a comment
There was a problem hiding this comment.
Thanks for making these changes. Let's make sure tests pass but I think this is okay to merge.
I added JOB_INFO LogEvent to the info aboout temp files. Previously there was no way to use the quiet flag to filter these messages. I would argue the the removal of temp files are related to the JOB_INFO logs.
Also check if formatted_message is empty otherwise logger would print empt lines for quieted rules
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
src/snakemake/logging.py (1)
509-509: Consider using 'in' operator for better readabilityThe current condition works correctly but could be more readable using the
inoperator as suggested by the static analysis tool.- if formatted_message == "None" or formatted_message=="": + if formatted_message in ("None", ""):This change improves readability and follows Python best practices for multiple equality comparisons.
🧰 Tools
🪛 Pylint (3.3.7)
[refactor] 509-509: Consider merging these comparisons with 'in' by using 'formatted_message in ('None', '')'. Use a set instead if elements are hashable.
(R1714)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
src/snakemake/dag.py(2 hunks)src/snakemake/logging.py(5 hunks)src/snakemake/scheduler.py(2 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest type annotations for functions that are defined inside of functions or methods. Do not suggest type annotation of the `s...
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
src/snakemake/dag.pysrc/snakemake/logging.pysrc/snakemake/scheduler.py
🧬 Code Graph Analysis (2)
src/snakemake/logging.py (1)
src/snakemake/settings/enums.py (1)
Quietness(24-29)
src/snakemake/scheduler.py (1)
src/snakemake/settings/enums.py (1)
Quietness(24-29)
🪛 Pylint (3.3.7)
src/snakemake/dag.py
[convention] 33-33: Imports from package snakemake are not grouped
(C0412)
[convention] 964-964: Line too long (121/100)
(C0301)
[convention] 967-967: Line too long (121/100)
(C0301)
[warning] 965-965: Use lazy % formatting in logging functions
(W1203)
[warning] 968-968: Use lazy % formatting in logging functions
(W1203)
src/snakemake/logging.py
[refactor] 509-509: Consider merging these comparisons with 'in' by using 'formatted_message in ('None', '')'. Use a set instead if elements are hashable.
(R1714)
src/snakemake/scheduler.py
[convention] 73-73: Line too long (106/100)
(C0301)
🔇 Additional comments (7)
src/snakemake/dag.py (1)
33-33: LGTM!The import of
Quietnessenum is necessary for the conditional logging logic implemented in thehandle_tempmethod.🧰 Tools
🪛 Pylint (3.3.7)
[convention] 33-33: Imports from package snakemake are not grouped
(C0412)
src/snakemake/scheduler.py (1)
73-73: LGTM: Improved granular quietness control for progress printingThis change enhances the quietness handling by allowing progress printing when the specific
Quietness.PROGRESSflag is not set, even if other quietness flags are present. This provides more fine-grained control over what gets suppressed and aligns well with the broader logging improvements in this PR.🧰 Tools
🪛 Pylint (3.3.7)
[convention] 73-73: Line too long (106/100)
(C0301)
src/snakemake/logging.py (5)
246-246: LGTM: Simplified shell command formattingThe shell command formatter now unconditionally returns the message, which is correct since the filtering logic has been moved to the
DefaultFilterclass. This separation of concerns makes the code cleaner and more maintainable.
369-369: LGTM: Enhanced filter constructor for shell command controlThe addition of the
printshellcmdsparameter to theDefaultFilterconstructor enables proper filtering of shell command messages based on the CLI flag. This is a much cleaner approach than the previous method of setting messages to empty strings.Also applies to: 375-375
393-393: LGTM: Corrected quietness mapping for shell commandsMoving
LogEvent.SHELLCMDfromQuietness.PROGRESStoQuietness.RULESaligns shell command log events with the rules quietness level. This addresses the root cause of blank lines appearing when runningsnakemake --quiet rules, as mentioned in the PR discussion.
399-401: LGTM: Proper shell command filtering implementationThis filtering logic correctly implements the approach discussed in the PR comments. Shell command messages are now filtered out when
printshellcmdsisFalse, preventing empty messages from reaching the emit method and causing blank lines.
618-621: LGTM: Consistent parameter passing to filterThe
LoggerManagercorrectly passes theprintshellcmdssetting to theDefaultFilterconstructor, enabling the new filtering logic. This completes the implementation of the shell command filtering approach discussed in the PR.
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (1)
src/snakemake/dag.py (1)
964-975: Address code quality issues raised in previous reviews.The conditional logging implementation correctly addresses the PR objectives, but multiple code quality issues remain unresolved from previous reviews, including line length violations, code duplication, and improper use of f-strings in logging functions.
The suggested refactor from previous reviews should be applied:
- Extract the duplicated condition to a variable
- Use lazy formatting (% formatting) in logger.info calls
- Combine the logging statements to eliminate duplication
- Keep lines under 100 characters
This maintains the correct logic while significantly improving code quality and readability.
🧰 Tools
🪛 Pylint (3.3.7)
[warning] 968-968: Use lazy % formatting in logging functions
(W1203)
[warning] 974-974: Use lazy % formatting in logging functions
(W1203)
🧹 Nitpick comments (2)
src/snakemake/dag.py (1)
33-33: Import grouping issue flagged by static analysis.The import is functionally correct but static analysis suggests it should be grouped with other snakemake imports for better organization.
🧰 Tools
🪛 Pylint (3.3.7)
[convention] 33-33: Imports from package snakemake are not grouped
(C0412)
src/snakemake/logging.py (1)
509-509: Improve readability with 'in' operator.The addition of the empty string check correctly prevents blank lines from being emitted. However, consider improving readability by using the 'in' operator as suggested by static analysis.
Apply this diff to improve readability:
- if formatted_message == "None" or formatted_message == "": + if formatted_message in ("None", ""):🧰 Tools
🪛 Pylint (3.3.7)
[refactor] 509-509: Consider merging these comparisons with 'in' by using 'formatted_message in ('None', '')'. Use a set instead if elements are hashable.
(R1714)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
src/snakemake/dag.py(2 hunks)src/snakemake/logging.py(5 hunks)src/snakemake/scheduler.py(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- src/snakemake/scheduler.py
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest type annotations for functions that are defined inside of functions or methods. Do not suggest type annotation of the `s...
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
src/snakemake/dag.pysrc/snakemake/logging.py
🧬 Code Graph Analysis (1)
src/snakemake/logging.py (1)
src/snakemake/settings/enums.py (1)
Quietness(24-29)
🪛 Pylint (3.3.7)
src/snakemake/dag.py
[convention] 33-33: Imports from package snakemake are not grouped
(C0412)
[warning] 968-968: Use lazy % formatting in logging functions
(W1203)
[warning] 974-974: Use lazy % formatting in logging functions
(W1203)
src/snakemake/logging.py
[refactor] 509-509: Consider merging these comparisons with 'in' by using 'formatted_message in ('None', '')'. Use a set instead if elements are hashable.
(R1714)
🔇 Additional comments (5)
src/snakemake/logging.py (5)
244-246: LGTM! Proper separation of concerns.The simplification of
format_shellcmdto unconditionally return the message is correct. Moving the conditional logic to the filtering stage improves the architecture by keeping formatters focused on formatting and filters focused on filtering decisions.
369-375: LGTM! Necessary infrastructure for shell command filtering.The addition of the
printshellcmdsparameter to theDefaultFilterconstructor and its storage as an instance attribute properly supports the new filtering logic for shell commands.
393-393: LGTM! Improved quietness level alignment.Mapping
LogEvent.SHELLCMDtoQuietness.RULESinstead ofQuietness.PROGRESSproperly aligns shell command logging with rule-related quietness control. This change ensures that--quiet ruleswill suppress shell commands as expected.Note: This may change behavior for users who previously used
--quiet progressto suppress shell commands.
399-401: LGTM! Correct implementation of shell command filtering.The new filtering logic properly filters out shell command log events when
printshellcmdsisFalse. This addresses the core issue of blank lines appearing in quiet mode by preventing shell command messages from reaching the formatter when they shouldn't be displayed.
618-621: LGTM! Proper wiring of printshellcmds setting.The addition of
self.settings.printshellcmdsto theDefaultFilterconstructor call correctly connects the settings to the filtering logic, enabling the new shell command filtering behavior.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
src/snakemake/logging.py (2)
387-390: Consider simplifying the nested if statement.The logic is correct, but the nested if statements can be simplified for better readability.
- if hasattr(record, "quietness"): - if record.quietness in self.quiet: - return False + if hasattr(record, "quietness") and record.quietness in self.quiet: + return False🧰 Tools
🪛 Ruff (0.11.9)
387-388: Use a single
ifstatement instead of nestedifstatementsCombine
ifstatements usingand(SIM102)
513-513: Consider using 'in' operator for cleaner comparison.The logic is correct for preventing empty message emission, but the comparison can be simplified.
- if formatted_message == "None" or formatted_message == "": + if formatted_message in ("None", ""):🧰 Tools
🪛 Pylint (3.3.7)
[refactor] 513-513: Consider merging these comparisons with 'in' by using 'formatted_message in ('None', '')'. Use a set instead if elements are hashable.
(R1714)
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
src/snakemake/dag.py(2 hunks)src/snakemake/logging.py(5 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- src/snakemake/dag.py
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Do not try to improve formatting. Do not suggest type annotations for functions that are defined inside of functions or methods. Do not suggest type annotation of the `s...
**/*.py: Do not try to improve formatting.
Do not suggest type annotations for functions that are defined inside of functions or methods.
Do not suggest type annotation of theselfargument of methods.
Do not suggest type annotation of theclsargument of classmethods.
Do not suggest return type annotation if a function or method does not contain areturnstatement.
src/snakemake/logging.py
🧬 Code Graph Analysis (1)
src/snakemake/logging.py (1)
src/snakemake/settings/enums.py (1)
Quietness(24-29)
🪛 Ruff (0.11.9)
src/snakemake/logging.py
387-388: Use a single if statement instead of nested if statements
Combine if statements using and
(SIM102)
🪛 Pylint (3.3.7)
src/snakemake/logging.py
[refactor] 513-513: Consider merging these comparisons with 'in' by using 'formatted_message in ('None', '')'. Use a set instead if elements are hashable.
(R1714)
🔇 Additional comments (5)
src/snakemake/logging.py (5)
246-246: LGTM! Shell command formatting now unconditional.This change aligns with the PR objective to move shell command filtering from the formatting stage to the filtering stage. The formatter now always returns the message, allowing the filter to handle visibility logic.
369-375: LGTM! Filter now handlesprintshellcmdsparameter.Good refactor moving the
printshellcmdslogic from the formatter to the filter. This allows proper filtering of shell command messages instead of setting them to empty strings, which was causing the blank lines issue.
397-397: LGTM! Correct quietness level mapping for shell commands.This change fixes the root cause identified in the PR comments. Mapping
SHELLCMDtoQuietness.RULESinstead ofQuietness.PROGRESSensures shell command messages are properly filtered when using--quiet rules.
403-406: LGTM! Proper shell command filtering based on printshellcmds flag.This is the key fix that prevents blank lines. Instead of setting shell command messages to empty strings in the formatter, they are now properly filtered out when
printshellcmdsis False.
622-626: LGTM! LoggerManager now passes printshellcmds to filter.Correct update to pass the
printshellcmdsparameter to theDefaultFilterconstructor, completing the refactor that moves shell command filtering logic from formatter to filter.
I am using SnakeMake with wildcards for a very large number of files. I observed that executimg `snakemake --quiet rules` would still leave blank lines corresponding to the space where the rules would have been.  I am not fully fammiliar with the snakemake logging setup but i narrowed down the problem to the `emit` mehod of `ColorizingTextHandler`. My solution was to check for an empty `formatted_message`. Our large number of files also use a corresponding number of temp files which results in thousands of print messages about which temporary files would be remove. I purpose adding `LogEvent.JOB_INFO` to the temporary file messages in order to allow quieting these messages. ### QC <!-- Make sure that you can tick the boxes below. --> * [x] The PR contains a test case for the changes or the changes are already covered by an existing test case. * [x] The documentation (`docs/`) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Improved logging to prevent empty or "None" messages from appearing in the output. - Refined logging behavior to respect quiet mode settings, reducing unnecessary log messages during temporary file removal and job scheduling progress updates. - Enhanced control over shell command and progress log visibility based on specific quietness flags for a clearer user experience. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Co-authored-by: Christoffer <christofferrs@protonmail.com> Co-authored-by: Cade Mirchandani <cmirchan@ucsc.edu>


I am using SnakeMake with wildcards for a very large number of files. I observed that executimg
snakemake --quiet ruleswould still leave blank lines corresponding to the space where the rules would have been.I am not fully fammiliar with the snakemake logging setup but i narrowed down the problem to the
emitmehod ofColorizingTextHandler. My solution was to check for an emptyformatted_message.Our large number of files also use a corresponding number of temp files which results in thousands of print messages about which temporary files would be remove. I purpose adding
LogEvent.JOB_INFOto the temporary file messages in order to allow quieting these messages.QC
docs/) is updated to reflect the changes or this is not necessary (e.g. if the change does neither modify the language nor the behavior or functionalities of Snakemake).Summary by CodeRabbit