Smart input function exception handling#479
Merged
johanneskoester merged 15 commits intomasterfrom Jul 7, 2020
Merged
Conversation
Report all errors if no candidate rule for a file was found.
Contributor
|
Please format your code with black: |
… always exactly the same
|
Kudos, SonarCloud Quality Gate passed!
|
huddlej
added a commit
to nextstrain/ncov
that referenced
this pull request
Nov 25, 2021
Fixes multiple input function exceptions that occur when running the Nextstrain profiles with older versions of Snakemake (pre-5.20.0). Since Nextstrain builds produce dated JSONs (e.g., with "YYYY-MM-DD.json" suffixes), the change to allow numbers in build names makes these dated JSONs ambiguous to create from Snakemake's perspective. Despite the earlier inclusion of rule orders to prioritize the "dated_json" rule over the now-ambiguous "finalize" rule, Snakemake still needs to walk through the graph of possible ways to create the dated JSON files before it accounts for the rule order. If any exceptions occur during this graph walk, pre-5.20.0 Snakemake versions would exit with an "InputFunctionException". As of 5.20.0, Snakemake catches these exceptions internally and keeps searching the graph for valid paths [1]. Because Snakemake initially interprets the value of the "build_name" wildcard to be something like "europe_2021-11-24" instead of "europe", all dictionary lookups in the workflow that expect "europe" use the dated name instead and raise key errors. The solution in this commit is to replace hardcoded key lookups with `get` method calls and a default empty dictionary. Although we should really update the Nextstrain Docker base image to use the latest Snakemake (6.10.0 from Oct 2021) instead of 5.10.0 (from Jan 2020), this commit fixes the input function exceptions that were breaking the CI build in the Docker image and likely the cause of the error that @rneher mentioned in the original PR comments [2]. [1] snakemake/snakemake#479 [2] #524 (comment)
huddlej
added a commit
to nextstrain/ncov
that referenced
this pull request
Apr 12, 2022
Fixes multiple input function exceptions that occur when running the Nextstrain profiles with older versions of Snakemake (pre-5.20.0). Since Nextstrain builds produce dated JSONs (e.g., with "YYYY-MM-DD.json" suffixes), the change to allow numbers in build names makes these dated JSONs ambiguous to create from Snakemake's perspective. Despite the earlier inclusion of rule orders to prioritize the "dated_json" rule over the now-ambiguous "finalize" rule, Snakemake still needs to walk through the graph of possible ways to create the dated JSON files before it accounts for the rule order. If any exceptions occur during this graph walk, pre-5.20.0 Snakemake versions would exit with an "InputFunctionException". As of 5.20.0, Snakemake catches these exceptions internally and keeps searching the graph for valid paths [1]. Because Snakemake initially interprets the value of the "build_name" wildcard to be something like "europe_2021-11-24" instead of "europe", all dictionary lookups in the workflow that expect "europe" use the dated name instead and raise key errors. The solution in this commit is to replace hardcoded key lookups with `get` method calls and a default empty dictionary. Although we should really update the Nextstrain Docker base image to use the latest Snakemake (6.10.0 from Oct 2021) instead of 5.10.0 (from Jan 2020), this commit fixes the input function exceptions that were breaking the CI build in the Docker image and likely the cause of the error that @rneher mentioned in the original PR comments [2]. [1] snakemake/snakemake#479 [2] #524 (comment)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR enables to consider alternative paths in the DAG if an input function fails. To make this change more accesible, the --debug-dag output has been extended to include all ignored exceptions that occur during DAG evaluation.