Skip to content

Smart input function exception handling#479

Merged
johanneskoester merged 15 commits intomasterfrom
smart-input-function-exception-handling
Jul 7, 2020
Merged

Smart input function exception handling#479
johanneskoester merged 15 commits intomasterfrom
smart-input-function-exception-handling

Conversation

@johanneskoester
Copy link
Copy Markdown
Contributor

@johanneskoester johanneskoester commented Jul 7, 2020

This PR enables to consider alternative paths in the DAG if an input function fails. To make this change more accesible, the --debug-dag output has been extended to include all ignored exceptions that occur during DAG evaluation.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Jul 7, 2020

Please format your code with black: black snakemake tests/*.py.

@johanneskoester johanneskoester merged commit 249ff5c into master Jul 7, 2020
@johanneskoester johanneskoester deleted the smart-input-function-exception-handling branch July 7, 2020 16:50
@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Jul 7, 2020

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities (and Security Hotspot 0 Security Hotspots to review)
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

huddlej added a commit to nextstrain/ncov that referenced this pull request Nov 25, 2021
Fixes multiple input function exceptions that occur when running the
Nextstrain profiles with older versions of Snakemake (pre-5.20.0). Since
Nextstrain builds produce dated JSONs (e.g., with "YYYY-MM-DD.json"
suffixes), the change to allow numbers in build names makes these dated
JSONs ambiguous to create from Snakemake's perspective.

Despite the earlier inclusion of rule orders to prioritize the
"dated_json" rule over the now-ambiguous "finalize" rule, Snakemake
still needs to walk through the graph of possible ways to create the
dated JSON files before it accounts for the rule order. If any
exceptions occur during this graph walk, pre-5.20.0 Snakemake versions
would exit with an "InputFunctionException". As of 5.20.0, Snakemake
catches these exceptions internally and keeps searching the graph for
valid paths [1]. Because Snakemake initially interprets the value of the
"build_name" wildcard to be something like "europe_2021-11-24" instead
of "europe", all dictionary lookups in the workflow that expect "europe"
use the dated name instead and raise key errors. The solution in this
commit is to replace hardcoded key lookups with `get` method calls and a
default empty dictionary.

Although we should really update the Nextstrain Docker base image to use
the latest Snakemake (6.10.0 from Oct 2021) instead of 5.10.0 (from Jan
2020), this commit fixes the input function exceptions that were
breaking the CI build in the Docker image and likely the cause of the
error that @rneher mentioned in the original PR comments [2].

[1] snakemake/snakemake#479
[2] #524 (comment)
huddlej added a commit to nextstrain/ncov that referenced this pull request Apr 12, 2022
Fixes multiple input function exceptions that occur when running the
Nextstrain profiles with older versions of Snakemake (pre-5.20.0). Since
Nextstrain builds produce dated JSONs (e.g., with "YYYY-MM-DD.json"
suffixes), the change to allow numbers in build names makes these dated
JSONs ambiguous to create from Snakemake's perspective.

Despite the earlier inclusion of rule orders to prioritize the
"dated_json" rule over the now-ambiguous "finalize" rule, Snakemake
still needs to walk through the graph of possible ways to create the
dated JSON files before it accounts for the rule order. If any
exceptions occur during this graph walk, pre-5.20.0 Snakemake versions
would exit with an "InputFunctionException". As of 5.20.0, Snakemake
catches these exceptions internally and keeps searching the graph for
valid paths [1]. Because Snakemake initially interprets the value of the
"build_name" wildcard to be something like "europe_2021-11-24" instead
of "europe", all dictionary lookups in the workflow that expect "europe"
use the dated name instead and raise key errors. The solution in this
commit is to replace hardcoded key lookups with `get` method calls and a
default empty dictionary.

Although we should really update the Nextstrain Docker base image to use
the latest Snakemake (6.10.0 from Oct 2021) instead of 5.10.0 (from Jan
2020), this commit fixes the input function exceptions that were
breaking the CI build in the Docker image and likely the cause of the
error that @rneher mentioned in the original PR comments [2].

[1] snakemake/snakemake#479
[2] #524 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant