is it going to be confusing that the additional_inputs files are not compressed?

Good question! I'll drop these comments for now as out of date. Ideally we should be able to handle compressed or uncompressed inputs from s3, https, local, or workflow specific in the final implementation

Reminder to update this comment. I investigated the supported compressions a little bit as part of #89 (under the "File Compression" section of the PR description). What compressions we actually support will be influenced by what you do with the merge rules.

Dropped comments entirely in 72f9d21
But let me know if I misunderstood something here

I think it'd be great to document what compressions are supported, although this is not entirely generalizable across pathogen repos in the way we're aiming for with the multiple_inputs so we have to be careful with how we write the comment. As I understand it, and assuming we use separate merge_sequences and merge_metadata rules, then the supported compressions are:

• Metadata
  • Multiple sources: whatever augur merge supports
  • Single source: what's supported by the rules where input_metadata is used (4 different rules for zika as it currently stands)
• Sequences
  • Multiple sources: whatever augur merge supports
  • Single source: what's supported by the rules where input_sequences is used (only one rule for zika as it currently stands)

Perhaps I can link to the augur docs on different compression support, similar to Nextclade docs?

• https://docs.nextstrain.org/projects/nextclade/en/stable/user/input-files/compression.html
  If it exists in augur docs somewhere...

Migrating this to an issue: #93

I'm on the fence about this include statement going here or the main Snakefile. Does anyone have a preference?

I have slight preference for adding the include statment to the main Snakefile in case we want to use the shared path_or_url function in other parts of the workflow.

Good point about scoping! I leaned toward main Snakefile so fixed up in 75dc2e0

Agreed - the intention was for path_or_url to be available for wider use than just the inputs.

One use-case we don't support (as discussed elsewhere) is something like:

inputs:
  foo = path_or_url(config['bar'])

• If config['bar'] is a URL it'll just work when run from either the pathogen repo or an analysis directory
• If config['bar'] is a path which exists in the analysis directory (and we're running it in an analysis directory) it'll just work
• If config['bar'] is a path which exists in the pathogen repo and we're running it in an analysis directory then it won't work

(There's no easy solution as far as I can see. You can't just use something like resolve_config_path within path_or_url because we support output files and because some workflows use modified versions of resolve_config_path. We could expand path_or_url to take a path_resolver argument which would force the path to exist, i.e. only be useful in the context of params/inputs. Happy to implement this if people want.)

If config['bar'] is a path which exists in the pathogen repo and we're running it in an analysis directory then it won't work

Hmm, I thought we could wrap resolve_config_path around path_or_url, since path_or_url should return the local file location for a URL.

inputs:
  foo = resolve_config_path(path_or_url(config['bar']))

but this currently runs into a weird WorkflowError (reported in snakemake/snakemake#3027)

Error:
  WorkflowError:
    Flags ({'storage_object': <snakemake_storage_plugin_http.StorageObject object at 0xffffa0415f90>}) in file pattern '.snakemake/storage/http/github.com/nextstrain/augur/blob/fe3f57975c34ff180cb437bebfefd68fa4a4afed/augur/data/geolocation_rules.tsv' given to expand() are invalid. Flags (e.g. temp(), directory()) have to be applied outside of expand (e.g. 'temp(expand("plots/{sample}.pdf", sample=SAMPLES))').
Wildcards:

Beyond the workflow error you get, conceptually the .snakemake/storage file may not exist when the inputs are evaluated so our path resolution approach wouldn't work. This is somewhat analogous to using path_or_url for outputs. Path evaluation within path_or_url will be straightforward I think (via something like path_or_url(uri, path_resolver))