Skip to content

Remove input and output from properties in jobscript #2097

@mbhall88

Description

@mbhall88

Is your feature request related to a problem? Please describe.

Currently, the properties injected into the jobscript include input and output

snakemake/snakemake/jobs.py

Lines 998 to 1010 in 512e49b

properties = {
"type": "single",
"rule": self.rule.name,
"local": self.is_local,
"input": self.input,
"output": self.output,
"wildcards": self.wildcards_dict,
"params": params,
"log": self.log,
"threads": self.threads,
"resources": resources,
"jobid": self.dag.jobid(self),
}

When running a job that has thousands of input or output files this creates a problem on slurm. Slurm's default script size if 4MB. I have an example of a job that takes 44,000 input files and the jobscript is 4.6MB. As such, it fails submission to slurm.

Describe the solution you'd like

Do input and output need to be in the properties injected into the jobscript? The slurm and lsf profiles dont use these properties, is there other things using it?

If not, can we just remove them?

Additional context

This has arisen in the slurm profile before also - see Snakemake-Profiles/slurm#87

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions