-
Notifications
You must be signed in to change notification settings - Fork 633
Remove input and output from properties in jobscript #2097
Description
Is your feature request related to a problem? Please describe.
Currently, the properties injected into the jobscript include input and output
Lines 998 to 1010 in 512e49b
| properties = { | |
| "type": "single", | |
| "rule": self.rule.name, | |
| "local": self.is_local, | |
| "input": self.input, | |
| "output": self.output, | |
| "wildcards": self.wildcards_dict, | |
| "params": params, | |
| "log": self.log, | |
| "threads": self.threads, | |
| "resources": resources, | |
| "jobid": self.dag.jobid(self), | |
| } |
When running a job that has thousands of input or output files this creates a problem on slurm. Slurm's default script size if 4MB. I have an example of a job that takes 44,000 input files and the jobscript is 4.6MB. As such, it fails submission to slurm.
Describe the solution you'd like
Do input and output need to be in the properties injected into the jobscript? The slurm and lsf profiles dont use these properties, is there other things using it?
If not, can we just remove them?
Additional context
This has arisen in the slurm profile before also - see Snakemake-Profiles/slurm#87