It seems like when I use the http storage plugin with the kubernetes executer, it submits the job but with an added argument --storage-http-allow-redirects.
Snakefile:
storage HTTP:
provider="http"
rule all:
input:
"results/summary.txt"
rule get_file:
input:
storage.HTTP("https://example-files.online-convert.com/document/txt/example.txt")
output:
"results/summary.txt"
shell:
"cat {input} > {output}"
profile:
storage-gcs-project: orcestra-388613
default-storage-provider: gcs
default-storage-prefix: gs://orcestradata/snakemake8_test
executor: kubernetes
jobs: 5
log:
pixi run snake --verbose --workflow-profile default -F
✨ Pixi task (snake in default): snakemake -c1 --verbose --workflow-profile default -F
Using workflow specific profile default for setting default command line arguments.
Path results/summary.txt, converted to .snakemake/storage/gcs/orcestradata/snakemake8_test/results/summary.txt
Path results/summary.txt, converted to .snakemake/storage/gcs/orcestradata/snakemake8_test/results/summary.txt
Building DAG of jobs...
shared_storage_local_copies: False
remote_exec: False
Uploading source archive to storage provider...
Checking status of 0 jobs
Using snakemake/snakemake:v8.10.8 for Kubernetes jobs.
Using shell: /bin/bash
Provided remote nodes: 5
Job stats:
job count
-------- -------
all 1
get_file 1
total 2
Resources before job selection: {'_cores': 9223372036854775807, '_nodes': 5}
Ready jobs (1)
Select jobs to execute...
Using greedy selector because only single job has to be scheduled.
Selected jobs (1)
Resources after job selection: {'_cores': 9223372036854775806, '_nodes': 4}
Execute 1 jobs...
[Thu Apr 25 13:22:45 2024]
rule get_file:
output: gs://orcestradata/snakemake8_test/results/summary.txt (send to storage)
jobid: 1
reason: Forced execution
resources: tmpdir=<TBD>
General args: ['--force', '--target-files-omit-workdir-adjustment', '--keep-storage-local-copies', '--max-inventory-time 0', '--nocolor', '--notemp', '--no-hooks', '--nolock', '--ignore-incomplete', '', '--verbose ', '--rerun-triggers params software-env mtime code input', '', '', '', '--conda-frontend mamba', '', '', '', '', '', '--shared-fs-usage none', '', '--wrapper-prefix https://github.com/snakemake/snakemake-wrappers/raw/', '', '', '', '', '', '--latency-wait 5', '--scheduler ilp', '--local-storage-prefix .snakemake/storage', '', '', '', '', '', '--storage-gcs-retries 5', '--storage-http-allow-redirects True', '', '', '--default-storage-prefix gs://orcestradata/snakemake8_test --default-storage-provider gcs', '--default-resources base64//dG1wZGlyPXN5c3RlbV90bXBkaXI=']
Executing job: pip install --target '.snakemake/pip-deployments' snakemake-storage-plugin-gcs && python -m snakemake --deploy-sources gs://orcestradata/snakemake8_test/snakemake-workflow-sources.aa2971cef4d04f37d9a1a2a2ba973d589bf0fb2e4c198081e94539acfe32f4fc.tar.xz aa2971cef4d04f37d9a1a2a2ba973d589bf0fb2e4c198081e94539acfe32f4fc --default-storage-prefix gs://orcestradata/snakemake8_test --default-storage-provider gcs --storage-gcs-retries 5 --storage-http-allow-redirects True && python -m snakemake --snakefile Snakefile --target-jobs 'get_file:' --allowed-rules 'get_file' --cores 1 --attempt 1 --force-use-threads --force --target-files-omit-workdir-adjustment --keep-storage-local-copies --max-inventory-time 0 --nocolor --notemp --no-hooks --nolock --ignore-incomplete --verbose --rerun-triggers params software-env mtime code input --conda-frontend mamba --shared-fs-usage none --wrapper-prefix https://github.com/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --scheduler ilp --local-storage-prefix .snakemake/storage --storage-gcs-retries 5 --storage-http-allow-redirects True --default-storage-prefix gs://orcestradata/snakemake8_test --default-storage-provider gcs --default-resources base64//dG1wZGlyPXN5c3RlbV90bXBkaXI= --mode remote
job resources: {'_cores': 1, '_nodes': 1, 'tmpdir': '<TBD>'}
k8s pod resources: {'cpu': '950m'}
Get status with:
kubectl describe pod snakejob-baa55f88-7989-50a6-9ecf-3c49ed01bedc
kubectl logs snakejob-baa55f88-7989-50a6-9ecf-3c49ed01bedc
Checking status of 1 jobs
Checking status of 1 jobs
[Thu Apr 25 13:23:05 2024]
Error in rule get_file:
message: For details, please issue:
kubectl describe pod snakejob-baa55f88-7989-50a6-9ecf-3c49ed01bedc
kubectl logs snakejob-baa55f88-7989-50a6-9ecf-3c49ed01bedcFor further error details see the cluster/cloud log and the log files of the involved rule(s).
jobid: 1
output: gs://orcestradata/snakemake8_test/results/summary.txt (send to storage)
log: /var/folders/8t/rwh6rzg93jxfqkb63gt2n4940000gn/T/snakemakex_xytdlb/persistence/auxiliary/kubernetes-logs/snakejob-baa55f88-7989-50a6-9ecf-3c49ed01bedc.log (check log file(s) for error details)
shell:
wget https://example-files.online-convert.com/document/txt/example.txt -O results/summary.txt
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
external_jobid: snakejob-baa55f88-7989-50a6-9ecf-3c49ed01bedc
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-04-25T132244.819888.snakemake.log
bold text in the logged command shows where its inserted:
Executing job: pip install --target '.snakemake/pip-deployments' snakemake-storage-plugin-gcs && python -m snakemake --deploy-sources gs://orcestradata/snakemake8_test/snakemake-workflow-sources.aa2971cef4d04f37d9a1a2a2ba973d589bf0fb2e4c198081e94539acfe32f4fc.tar.xz aa2971cef4d04f37d9a1a2a2ba973d589bf0fb2e4c198081e94539acfe32f4fc --default-storage-prefix gs://orcestradata/snakemake8_test --default-storage-provider gcs --storage-gcs-retries 5 --storage-http-allow-redirects True && python -m snakemake --snakefile Snakefile --target-jobs 'get_file:' --allowed-rules 'get_file' --cores 1 --attempt 1 --force-use-threads --force --target-files-omit-workdir-adjustment --keep-storage-local-copies --max-inventory-time 0 --nocolor --notemp --no-hooks --nolock --ignore-incomplete --verbose --rerun-triggers params software-env mtime code input --conda-frontend mamba --shared-fs-usage none --wrapper-prefix https://github.com/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --scheduler ilp --local-storage-prefix .snakemake/storage --storage-gcs-retries 5 --storage-http-allow-redirects True --default-storage-prefix gs://orcestradata/snakemake8_test --default-storage-provider gcs --default-resources base64//dG1wZGlyPXN5c3RlbV90bXBkaXI= --mode remote
It seems like when I use the http storage plugin with the kubernetes executer, it submits the job but with an added argument
--storage-http-allow-redirects.Snakefile:
profile:
log:
bold text in the logged command shows where its inserted:
Executing job: pip install --target '.snakemake/pip-deployments' snakemake-storage-plugin-gcs && python -m snakemake --deploy-sources gs://orcestradata/snakemake8_test/snakemake-workflow-sources.aa2971cef4d04f37d9a1a2a2ba973d589bf0fb2e4c198081e94539acfe32f4fc.tar.xz aa2971cef4d04f37d9a1a2a2ba973d589bf0fb2e4c198081e94539acfe32f4fc --default-storage-prefix gs://orcestradata/snakemake8_test --default-storage-provider gcs --storage-gcs-retries 5 --storage-http-allow-redirects True && python -m snakemake --snakefile Snakefile --target-jobs 'get_file:' --allowed-rules 'get_file' --cores 1 --attempt 1 --force-use-threads --force --target-files-omit-workdir-adjustment --keep-storage-local-copies --max-inventory-time 0 --nocolor --notemp --no-hooks --nolock --ignore-incomplete --verbose --rerun-triggers params software-env mtime code input --conda-frontend mamba --shared-fs-usage none --wrapper-prefix https://github.com/snakemake/snakemake-wrappers/raw/ --latency-wait 5 --scheduler ilp --local-storage-prefix .snakemake/storage --storage-gcs-retries 5 --storage-http-allow-redirects True --default-storage-prefix gs://orcestradata/snakemake8_test --default-storage-provider gcs --default-resources base64//dG1wZGlyPXN5c3RlbV90bXBkaXI= --mode remote