Fix .command.stage file write when using Google Cloud#6558
Fix .command.stage file write when using Google Cloud#6558bentsherman merged 6 commits intomasterfrom
Conversation
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
✅ Deploy Preview for nextflow-docs-staging canceled.
|
|
Ideally it should not write the stage script at all for google batch. That was the intention of this check: But I didn't consider that the google batch script launcher overwrites the task workDir to be the local container-mounted path. So I'm not sure if there is a clean way to fix this check |
|
I fix changing this because it was the simpler one that implied less modifications. The easy approaches I was considering is overriding the |
|
I feel like the cleanest solution is to add a method to For that I think you'll have to add a field to the |
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
|
I have finally implemented the fix by adding the |
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
This reverts commit cda4ac1.
) Improvement of #6558 providing: 1. More declarative approach 2. Better control on enabling/disabling stage file capability 3. Fix Google Batch stage file method - Add stageFileEnabled field to TaskBean - Add isStageFileEnabled() to TaskRun delegating to executor - Add isStageFileEnabled() to Executor with NXF_WRAPPER_STAGE_FILE_ENABLED env var support, defaulting to true for AbstractGridExecutor - Enable stageFileEnabled for GoogleBatchExecutor - Add comprehensive tests Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
) [ci fast] * Add stageFileEnabled flag to control .command.stage file creation (#4279) Improvement of #6558 providing: 1. More declarative approach 2. Better control on enabling/disabling stage file capability 3. Fix Google Batch stage file method - Add stageFileEnabled field to TaskBean - Add isStageFileEnabled() to TaskRun delegating to executor - Add isStageFileEnabled() to Executor with NXF_WRAPPER_STAGE_FILE_ENABLED env var support, defaulting to true for AbstractGridExecutor - Enable stageFileEnabled for GoogleBatchExecutor - Add comprehensive tests Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
close #5888
.command.stage file is created when stage script is larger than 1MB and stage file is in the default filesystem. In the case of Google Batch executor, the stage file is /mnt/xxx/ because of the gcsfuse. Later, it tries to write the file that doesn't exist in the head node, so it fails. It is mainly due to the targetStage file returns the mount path instead of the remote path as for .command.run, etc.
In this PR, I have fixed the error by returning he remote version of the stage file instead of the localone when requesting the targetStageFile.