Has this been tested?

Do you think there's a real need for this log? or it's just for consistency with other executors?

There are cases where things get printed to .command.log but not .command.out / .command.err. I ran into these cases during my PhD work. But it would also make the nextflow / k8s experience more consistent with other executors.

I think these changes are correct, but it wasn't working during the full test because kuberun wasn't using my development version of Nextflow on the k8s cluster. I assume the autoMountHostPaths allows you to bypass that on a local k8s cluster so I will try that.

Okay, I figured out how to test with k8s like you showed, but I hit some weird errors. So I backed up to master, tried again, and got this error:

 ./launch.sh -c k8s.config run hello
N E X T F L O W  ~  version 22.05.0-SNAPSHOT
Launching `https://github.com/nextflow-io/hello` [mighty_goldwasser] DSL2 - revision: 4eab81bd42 [master]
executor >  k8s (4)
executor >  k8s (4)
[77/d802dd] process > sayHello (1) [ 75%] 3 of 4, failed: 3
Error executing process > 'sayHello (3)'

Caused by:
  Process `sayHello (3)` terminated for an unknown reason -- Likely it has been terminated by the external system

Command executed:

  echo 'Hello world!'

Command exit status:
  -

Command output:
  (empty)

Command wrapper:
  /bin/bash: .command.run: No such file or directory

Work dir:
  /home/bent/workspace/nextflow-io_nextflow/work/eb/7880bd01e64a40002f55f7ade78c6b

However, I don't see anything wrong with the pod yaml:

apiVersion: v1
kind: Pod
metadata:
  name: nf-eb7880bd01e64a40002f55f7ade78c6b
  namespace: tower-nf
  labels: {app: nextflow, taskName: sayHello_3, sessionId: uuid-0b909bb8-de05-49e2-99b1-d4d07b3aad1c,
    processName: sayHello, runName: mighty_goldwasser}
spec:
  restartPolicy: Never
  containers:
  - name: nf-eb7880bd01e64a40002f55f7ade78c6b
    image: quay.io/nextflow/bash
    command: [/bin/bash, -ue, /home/bent/workspace/nextflow-io_nextflow/work/eb/7880bd01e64a40002f55f7ade78c6b/.command.run]
    resources:
      requests: {cpu: 1}
      limits: {cpu: 1}
    volumeMounts:
    - {name: vol-1, mountPath: /home/bent/workspace/nextflow-io_nextflow/work/eb/7880bd01e64a40002f55f7ade78c6b}
  volumes:
  - name: vol-1
    hostPath: {path: /home/bent/workspace/nextflow-io_nextflow/work/eb/7880bd01e64a40002f55f7ade78c6b}

It says it can't find .command.run but the work dir is mounted to the pod and .command.run is there.

It says .command.log: No such file or directory

/Users/pditommaso/Projects/nextflow/work/49/c3e72cd5686f8a328e553a0b482619/.command.run 2>&1 | tee /Users/pditommaso/Projects/nextflow/work/49/c3e72cd5686f8a328e553a0b482619/.command.log: No such file or directory

bit weird tho

ok, found the reasons, the pipe concatenation should be executed as a subshell. But still not sure, not see a big value to add this log because this is running within the container.

it only helps when the .command.run crashes .. (that the reason why you want to add :))

I believe the original motivation was that beforeScript and afterScript are only saved to .command.log. See the related issue for details.

Even if it's a small improvement, why not do it? The awsbatch executor does the same thing for a similar reason:

Also, thanks for the fix. I thought it was already executing in a shell but I forgot about the -c option.

Even if it's a small improvement, why not do it?

Well, mainly to prevent the creation of files, but I agree it can be useful to debug the launcher script and it's consistent with the existing pattern.

Can you please have a look to the failing tests and then I'll merge it. tx!

@pditommaso This PR is old so I updated it and cleaned it up. It is ready to merge whenever you want.

I believe fetching the log over the stream it's better because it allows capturing the error output even when the .command.log cannot be accessed (think Fusion enable to mount the bucket for example).

Likely the best would be to implement both approaches (.command.log and fallback to the stream if missing), but I think it's overkill without a real need.

Closing without merging. We can reconsider it in the future.

Fetching the log in this manner does not work when the pod is deleted immediately, for example:

WARN: Failed to copy log for job nf-b29b6c384d74ee3b41844239686e8a6c

Interestingly this particular failure happens because the file already exists:

java.nio.file.FileAlreadyExistsException: /scidev-eu-west-2/scratch/4Yy5aUWbYwTXhy/b2/9b6c384d74ee3b41844239686e8a6c/.command.log

But I checked this file and it is empty. Does Fusion create this file automatically?

The pod should not be deleted