Conversation
|
@phil-opp , I am struggling to understand this? Is this change ok? |
|
I think the proper fix is to use This PR places uses the same working dir for all dataflows run by the daemon, which can lead to conflicts when multiple dataflows are run in parallel. |
The working directory deliberatly included the session ID in distributed settings to avoid conflicts when multiple dataflows are run on the same daemon. The session ID is a unique identifier created for each dataflow.yaml file. By using this ID as part of the working directory, we can ensure that no other dataflow is running in the same directory. Sharing a working directory between multiple distinct dataflows could easily lead to unexpected behavior, e.g. by overwriting files of the other dataflow run. So it's better to avoid this. By using the session ID, the daemon will still be able to reuse build artifacts etc of previous dataflow runs. So incremental compilation still works. Reverts #1064
|
I saw that you're about to create a new release, so I merged #1065 to revert this change before you cut the new release. |
|
I opened #1066 with the proper fix (i.e. create the directories if they don't exist). Could you try whether this fixes your issue? (I'm open to changing the working directoy name if you don't like the |
The current working folder when using distributed dora-daemon is a folder that does not exist making it impossible to spawn build command.