Revert Plugins.init() at run command to original location#6568
Revert Plugins.init() at run command to original location#6568bentsherman merged 3 commits intomasterfrom
Conversation
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
✅ Deploy Preview for nextflow-docs-staging canceled.
|
|
Before we do this, I would like to understand why the params-file can be an S3 path but not an Azure path It doesn't seem like either of these use cases were ever intentionally supported, considering that the params file is loaded before the config, which means the params file shouldn't be able to access remote storage |
|
It never worked for google storage either: #5390 |
|
The difference is in the path factory. For S3, nextflow gets the config from Global.config. For Google and Azure, it gets from Global.session.config, and they return a exception if session is null. At the point of loading the params with S3, the Global.config is null, but as it is not failing, it just create an empty Map and works with the env vars, sso, instace role, or whatever is configured in the process running the run command. We can do the same for the other cases and it will work. However, I see a side effect for S3 and Azure. File systems are storing the client used to access the cloud storage. In these cases, file systems are set per bucket and there is a hashmap to cache file systems based on bucket name. So, if there are specific configurations in the nextflow config file for the bucket used in the params file, they will never be applied. It will not happen in Google because the key file system cache map includes the configuration. |
|
Quick patch for 25.10.1 and separate task trying to have a common approach to this problem? |
…i fast] Co-authored-by: Ben Sherman <bentshermann@gmail.com> Signed-off-by: Jorge Ejarque <jorgee@users.noreply.github.com>
--------- Signed-off-by: jorgee <jorge.ejarque@seqera.io>
close #6566
Commit 9e4c975 moved the Plugins.init() later in the code- Plugins.init() initializes the plugins manager, so autoStartIfMissing call done before loading the configuration will fail. It includes getting the Repository providers implemented in plugins or downloading remote param-files.
This PR moves again the Plugins.init() to begining of run command. This change doesn't affect secrets loading (tested in platform with
bentsherman/nf-tests -r plugin-secrets). The first time that the config is loaded, it uses the EmptySecretsProvider, it doesn't affect if a plugin has been loaded before.