Update to plugin registry V1 by jorgee · Pull Request #6316 · nextflow-io/nextflow

jorgee · 2025-07-29T16:52:35Z

This PR includes two changes to integrate Nextflow with plugin registry updates:

Update plugins download to the plugin registry API v1
POST /plugins/collect -> GET /v1/plugins/dependencies
Update OciAwareFileDownloader:

Fix automatic redirection problems (changing protocol http-> https and getting the correct filename) by doing a manual redirection.
Refactor with Java HTTP client

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

netlify · 2025-07-29T16:52:42Z

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	`1731036`
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/68998afb5fb4990008c470ae
😎 Deploy Preview	https://deploy-preview-6316--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

jorgee · 2025-07-30T10:03:26Z

@pditommaso and @bentsherman
When doing this change, I had a doubt about how to manage the api path inside nextflow. Currently, we require to set the NXF_PLUGINS_INDEX_URL=<plugins-host>/api and Nextflow adds the path /plugins/collect to get the plugins.

For the changes for v1, I have added the 'v1' in the path to do not change the environment variable. However, there are other options to manage it:
1- NXF_PLUGINS_INDEX_URL=<plugins-host>/api/v1 only add /plugins/...
2- NXF_PLUGINS_INDEX_URL=<plugins-host> and nextflow add the full api path /api/v1/....

I think (2) makes more sense, but I would like to know your opinion.

The same is happening for the gradle plugin and the nf-plugin-template. Depending on what we include in the URL, we need to update the template, the plugin or both.

pditommaso · 2025-07-30T10:11:00Z

The "correct" way it to keep as parameter api root path ie. https://<HOST>/api (without slash) and append /v1/etc

jorgee · 2025-07-30T10:25:12Z

The "correct" way it to keep as parameter api root path ie. https://<HOST>/api (without slash) and append /v1/etc

Then, this PR does not require more changes and I will create a PR in the gradle plugin to support v1.

pditommaso · 2025-07-30T10:53:06Z

Worth adding a retry logic e.g

nextflow/modules/nextflow/src/main/groovy/nextflow/scm/RepositoryProvider.groovy

Line 497 in 9a70cdb

    
           safeApply(()-> httpClient.send(request, HttpResponse.BodyHandlers.ofByteArray()))

I've made a little library to wrap the FailSafe boilerplate into a reusable component, see here https://github.com/seqeralabs/libseqera/tree/master/lib-retry

jorgee · 2025-07-31T07:01:35Z

There is already a retry logic with FailSafe here. I see it is not using the common RetryConfig, defaults are hardcoded. I will modify to use it. Or do you prefer to replace this retry logic by lib-retry?

pditommaso · 2025-07-31T08:06:50Z

Maybe i'm missing something, what about these sends

nextflow/modules/nf-commons/src/main/nextflow/plugin/OciAwareFileDownloader.groovy

Line 66 in adc87cc

    
           HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString())

nextflow/modules/nf-commons/src/main/nextflow/plugin/OciAwareFileDownloader.groovy

Line 129 in adc87cc

    
           HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString())

jorgee · 2025-07-31T08:12:31Z

Ah! yes, you are right. I was just looking at the fetch part.

pditommaso · 2025-07-31T08:13:56Z

Then let's adopt the new lib consistently at least here

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

jorgee · 2025-07-31T12:40:03Z

I have created a common RetryableHttpClient that wraps the Java Http Client with the Sequera lib-retry lib. I have used it in the fetch call in HttpPluginRepository and the download calls in OciAwareFileDownloader. With this change, all calls are retried for the retriable error codes (the old fail-safe mechanism in fetch was not doing it).

The retry configuration for these calls is obtained from the common retry configuration nextflow.retryConfig using RetryConfig. As it is not in the nf-commons package and it depends on other nextflow module classes, I have created the IRetryConfig interface that is implemented by RetryConfig and I have also added a new method in ISession to get the common retry configuration.

I tried to use the lib-retry Retryable.Config interface instead of using the IRetryConfig, but RetryConfig was using nextflow.utils. Duration instead of Java Duration and I wasn't sure if changing the type could introduce side effects, so I discarded this option.

modules/nf-commons/src/main/nextflow/ISession.groovy

modules/nf-commons/src/main/nextflow/util/IRetryConfig.groovy

bentsherman

Please allow me to review the config classes before merging. I need to make sure everything will work with the config refactor in #6271

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

This reverts commit cbd58d4.

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

bentsherman

Everything looks good from my side. We ended up not involving the retry config since it isn't essential. Now we just need to test against the registry

## Key Changes **RetryConfig Enhancement & Movement:** - Move RetryConfig from nextflow to nf-commons module for better reusability - Add support for multiplier parameter in retry configuration - Replace navigate() dependency with static compilation-friendly getNestedConfig() helper - Add NXF_RETRY_POLICY_MULTIPLIER environment variable (default: 2.0) **HTTP Client Modernization:** - Replace custom HttpRetryableClient with external io.seqera:lib-httpx library - Update HttpPluginRepository to use HxClient for better reliability - Enhanced OciAwareFileDownloader with improved authentication and error handling **Documentation:** - Add NXF_PLUGINS_REGISTRY_URL and NXF_RETRY_POLICY_MULTIPLIER to env-vars.md - Document retry policy multiplier configuration for exponential backoff **Dependencies:** - Add io.seqera:lib-httpx:1.0.0 and io.seqera:lib-retry:1.2.0 - Remove custom HTTP retry implementation in favor of external libraries This refactoring improves the plugin system's HTTP infrastructure reliability while maintaining backward compatibility and adding new configuration flexibility. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

pditommaso · 2025-08-11T06:17:56Z

Latest Updates - Plugin HTTP Infrastructure Refactoring

Key Changes

RetryConfig Enhancement & Movement:

Move RetryConfig from nextflow to nf-commons module for better reusability
Add support for multiplier parameter in retry configuration
Replace navigate() dependency with static compilation-friendly getNestedConfig() helper
Add NXF_RETRY_POLICY_MULTIPLIER environment variable (default: 2.0)

HTTP Client Modernization:

Replace custom HttpRetryableClient with external io.seqera:lib-httpx library
Update HttpPluginRepository to use HxClient for better reliability
Enhanced OciAwareFileDownloader with improved authentication and error handling

Documentation:

Add NXF_PLUGINS_REGISTRY_URL and NXF_RETRY_POLICY_MULTIPLIER to env-vars.md
Document retry policy multiplier configuration for exponential backoff

Dependencies:

Add io.seqera:lib-httpx:1.0.0 and io.seqera:lib-retry:1.2.0
Remove custom HTTP retry implementation in favor of external libraries

This refactoring improves the plugin system's HTTP infrastructure reliability while maintaining backward compatibility and adding new configuration flexibility.

add registry v1 and OciAwareDownloader refactor with java http client

adc87cc

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

jorgee added 2 commits July 31, 2025 14:08

add fail common failsafe for plugins http calls

f4ce378

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

add IRetryConfig interface header and comment

33eb7c9

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

bentsherman reviewed Jul 31, 2025

View reviewed changes

modules/nf-commons/src/main/nextflow/ISession.groovy Outdated Show resolved Hide resolved

bentsherman reviewed Jul 31, 2025

View reviewed changes

modules/nf-commons/src/main/nextflow/util/IRetryConfig.groovy Show resolved Hide resolved

bentsherman requested changes Jul 31, 2025

View reviewed changes

bentsherman added the plugins label Aug 1, 2025

jorgee added 4 commits August 1, 2025 11:04

remove IRetryConfig and move RetryConfig to nf-commons

cbd58d4

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

Revert "remove IRetryConfig and move RetryConfig to nf-commons"

e6312a0

This reverts commit cbd58d4.

revert IRetryConfig and set Retryable defaults

624d3d8

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

[ci fast] clean unused import

bb22cce

Signed-off-by: jorgee <jorge.ejarque@seqera.io>

jorgee requested a review from bentsherman August 1, 2025 18:22

bentsherman approved these changes Aug 5, 2025

View reviewed changes

pditommaso and others added 2 commits August 8, 2025 23:46

Merge branch 'master' into update-plugin-registry-v1

a776144

pditommaso requested a review from a team as a code owner August 11, 2025 06:17

pditommaso merged commit 9b1d1b6 into master Aug 11, 2025
21 of 23 checks passed

pditommaso deleted the update-plugin-registry-v1 branch August 11, 2025 07:32

Conversation

jorgee commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

netlify bot commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for nextflow-docs-staging ready!

Uh oh!

jorgee commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pditommaso commented Jul 30, 2025

Uh oh!

jorgee commented Jul 30, 2025

Uh oh!

pditommaso commented Jul 30, 2025

Uh oh!

jorgee commented Jul 31, 2025

Uh oh!

pditommaso commented Jul 31, 2025

Uh oh!

jorgee commented Jul 31, 2025

Uh oh!

pditommaso commented Jul 31, 2025

Uh oh!

jorgee commented Jul 31, 2025

Uh oh!

Uh oh!

Uh oh!

bentsherman left a comment

Choose a reason for hiding this comment

Uh oh!

bentsherman left a comment

Choose a reason for hiding this comment

Uh oh!

pditommaso commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Latest Updates - Plugin HTTP Infrastructure Refactoring

Key Changes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jorgee commented Jul 29, 2025 •

edited

Loading

netlify bot commented Jul 29, 2025 •

edited

Loading

jorgee commented Jul 30, 2025 •

edited

Loading

pditommaso commented Aug 11, 2025 •

edited

Loading