Skip to content

Add nextflow launch command#6491

Merged
pditommaso merged 32 commits intomasterfrom
nextflow-launch
Oct 21, 2025
Merged

Add nextflow launch command#6491
pditommaso merged 32 commits intomasterfrom
nextflow-launch

Conversation

@jorgee
Copy link
Contributor

@jorgee jorgee commented Oct 17, 2025

Summary

This PR introduces the nextflow launch command, enabling users to submit and monitor pipeline executions directly to Seqera Platform from the command line. This feature streamlines the workflow for launching remote pipeline runs without needing to use the Seqera Platform web interface.

Based on the original work by @ewels in ewels#4, this implementation has been refactored with common base classes and enhanced with real-time log streaming.

Key Features

New nextflow launch Command

Launch a workflow in Seqera Platform with a simple command:

nextflow launch nextflow-io/hello

Command Syntax:

nextflow launch [options] <pipeline>

Available Options:

  • -workspace <name> - Workspace name for execution
  • -compute-env <name> - Compute environment name (defaults to primary)
  • -name <name> - Assign a mnemonic name to the pipeline run
  • -w, -work-dir <path> - Directory where intermediate result files are stored
  • -r, -revision <revision> - Git branch, tag, or commit SHA to run
  • -profile <profile> - Configuration profile to use
  • -c, -config <file> - Add configuration file(s) to the configuration set
  • -params-file <file> - Load script parameters from a JSON/YAML file
  • -entry <name> - Entry workflow name to be executed
  • -resume [session] - Resume execution using cached results
  • -latest - Pull latest changes before running
  • -stub-run, -stub - Execute with command stubs instead of actual scripts
  • -main-script <file> - Specify the script file to execute
  • --<param>=<value> - Set pipeline parameters (e.g., --outdir results)

Examples:

# Launch a pipeline with default settings
nextflow launch nextflow-io/hello

# Launch with specific revision and profile
nextflow launch nextflow-io/rnaseq -r v3.0 -profile docker

# Launch with custom parameters and workspace
nextflow launch my-org/pipeline --input data.csv --outdir results -workspace my-workspace

# Launch with resume and specific compute environment
nextflow launch nextflow-io/hello -resume -compute-env aws-batch

# Launch with params file and custom config
nextflow launch my-org/pipeline -params-file params.json -c custom.config

Real-Time Log Streaming

After launching a pipeline, the command automatically:

  • Streams execution logs to the terminal in real-time
  • Shows workflow progress with animated status indicators
  • Displays the final execution status (SUCCEEDED, FAILED, etc.)
  • Provides the Seqera Platform tracking URL for detailed monitoring

Configuration Integration

The launch command integrates with existing Seqera Platform configuration settings established via nextflow auth login.

New Config Option:

tower.computeEnvId
: The compute environment ID in your Seqera Platform account used to launch pipelines. When set, this takes precedence over the -compute-env flag.
: Type: String
: Default: Primary compute environment in the selected workspace

Example Configuration:

tower {
    enabled = true
    endpoint = 'https://api.cloud.seqera.io'
    accessToken = 'your-access-token'
    workspaceId = '123456789'
    computeEnvId = 'abc123xyz'  // New option
}

The tower.computeEnvId setting is automatically configured when using nextflow auth config to set your primary compute environment.

Workflow

  1. User invokes nextflow launch <pipeline>
  2. Command validates authentication (requires prior nextflow auth login)
  3. Pipeline URL is resolved (supports GitHub shortcuts like owner/repo)
  4. Workspace and compute environment are resolved from flags or config
  5. Configuration files and parameters are parsed and merged
  6. Launch request is submitted to Seqera Platform API
  7. Workflow details are fetched and displayed
  8. Real-time log polling begins, streaming output to terminal
  9. Final workflow status is displayed with tracking URL

Prerequisites

Before using nextflow launch, users must:

  1. Authenticate with Seqera Platform:

    nextflow auth login
  2. (Optional) Configure default workspace and compute environment:

    nextflow auth config

Related Work


🤖 Generated with Claude Code

@netlify
Copy link

netlify bot commented Oct 17, 2025

Deploy Preview for nextflow-docs-staging canceled.

Name Link
🔨 Latest commit ab6f3db
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/68f78d671e35cc00087774b0

@jorgee
Copy link
Contributor Author

jorgee commented Oct 17, 2025

@ewels, for the compute environments, I see when a user selects the environment, it is not configuring the Nextflow config; it makes a change to the platform CE to make it primary. I see it as very intrusive with side effects for the rest of the users in the workspace. Moreover, it is only possible if the user has grants for this. I got a 403 error when I was trying to select a non-primary CE in a workspace different than my personal one.

I think it is better not to configure the environment in this way. We could add an extra config for this (despite it will only be used by the launch command) or do not configure the environment in config and rely on the -compute-env flag and prompt during the launch if no flag and no primary CE is set.

@ewels
Copy link
Member

ewels commented Oct 17, 2025

Yes I wasn't very happy with the solution either, but I wanted to make the process as smooth as possible. I explicitly do not want to mess with the user's environment variables and definitely don't want to force them to supply the compute environment ID on the command line.

How about we make a new config scope tower.computeEnvironmentId to match tower.workspaceId? The reason I took the route of setting as primary was because this config attribute doesn't exist. But I confess I never thought of the fact that we could just add it. Then if it's not set we fall back to the primary and if there's no primary we can exit with an error.

Ported from ewels#4 by @jorgee

Co-authored-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: Phil Ewels <phil.ewels@seqera.io>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee
Copy link
Contributor Author

jorgee commented Oct 17, 2025

In my opinion , having tower.computeEnvironmentId is better than modifying the CE primary. @bentsherman what do you think about it?

@bentsherman
Copy link
Member

The config option sounds like a good approach, let's do it

Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee jorgee marked this pull request as ready for review October 20, 2025 09:31
@jorgee
Copy link
Contributor Author

jorgee commented Oct 20, 2025

Added the config option for compute environment and removing the primary set.

@jorgee jorgee marked this pull request as draft October 20, 2025 09:46
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@jorgee
Copy link
Contributor Author

jorgee commented Oct 20, 2025

Initial documentation added.

@jorgee jorgee marked this pull request as ready for review October 20, 2025 11:38
@jorgee jorgee requested a review from a team as a code owner October 20, 2025 11:38
@jorgee
Copy link
Contributor Author

jorgee commented Oct 20, 2025

@claude review

@claude
Copy link

claude bot commented Oct 20, 2025

Claude finished @jorgee's task —— View job


Code Review Progress

  • Read and analyze all changed files
  • Review core launch command implementation
  • Review Tower integration and configuration
  • Review spinner utility implementation
  • Review test coverage and quality
  • Review documentation changes
  • Provide comprehensive feedback

Code Review Feedback

This is an excellent implementation of the nextflow launch command! The code is well-structured, follows good design patterns, and includes comprehensive testing. Here are my detailed findings:

Strengths

  1. Clean Architecture: Good separation of concerns with CmdLaunch handling CLI parsing and LaunchCommandImpl handling the business logic.

  2. Robust Parameter Handling: Excellent support for JSON/YAML parameter files, nested parameters, and kebab-case to camelCase conversion in LaunchCommandImpl.groovy:841-892.

  3. Comprehensive Configuration: The new tower.computeEnvId config option in TowerConfig.groovy:66-69 properly addresses the discussion about avoiding primary CE modification.

  4. Great User Experience:

    • Beautiful colored output with proper branding in LaunchCommandImpl.groovy:323-345
    • Real-time log polling with graceful shutdown handling
    • Informative error messages and validation
  5. Spinner Implementation: Well-designed SpinnerUtil.groovy with multiple animation modes, proper thread management, and ANSI color support.

  6. Security Conscious: Proper input validation, rejects local paths, and uses secure HTTP clients.

  7. Excellent Test Coverage: Comprehensive tests covering edge cases, error conditions, and parameter parsing scenarios.

🔍 Areas for Improvement

Minor Issues

  1. Hardcoded Constants: Some magic numbers could be named constants:

    // LaunchCommandImpl.groovy:556-567
    static final int LOG_POLL_INTERVAL_MS = 2000
    static final int LOG_GRACE_PERIOD_MS = 5000
    static final int LOG_SLEEP_INTERVAL_MS = 100

    Already implemented correctly

  2. Error Message Consistency: In LaunchCommandImpl.groovy:149, consider using a more specific error message format that matches other Nextflow commands.

  3. Thread Safety: The spinner implementation correctly uses AtomicBoolean for thread-safe communication between the main thread and spinner thread.

Documentation

  1. CLI Documentation: The documentation in docs/reference/cli.md:760-830 is clear but could benefit from more examples showing different parameter passing methods.

🏗️ Architecture Review

  1. Plugin Architecture: Excellent use of Nextflow's plugin system with proper extension points.

  2. Base Class Design: BaseCommandImpl provides good code reuse between auth and launch commands.

  3. Configuration Integration: Proper integration with Nextflow's config system and the new computeEnvId option.

  4. API Client: Clean HTTP client implementation with proper timeout handling and error responses.

🧪 Testing

The test suite is comprehensive and covers:

  • Pipeline validation (local vs remote)
  • Parameter file parsing (JSON/YAML)
  • Configuration file handling
  • Error conditions and edge cases
  • Launch context initialization

📚 Code Quality

  • Proper use of Groovy idioms and @CompileStatic annotations
  • Good logging with appropriate levels
  • Clean separation of display logic with ANSI color handling
  • Thread-safe implementation for concurrent operations

🎯 Overall Assessment

This is a high-quality implementation that follows Nextflow's coding standards and architectural patterns. The feature is well-designed, thoroughly tested, and provides excellent user experience. The code is production-ready.

Recommendation: ✅ APPROVE

The implementation successfully addresses all the requirements discussed in the PR comments, particularly the compute environment configuration approach that was agreed upon by the team.


Copy link
Collaborator

@christopher-hakkaart christopher-hakkaart left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding suggestions to standardize on what the cli option does rather than what the user needs to do.

@christopher-hakkaart
Copy link
Collaborator

Also, launch (and auth) should both be added to docs/migrations/25-10.md. It could be one or two sections?

pditommaso and others added 6 commits October 21, 2025 09:35
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
pditommaso and others added 11 commits October 21, 2025 09:37
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-authored-by: Chris Hakkaart <chris.hakkaart@seqera.io>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso

This comment was marked as outdated.

@pditommaso

This comment was marked as outdated.

jorgee and others added 7 commits October 21, 2025 10:55
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
---------

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
…es (#6495)

This commit addresses two issues stemming from the recent config.schema
to config.spec package rename in #6485:

1. End-to-End Test Infrastructure Fix
    - Fixed test-e2e/run.sh to use the proper `installScratch` Makefile target
    - Changed from separate `make assemble` + `make install` to combined
      `make releaseInfo assemble installScratch`
    - The `installScratch` target (BUILD_PACK=1) is specifically designed for
      creating scratch container images used in the e2e test pipeline
    - This ensures the Nextflow runtime is correctly prepared for containerization
      and deployment to the Seqera Platform test environments

2. Backward Compatibility
    - Added deprecated type aliases in the old `nextflow.config.schema` package
    - Prevents `NoClassDefFoundError: nextflow/config/schema/ConfigScope` when
      loading plugins compiled against older Nextflow versions
    - Added deprecated classes: ConfigScope, ConfigOption, ScopeName, PlaceholderName
    - All deprecated classes delegate to their new counterparts in config.spec
    - Includes @deprecated annotations with Javadoc pointing to new locations

The e2e test fixes ensure that the automated testing pipeline against Seqera
Platform staging and production environments works correctly, while the
compatibility aliases give plugin developers time to migrate to the new
package structure.

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso pditommaso requested a review from a team as a code owner October 21, 2025 09:02
Signed-off-by: jorgee <jorge.ejarque@seqera.io>
@pditommaso pditommaso merged commit 7c8fa3d into master Oct 21, 2025
23 checks passed
@pditommaso pditommaso deleted the nextflow-launch branch October 21, 2025 14:17
@bentsherman bentsherman added this to the 25.10 milestone Oct 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants