Summary

Add user-friendly support for enabling lightweight worker packages in dds-submit for any RMS plugin, not just localhost.

Background

Currently, lightweight worker packages are automatically enabled only for the localhost RMS plugin. However, when DDS is pre-installed on worker nodes (common in HPC environments), users can benefit from lightweight packages (~50KB instead of ~15MB) with any RMS plugin, providing 3x faster deployment speed.

Requested by: Alice production (Lubos Krcal)

Current Issues

1. Inefficiency: Even when DDS_COMMANDER_BIN_LOCATION and DDS_COMMANDER_LIBS_LOCATION are set on SLURM/SSH/PBS worker nodes, the commander still creates full worker packages (~15MB) that get ignored at runtime.

2. Limited scope: Automatic lightweight mode only works with localhost plugin.

3. No user control: Users cannot manually enable lightweight mode for other RMS plugins.

Proposed Solution

1. Add --lightweight command-line option

dds-submit -r slurm -n 10 --slots 8 --lightweight
dds-submit -r ssh -c hosts.cfg --lightweight

2. Add DDS_LIGHTWEIGHT_PACKAGE environment variable

# Enable globally
export DDS_LIGHTWEIGHT_PACKAGE=1
dds-submit -r slurm -n 10 --slots 8  # Uses lightweight automatically

# Accepts: 1, true, yes, on (case-insensitive)
# Command-line option takes precedence over environment variable

Benefits

• 3x faster deployment speed
• 300x smaller packages (~50KB vs ~15MB)
• Reduced network bandwidth usage
• User-friendly control via CLI option or environment variable
• Universal support for all RMS plugins (SLURM, SSH, PBS, LSF)
• CI/CD friendly for automated workflows
• Backward compatible - no impact on existing usage

Implementation Details

• Add ESubmitRequestFlags::enable_lightweight flag
• Modify ConnectionManager::_createWnPkg() to check the flag
• Update dds-submit options parsing
• Add environment variable support with precedence rules
• Update documentation and examples

Requirements

For lightweight packages to work:

• DDS must be pre-installed on worker nodes
• DDS_COMMANDER_BIN_LOCATION and DDS_COMMANDER_LIBS_LOCATION must be set
• dds-agent must be executable at the specified location

Use Cases

1. HPC clusters with DDS pre-installed via module system
2. Containerized environments with DDS in the container image
3. CI/CD pipelines with consistent DDS installations
4. Development environments with shared DDS installations