StarVLA Training Efficiency Report

Motivation

We have noticed huge discrepancies between the training efficiency of StarVLA reported in the community feedback and our empirical observations. We report training efficiency measurements for StarVLA to provide actionable scaling guidance, focusing on common distributed-training bottlenecks (compute and communication).

Experimental setup

Unless otherwise specified:

• Model: StarVLA-GR00T
• Backbone: Qwen3-VL-4B
• Dataset: RoboCasa-GR1
• Hardware: NVIDIA A100 80GB
• Reported unit: wall-clock time per 100K optimization steps, including distributed communication and system overhead

Metrics

We distinguish two throughput notions:

1. Step throughput: seconds/step (lower is better)
2. Sample throughput: samples/s (higher is better)

Sample throughput is computed as:

$$ \text{samples/s} = \frac{\text{global batch}}{\text{seconds per step}} $$

This distinction matters because distributed scaling often decreases step throughput (more synchronization) while increasing sample throughput (larger global batch).

Single-node scaling: batch size sweep (8× A100)

Table below summarizes a single-node sweep varying the per-GPU batch size. We omit derived “24-hour” projections and focus on directly measured quantities and the implied sample throughput.

Figure: step latency vs. per-GPU batch (left), and sample throughput with GPU utilization (right).

Interpretation: Smaller per-GPU batches yield faster steps, while larger per-GPU batches improve sample throughput up to a point, then suffer from sharply increased step latency (reflecting increased per-step computation and reduced kernel efficiency).

Multi-node scaling: strong scaling vs. throughput scaling

Here we fix per-GPU batch size to 8 and scale the number of GPUs.

Figure: step latency (left) and sample throughput (right), with an ideal linear reference from the 8-GPU baseline.

Interpretation: Time per step increases mildly with more GPUs (non-trivial synchronization and scheduling overhead). However, sample throughput scales strongly with global batch, which is the relevant metric when the objective is to process a fixed amount of data quickly.

Note on cross-report throughput comparisons

Throughput numbers reported across different projects/papers can differ substantially due to hardware and training configuration, not only due to algorithmic or implementation differences.

Common sources of variation include:

• GPU type and system scale (e.g., A100 vs. H100, GPU count, interconnect)
• Batching (per-GPU batch and global batch), which directly changes samples/s and can also impact seconds/step via kernel efficiency
• Other settings (precision, parallelism/ZeRO stage, sequence length, image resolution, gradient checkpointing)

For fair comparisons, we recommend reporting (or matching) hardware + global batch, and interpreting cross-report throughput numbers with care.

Key takeaways

• Increasing GPU count tends to reduce step throughput due to synchronization overhead, but sample throughput can still scale strongly via larger global batch.
• On a single node, very large per-GPU batches may maximize utilization yet degrade step latency sharply; a moderate per-GPU batch often provides a better balance.
• For large-scale training, reporting both step-level and sample-level throughput is necessary to avoid misleading conclusions.