UniDriveVLA

UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving

Yongkang Li^1,2, Lijun Zhou², Sixu Yan¹, Bencheng Liao¹, Tianyi Yan^2,3, Kaixin Xiong², Long Chen², Hongwei Xie², Bing Wang², Guang Chen², Hangjun Ye², Wenyu Liu¹, Haiyang Sun^†2, Xinggang Wang^✉1

¹Huazhong University of Science and Technology   ²Xiaomi EV   ³SKL-IOTSC, University of Macau

(†) Project Leader.   (✉) Corresponding Author.

April 3, 2026

• 🔥 [2026-04-03] We release the paper, training/inference code, and model weights of UniDriveVLA!

• Release paper
• Release code and training scripts
• Release model weights on HuggingFace
• Release nuScenes and Bench2Drive evaluation frameworks
• Release model on Navsim

• News
• Updates
• Table of Contents
• Abstract
• Architecture
• Getting Started
  • Data Preparation
  • Installation
  • VLM Pretraining
  • Training and Evaluation on nuScenes
  • Training and Evaluation on Bench2Drive
• Checkpoints
• Contact
• Acknowledgement
• Citation

Vision-Language-Action (VLA) models have recently emerged in autonomous driving, with the promise of leveraging rich world knowledge to improve the cognitive capabilities of driving systems. However, adapting such models for driving tasks currently faces a critical dilemma between spatial perception and semantic reasoning. Consequently, existing VLA systems are forced into suboptimal compromises: directly adopting 2D Vision-Language Models yields limited spatial perception, whereas enhancing them with 3D spatial representations often impairs the native reasoning capacity of VLMs. We argue that this dilemma largely stems from the coupled optimization of spatial perception and semantic reasoning within shared model parameters. To overcome this, we propose UniDriveVLA, a Unified Driving Vision-Language-Action model based on Mixture-of-Transformers that addresses the perception–reasoning conflict via expert decoupling. Specifically, it comprises three experts for driving understanding, scene perception, and action planning, which are coordinated through masked joint attention. In addition, we combine a sparse perception paradigm with a three-stage progressive training strategy to improve spatial perception while maintaining semantic reasoning capability. Extensive experiments show that UniDriveVLA achieves state-of-the-art performance in open-loop evaluation on nuScenes and closed-loop evaluation on Bench2Drive. Moreover, it demonstrates strong performance across a broad range of perception, prediction, and understanding tasks, including 3D detection, online mapping, motion forecasting, and driving-oriented VQA, highlighting its broad applicability as a unified model for autonomous driving.

UniDriveVLA adopts a Mixture-of-Transformers architecture with three specialized experts:

• Understanding Expert: Leverages a pre-trained 2D VLM (Qwen3-VL) for semantic scene comprehension and driving QA
• Perception Expert: Introduce sparse perception that extracts spatial priors from 2D VLM features, supporting detection, mapping, occupancy, and motion forecasting
• Planning Expert: Fuses VLM semantic features and spatial perception features to generate safe, precise trajectories

• Data Preparation
• Installation
• VLM Pretraining
• Training and Evaluation on nuScenes
• Training and Evaluation on Bench2Drive

nuScenes Open-Loop Results (ST-P3 metrics, without ego status)

Bench2Drive Closed-Loop Results

Perception Results on nuScenes val

All Model Weights (Stage 1 & 2 are intermediate checkpoints for progressive training; Stage 3 is the final model)

If you have any questions, please contact Yongkang Li via email (liyk@hust.edu.cn).

UniDriveVLA is built upon the following outstanding open-source works:

• Openpi — VLA training framework
• InternVLA-A1 — VLA model for robotic manipulation
• HiP-AD — Hierarchical planning for autonomous driving
• SparseDrive — Sparse 3D perception framework for autonomous driving
• Bench2Drive — Closed-loop evaluation in CARLA

If you find UniDriveVLA useful in your research or applications, please consider giving us a star 🌟 and citing it by the following BibTeX entry:

@article{li2026unidrivevla,
  title={UniDriveVLA: Unifying Understanding, Perception, and Action Planning for Autonomous Driving},
  author={Li, Yongkang and Zhou, Lijun and Yan, Sixu and Liao, Bencheng and Yan, Tianyi and Xiong, Kaixin and Chen, Long and Xie, Hongwei and Wang, Bing and Chen, Guang and Ye, Hangjun and Sun, Haiyang and Liu, Wenyu and Wang, Xinggang},
  journal={arXiv preprint arXiv:2604.02190},
  year={2026}
}