HiP-AD

HiP-AD: Hierarchical and Multi-Granularity Planning with Deformable Attention for Autonomous Driving in a Single Decoder

ICCV 2025

• We propose a multi-granularity planning query representation that integrates various characteristics of waypoints to enhance the diversity and receptive field of the planning trajectory, enabling additional supervision and precise control in a closed-loop system.
• We propose a planning deformable attention mechanism that explicitly utilizes the geometric context of the planning trajectory, enabling dynamic retrieving image features from the physical neighborhoods of waypoints to learn sparse scene representations.
• We propose a unified decoder where a comprehensive interaction is iteratively conducted on planning-perception and planning-images, making an effective exploration of end-to-end driving in both BEV and perspective view, respectively.

a 3-minute YouTube video showcasing closed-test on CARLA

Closed-loop evaluation results on Bench2Drive dataset

1. Installation
2. Data Preparation
3. Training
4. Evaluation

1. create environment

conda create -n hipad python=3.8 -y
conda activate hipad
pip install --upgrade pip
pip install torch==1.13.0+cu117 torchvision==0.14.0+cu117 torchaudio==0.13.0 --extra-index-url https://download.pytorch.org/whl/cu117
pip install -r requirements.txt

2. compile the deformable_aggregation op

cd projects/mmdet3d_plugin/ops
python setup.py develop
cd ../../../

1. download the Bench2Drive [Base] dataset with the official configuration.

ln -s bench2drive_root ./data/bench2drive

2. create pkl files from data converter.

python ./tools/data_converter/bench2drive_converter.py

3. create npy files by using kmeans clustering.

bash ./tools/kmeans/kemans.sh

4. check the file structure as follows:

HiP-AD/
├── assets/
├── bench2drive/
├── ckpts/
├── data/
│   ├── bench2drive
│   │   ├── maps/
│   │   ├── v1/
│   ├── infos
│   │   ├── b2d_infos_train.pkl
│   │   ├── b2d_infos_val.pkl
│   │   ├── b2d_map_infos.pkl
│   ├── kmeans
│   │   ├── b2d_det_900.npy
│   │   ├── b2d_map_100.npy
│   │   ├── b2d_motion_6.npy
│   │   ├── b2d_plan_spat_6x8_2m.npy
│   │   ├── b2d_plan_spat_6x8_5m.npy
│   ├── splits
├── projects/
├── tools/

1. download pretrained ResNet50 backbone weight

mkdir ckpts && cd ckpts
wget https://download.pytorch.org/models/resnet50-19c8e357.pth

2. change the project_dir to your path in projects/configs/hipad_b2d_stage1.py and projects/configs/hipad_b2d_stage2.py

project_dir = "/opt/data/private/project/HiP-AD"

3. training HiP-AD: stage1 and stage2 cost about 14h and 46h, respectively, on 8 Nvidia 4090 GPUs

bash ./tools/dist_train.sh ./configs/hipad_b2d_stage1.py 8 --no-validate
bash ./tools/dist_train.sh ./configs/hipad_b2d_stage2.py 8 --no-validate

Open-Loop

bash ./tools/dist_test.sh./projects/configs/hipad_b2d_stage2.py ./work_dirs/b2d_r50_stage2/latest.pth 8 --eval bbox

Closed-Loop

1. install CARLA 0.9.15

mkdir -p carla && cd carla
wget https://carla-releases.s3.us-east-005.backblazeb2.com/Linux/CARLA_0.9.15.tar.gz
wget https://carla-releases.s3.us-east-005.backblazeb2.com/Linux/AdditionalMaps_0.9.15.tar.gz

tar -xvf CARLA_0.9.15.tar.gz
cd Import && tar -xvf ../AdditionalMaps_0.9.15.tar.gz
cd .. && bash ImportAssets.sh

export CARLA_ROOT=YOUR_CARLA_PATH
echo "$CARLA_ROOT/PythonAPI/carla/dist/carla-0.9.15-py3.7-linux-x86_64.egg" >> $CONDA_PREFIX/lib/python3.8/site-packages/carla.pth

2. create multi-route splits

python ./bench2drive/tools/split_xml.py

3. change the project_dir to your path in ./projects/configs/hipad_b2d_stage2.py

project_dir = "/opt/data/private/project/HiP-AD"

4. change the WORK_DIR and CARLA_ROOT to your path in ./bench2drive/leaderboard/scripts/run_evaluation.sh

export WORK_DIR=/opt/data/private/project/HiP-AD # your work dir
export CARLA_ROOT=/opt/data/private/project/Carla # your carla path

5. download pretrained weights and change the weight path of TEAM_CONFIG in ./bench2drive/leaderboard/scripts/run_evaluation_multi_hipad.sh

6. run closed-loop evaluation and check outputs in ./evaluation/hipad_b2d_stage2

bash ./bench2drive/leaderboard/scripts/run_evaluation_multi_hipad.sh

7. evaluate the closed-loop results

python ./bench2drive/tools/statistic_route_json.py

8. create videos (optional)

python ./bench2drive/tools/generate_video.py

Thanks to these excellent open-source works:

UniAD, VAD, SparseDrive, SimLingo, Bench2Drive, MMDetection3D

If this work is helpful to you, we would appreciate it if you could star and cite it.

@article{tang2025hipad,
  title={HiP-AD: Hierarchical and Multi-Granularity Planning with Deformable Attention for Autonomous Driving in a Single Decoder},
  author={Yingqi Tang and Zhuoran Xu and Zhaotie Meng and Erkang Cheng},
  journal={arXiv preprint arXiv:2503.08612},
  year={2025}
}