• [2025-09-18] 🎉 Our paper has been accepted to NeurIPS 2025 as a poster! 🎉

• 🔗 Unified Framework: Integrates world modeling and trajectory planning in a single architecture
• 🧠 Human-like Anticipation: Mimics anticipatory perception through collaborative state-action prediction
• ⚡ Efficient Video Forecasting: Dynamic parallel token generation with context-guided tokenizer
• 📊 State-of-the-Art Performance: Exceeds existing methods on benchmark datasets

• Release arXiv paper
• Release training and evaluation code
• Release model weights
• Provide guidelines

This guide provides step-by-step instructions for fine-tuning and evaluating the Policy-World-Model (PWM) on NuScenes and NavSim datasets. Ensure you have completed the environment setup and data preparation before proceeding.

First, clone the repository and navigate to the project root directory:

git clone https://github.com/6550Zhao/Policy-World-Model.git
cd Policy-World-Model  # Enter the project folder (replace with your actual path if needed)

Create and activate the environment with the provided environment.yaml:

# Create environment from the configuration file
conda env create -f environment.yaml

# Activate the PWM environment (name: pwm)
conda activate pwm

Download the following resources to run fine-tuning/evaluation:

Pre-trained weights: Download from the official website (Download).

Dataset files: Download dataset files from the specified source (for convenience, some required additional dataset files are available for download here: Download).

Organize your dataset folder as follows (ensure the path matches the configuration in yaml files):

dataset/
├── nuscenes/
│   ├── maps/
│   ├── samples/
│   ├── sweeps/
│   └── ominidrive/ # Download from the provided link
└── navsim/
    ├── maps/
    ├── cache/
    ├── navsim_logs/
    ├── sensor_blobs/
    ├── nuplan_img_logs/ # Download from the provided link
    ├── nuplan_scene_blobs/
    │   ├── 10hz_train/
    │   │   ├── 2021.05.12.19.36.12_veh-35_00005_00204/
    │   │   ├── 2021.05.12.19.36.12_veh-35_00215_00405/
    │   │   └── ... (other training scenes)
    │   ├── 10hz_test/
    │   └── 10hz_val/

4.1 Evaluate on NuScenes Modify the configuration file to enable evaluation mode: Open configs/sft_nuscenes/nuscenes.yaml Set experiment.eval_only = True (ensure no extra spaces or syntax errors) Run the evaluation script:

bash scripts/finetune/nuscenes/run_sft_nusc_no_ego_baseline.sh

4.2 Evaluate on NavSim Modify the configuration file to enable evaluation mode: Open configs/sft_nuscenes/nuscenes.yaml Set experiment.eval_only = True Run the evaluation script:

bash scripts/finetune/navsim/run_sft_navsim_baseline.sh

5.1 Fine-tune on NuScenes Modify the configuration file to enable training mode: Open configs/sft_nuscenes/nuscenes.yaml Set experiment.eval_only = False Start fine-tuning:

bash scripts/finetune/nuscenes/run_sft_nusc_no_ego_baseline.sh

5.2 Fine-tune on NavSim Modify the configuration file to enable training mode: Open configs/sft_nuscenes/nuscenes.yaml Set experiment.eval_only = False Start fine-tuning:

bash scripts/finetune/navsim/run_sft_navsim_baseline.sh

We thank the reviewers and the research community for their valuable feedback and support.

If you find our work useful, please cite:

@inproceedings{zhao2025pwm,
  title={From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction},
  author={Zhao, Zhida and Fu, Talas and Wang, Yifan and Wang, Lijun and Lu, Huchuan},
  booktitle={Advances in Neural Information Processing Systems},
  year={2025}
}

This project is licensed under the MIT License - see the LICENSE file for details.