Digging into Contrastive Learning for Robust Depth Estimation with Diffusion Models

Jiyuan Wang¹•Chunyu lin¹•Lang Nie¹•Shuwei Shao²

¹Beijingjiaotong University ¹Beihang University

ACM MM 2024

• release code for training and testing
• train/validate data release(Please refer to Weatherdepth)
• finish the ReadMe
• !NEW! test data release(Dense dataset are available)
• !NEW! model release (D4RD, D4RD+ will are available at huggingface)
• realease the introduction video for the paper

To make the reproduction easy, we provide the condapack package of our environment at here

We implement our method on MonoViT, Mono-Diffusion, and WeatherDepth baselines. If the tar.gz file is not adapted to your machine, you can refer to:

• MonoViT(3DV2022)
• MonoDiffusion(TCSVT2024)
• WeatherDepth(ICRA2024)

The training and inference code was tested on:

• Ubuntu 18.04 LTS, Python 3.8.8, CUDA 11.3, GeForce RTX 3090 (pip, Conda)
• Ubuntu 16.04 LTS, Python 3.7.15, CUDA 10.0, GeForce RTX 2080Ti (pip, Conda)

For WeatherDepth dataset Training, please refer to WeatherDepth. The training data is exactly the same as theirs.

For KITTI-C dataset Training, please refer to ECDepth. The KITTI-C dataset download and arrangement are basically the same as theirs. We also provide the this dataset image/depth GT at here

For DrivingStereo dataset Testing, please refer to drivingstereo-website.

For Dense dataset Testing, please download the tar.gz file from here and untar it. The dataset will be arranged as:

Path-to-your-Dense-Datset > tree
.
├── gt_depths.npy
└── snowy
    ├── 2018-02-04_11-14-31_00200.png
    ├── 2018-02-04_11-14-31_00400.png
    ├── 2018-02-04_11-18-24_00100.png
    ├── 2018-02-04_11-20-41_00000.png
    ...

Use the scripts below to inference and evaluate the model:

python Evaluate -lwf [Pretrained-Model-Path] --eval_split [Test-Split] --width 640 --height 192 --net_type vit --twt --ud --das -uC --ec

The test splits can be choose from: stereos(Sunny subset of Drivingstereo Dataset); stereoc(Cloudy subset of Drivingstereo Dataset);stereof(Foggy subset of Drivingstereo Dataset);stereor(Rainy subset of Drivingstereo Dataset);dense(Snowy data in Dense dataset)

The teacher model is avalible at ...

@inproceedings{Wang_2024, series={MM ’24},
   title={Digging into Contrastive Learning for Robust Depth Estimation with Diffusion Models},
   url={http://dx.doi.org/10.1145/3664647.3681168},
   DOI={10.1145/3664647.3681168},
   booktitle={Proceedings of the 32nd ACM International Conference on Multimedia},
   publisher={ACM},
   author={Wang, Jiyuan and Lin, Chunyu and Nie, Lang and Liao, Kang and Shao, Shuwei and Zhao, Yao},
   year={2024},
   month=oct, pages={4129–4137},
   collection={MM ’24} }

This project is licensed under the MIT License - see the LICENSE file for details. The code is based on the MonoViT , Monodiffusion and WeatherDepth repositories. We thank the authors for their contributions. The data is based on the KITTI, Dense, and DrivingStereo datasets. We thank the authors for their contributions. If you have any questions, please feel free to contact us with issues or email.