Kun Huang, Fang-Lue Zhang*, Fangfang Zhang, Yu-Kun Lai, Paul Rosin, Neil A. Dodgson,

*Corresponding authors

Geometric estimation is required for scene understanding and analysis in panoramic 360° images. Current methods usually predict a single feature, such as depth or surface normal. These methods can lack robustness, especially when dealing with intricate textures or complex object surfaces. We introduce a novel multi-task learning (MTL) network that simultaneously estimates depth and surface normals from 360° images. Our first innovation is our MTL architecture, which enhances predictions for both tasks by integrating geometric information from depth and surface normal estimation, enabling a deeper understanding of 3D scene structure. Another innovation is our fusion module, which bridges the two tasks, allowing the network to learn shared representations that improve accuracy and robustness.

Our MTL model provides more accurate geometric estimations for 360° images compared to other methods, particularly in the red rectangle highlighted regions. The results are visualized as 3D point clouds, with both RGB data and color-coded surface normal maps.

Our network architecture. Our network consists of two branches: $B_{depth}$ (in blue) and $B_{normal}$ (in red), dedicated to depth and surface normal estimation, respectively. A fusion module (in green) is employed to fuse the feature maps between each encoder level of $B_{depth}$ and $B_{normal}$ and feed the fused features into the next encoder level. The fused features are also concatenated with the original depth or normal features and fed to the corresponding decoder blocks. The final depth and normal maps are predicted in a multi-scale manner.

Provide installation instructions for your project. Include any dependencies and commands needed to set up the project.

# Clone the repository
git clone https://github.com/huangkun101230/360MTLGeometricEstimation.git
cd 360MultitaskGeometricEstimation

# Install dependencies
conda env create -f conda_env.yml
conda activate mtl360

Please download our pretrained models, and save these models to "saved_models/models". To test on provided data in "./input_data"

python evaluate.py

The results will be saved at "./results/saved_models/"

For training our model, please modify the path in our dataset: For example, in datasets/dataset3D60.py, function gather_filepaths, change local="./input_data/" with your downloaded path

and run

python train.py

We mainly evaluate our method on 3D60 dataset and Structured3D dataset.

If you find this repository useful in your project, please cite the following work. :)

@article{huang2024multi,
  title={Multi-task Geometric Estimation of Depth and Surface Normal from Monocular 360 $\{$$\backslash$deg$\}$ Images},
  author={Huang, Kun and Zhang, Fang-Lue and Zhang, Fangfang and Lai, Yu-Kun and Rosin, Paul and Dodgson, Neil A},
  journal={arXiv preprint arXiv:2411.01749},
  year={2024}
}

This project is licensed under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license.

You are free to:

✅ Share — copy and redistribute the material in any medium or format

✅ Adapt — remix, transform, and build upon the material

Under the following terms:

🔗 Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made.

🚫 NonCommercial — You may not use the material for commercial purposes.

Note: For any commercial use or licensing inquiries, please contact the project maintainer.