In this project we trained different state-of-the-art (SOTA) object detection models for the task of pothole detection and evaluated their performance under self-synthesized motion blur noise. This noise was generated using randomized kernel types and sizes to simulate real-world conditions where camera shake or motion artifacts may degrade image quality. The study provides insights into how motion blur affects detection accuracy and offers recommendations for improving the robustness of automated road maintenance systems.

Moreover, we created our own dataset where we classified the potholes into three severity levels: low, medium, and high. This classification was based on how dangerous a pothole is to cars and pedestrians, which can help prioritize road maintenance efforts and allocate resources more effectively. The dataset was annotated with bounding boxes and severity labels, enabling the models to learn to detect and classify potholes simultaneously.

• Files in the repository
• Installation Instructions
  • Libraries to Install
• How to Use
• Dataset
• Object Detection Models
  • Hyperparameter Tuning
• Training Results
• Motion Blur Noise
• Data Augmentations
• Post Augmentations Results
• Potholes Severity
• References

1. Get Anaconda with Python 3, follow the instructions according to your OS (Windows/Mac/Linux) at link
2. Install the basic packages using the provided environment.yml file by running: conda env create -f config/environment.yml which will create a new conda environment named deep_learn. You can use config/environment_no_cuda.yml for an environment that uses pytorch cpu version.
3. Alternatively, you can create a new environment and install packages from scratch: In Windows open Anaconda Prompt from the start menu, in Mac/Linux open the terminal and run conda create --name deep_learn. Full guide at link
4. To activate the environment, open the terminal (or Anaconda Prompt in Windows) and run conda activate deep_learn
5. Install the required libraries according to the table below (to search for a specific library and the corresponding command you can also look at link)

After installing the required libraries, you can run the main.ipynb notebook to follow through with the project results and anlysis.

If you wish to train the models and evaluate them your self you can run the notebooks under models_evaluation_with_noise which contain the training and evaluation process for the all the models we analysed with the motion blur noise.

You can also download our weights from the following link:

• Google Drive

  Just download the models.zip file and replace it with the existing ./data/models folder in the repository.

The optuna hyperparameters tuning can be run using the torchvision_models_train.ipynb notebook. The results of the hyperparameters tuning can be viewed using the optuna dashboard (see below).

You can check the pothole severity classification we added to the dataset in the models_evaluation_with_severity_levels.ipynb notebook.

We used the following dataset from kaggle chitholian_annotated_potholes_dataset

The Dataset contains 665 images of potholes on roads with corresponding annotations boxes for each pothole.

We created a custom torch dataset named PotholeDetectionDataset and then split them to seperate train, validation and test sets (70-10-20).

In this project we trained the following SOTA object detection models:

torchvision models:

• ssd SSD: Single Shot MultiBox Detector
• faster rcnn Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• retinanet Focal Loss for Dense Object Detection
• fcos FCOS: Fully Convolutional One-Stage Object Detection

ultralytics yolo:

• yolov8m YOLOv8 models documentation

In this project, hyperparameter tuning was conducted using Optuna to optimize training on various torchvision models. A comprehensive search was performed across multiple hyperparameter spaces, including model preweight strategies, optimizer types, learning rates, momentum, and weight decay parameters. The study involved carefully selecting combinations of hyperparameters to achieve the best validation mean average precision (mAP). The objective function was optimized using MedianPruner and TPESampler for better exploration and exploitation across trials. Below is a detailed table of the tuned hyperparameters.

• The best configurations for each model was saved for future training (i.e. data/models/fasterrcnn_resnet50_fpn/fasterrcnn_resnet50_fpn_best_params.json)
• Results available in Optuna dashboard at ./data/models/db.sqlite3
  • launch it using:

  optuna-dashboard ./data/models/db.sqlite3
  

  • then open your browser at http://localhost:8080
• At the end, each configuration was set to be trained with a batch size of 8 for 100 epochs.
• Best weights achieving the highest mAP@50 on the validation set were saved.

To cope with the motion blur noise, we applied the following data augmentations using kornia:

These augmentations were chosen to help the model generalize better to different types of blur that might be encountered in real-world scenarios.

1. Z. Zou, K. Chen, Z. Shi, Y. Guo, and J. Ye. Object Detection in 20 Years: A Survey. Proceedings of the IEEE, vol. 111, no. 3, pp. 257-276, March 2023. DOI: 10.1109/JPROC.2023.3238524.

2. Object Detection on COCO. Papers with Code. Link.

3. Object Detection Leaderboard. Hugging Face. Link.

4. Faster R-CNN vs YOLO vs SSD: Object Detection Algorithms. Medium, IBM Data & AI. Link.

5. W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. SSD: Single Shot MultiBox Detector. In Computer Vision – ECCV 2016, Springer International Publishing, 2016, pp. 21–37. DOI: 10.1007/978-3-319-46448-0_2.

6. S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. arXiv preprint arXiv:1506.01497, 2016. Link.

7. T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal Loss for Dense Object Detection. arXiv preprint arXiv:1708.02002, 2018. Link.

8. Z. Tian, C. Shen, H. Chen, and T. He. FCOS: Fully Convolutional One-Stage Object Detection. arXiv preprint arXiv:1904.01355, 2019. Link.

9. Ultralytics. YOLOv8 models documentation. Link.

10. T. Tamagusko and A. Ferreira. Optimizing Pothole Detection in Pavements: A Comparative Analysis of Deep Learning Models. Eng. Proc. 2023, 36, 11. DOI: 10.3390/engproc2023036011.

11. A. Levin, Y. Weiss, F. Durand, and W. T. Freeman. Understanding and Evaluating Blind Deconvolution Algorithms. 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 2009, pp. 1964-1971. DOI: 10.1109/CVPR.2009.5206815.

12. A. Rahman and S. Patel. Annotated Potholes Image Dataset. Kaggle, 2020. DOI: 10.34740/KAGGLE/DSV/973710.