Skip to main content
Springer Nature Link
Log in

An adaptive interpolation and 3D reconstruction algorithm for underwater images

  • ORIGINAL PAPER
  • Published:
Save article
View saved research
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

3D reconstruction technology is gradually applied to underwater scenes, which has become a crucial research direction for human ocean exploration and exploitation. However, due to the complexity of the underwater environment, the number of high-quality underwater images acquired by underwater robots is limited and cannot meet the requirements of 3D reconstruction. Therefore, this paper proposes an adaptive 3D reconstruction algorithm for underwater targets. We apply the frame interpolation technique to underwater 3D reconstruction, an unprecedented technical attempt. In this paper, we design a single-stage large-angle span underwater image interpolation model, which has an excellent enhancement effect on degraded underwater 2D images compared with other methods. Current methods make it challenging to balance the relationship between feature information acquisition and underwater image quality improvement. In this paper, an optimized cascaded feature pyramid scheme and an adaptive bidirectional optical flow estimation algorithm based on underwater NRIQA metrics are proposed and applied to the proposed model to solve the above problems. The intermediate image output from the model improves the image quality and retains the detailed information. Experiments show that the method proposed in this paper outperforms other methods when dealing with several typical degradation types of underwater images. In underwater 3D reconstruction, the intermediate image generated by the model is used as input instead of the degraded image to obtain a denser 3D point cloud and better visualization. Our method is instructive to the problem of acquiring underwater high-quality target images and underwater 3D reconstruction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from €37.37 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price includes VAT (Netherlands)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
The alternative text for this image may have been generated using AI.
Fig. 2
The alternative text for this image may have been generated using AI.
Fig. 3
The alternative text for this image may have been generated using AI.
Fig. 4
The alternative text for this image may have been generated using AI.
Fig. 5
The alternative text for this image may have been generated using AI.
Fig. 6
The alternative text for this image may have been generated using AI.
Fig. 7
The alternative text for this image may have been generated using AI.
Fig. 8
The alternative text for this image may have been generated using AI.
Fig. 9
The alternative text for this image may have been generated using AI.
Fig. 10
The alternative text for this image may have been generated using AI.
Fig. 11
The alternative text for this image may have been generated using AI.
Fig. 12
The alternative text for this image may have been generated using AI.
Fig. 13
The alternative text for this image may have been generated using AI.
Fig. 14
The alternative text for this image may have been generated using AI.
Fig. 15
The alternative text for this image may have been generated using AI.
Fig. 16
The alternative text for this image may have been generated using AI.
Fig. 17.
The alternative text for this image may have been generated using AI.

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

Data availability

Data available on request from the authors.

References

  1. Bao, W., Zhang, X., Chen, L., et al.: High-order model and dynamic filtering for frame rate up-conversion. IEEE Trans. Image Process. 27(8), 3813–3826 (2018)

    Article  MathSciNet  Google Scholar 

  2. Jiang, H., Sun, D., Jampani, V., et al: Super slomo: high quality estimation of multiple intermediate frames for video interpolation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9000–9008 (2018)

  3. Choi, H., Bajić, I.V.: Deep frame prediction for video coding. IEEE Trans. Circuits Syst. Video Technol. 30(7), 1843–1855 (2019)

    Google Scholar 

  4. Dosovitskiy, A., Fischer, P., Ilg, E., et al: Flownet: learning optical flow with convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2758–2766 (2015)

  5. Long, G., Kneip, L., Alvarez, J.M., et al: Learning image matching by simply watching video. In: Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VI 14, pp. 434–450. Springer (2016)

  6. Sun, D., Yang, X., Liu, M.Y., et al: Pwc-net: cnns for optical flow using pyramid, warping, and cost volume. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8934–8943 (2018)

  7. Niklaus, S., Liu, F.: Context-aware synthesis for video frame interpolation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1701–1710 (2018)

  8. Bao, W., Lai, W.S., Zhang, X., et al.: Memc-net: motion estimation and motion compensation driven neural network for video interpolation and enhancement. IEEE Trans. Pattern Anal. Mach. Intell. 43(3), 933–948 (2019)

    Article  Google Scholar 

  9. Niklaus, S., Mai, L., Liu, F: Video frame interpolation via adaptive separable convolution. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 261–270 (2017)

  10. Niklaus, S., Mai, L., Liu, F.: Video frame interpolation via adaptive convolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 670–679 (2017)

  11. Niklaus, S., Liu, F.: Softmax splatting for video frame interpolation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5437–5446 (2020)

  12. Park, J., Lee, C., Kim, C.S.: Asymmetric bilateral motion estimation for video frame interpolation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14539–14548 (2021)

  13. Sim, H., Oh, J., Kim, M.: Xvfi: extreme video frame interpolation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14489–14498 (2021)

  14. Reda, F., Kontkanen, J., Tabellion, E., et al: Film: frame interpolation for large motion. In: European Conference on Computer Vision, pp. 250–266. Springer, Cham (2022)

  15. Peng, Y.T., Cosman, P.C.: Underwater image restoration based on image blurriness and light absorption. IEEE Trans. Image Process. 26(4), 1579–1594 (2017)

    Article  MathSciNet  Google Scholar 

  16. Panetta, K., Gao, C., Agaian, S.: Human-visual-system-inspired underwater image quality measures. IEEE J. Oceanic Eng. 41(3), 541–551 (2015)

    Article  Google Scholar 

  17. Yang, M., Sowmya, A.: An underwater color image quality evaluation metric. IEEE Trans. Image Process. 24(12), 6062–6071 (2015)

    Article  MathSciNet  Google Scholar 

  18. Wang, Y., Li, N., Li, Z., et al.: An imaging-inspired no-reference underwater color image quality assessment metric. Comput. Electr. Eng. 70, 904–913 (2018)

    Article  Google Scholar 

  19. Yang, N., Zhong, Q., Li, K., et al.: A reference-free underwater image quality assessment metric in frequency domain. Signal Process. Image Commun. 94, 116218 (2021)

    Article  Google Scholar 

  20. Luo, Z., Tang, Z., Jiang, L., et al.: An underwater-imaging-model-inspired no-reference quality metric for images in multi-colored environments. Expert Syst. Appl. 191, 116361 (2022)

    Article  Google Scholar 

  21. Werlberger, M., Pock, T., Unger, M., et al: Optical flow guided TV-L 1 video interpolation and restoration. In: Energy Minimization Methods in Computer Vision and Pattern Recognition: 8th International Conference, EMMCVPR 2011, St. Petersburg, Russia, July 25–27, 2011. Proceedings 8, pp. 273–286. Springer, Berlin, Heidelberg (2011)

  22. Baker, S., Scharstein, D., Lewis, J.P., et al.: A database and evaluation methodology for optical flow. Int. J. Comput. Vis. 92, 1–31 (2011)

    Article  Google Scholar 

  23. Yu, Z., Li, H., Wang, Z., et al.: Multi-level video frame interpolation: Exploiting the interaction among different levels. IEEE Trans. Circuits Syst. Video Technol. 23(7), 1235–1248 (2013)

    Article  Google Scholar 

  24. Menze, M., Geiger, A.: Object scene flow for autonomous vehicles. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3061–3070 (2015)

  25. Janai, J., Guney, F., Wulff, J., et al: Slow flow: exploiting high-speed cameras for accurate and diverse optical flow reference data. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3597–3607 (2017)

  26. He, J., Yang, G., Liu, X., et al.: Spatio-temporal saliency-based motion vector refinement for frame rate up-conversion. ACM Trans. Multimedia Comput. Commun. Appl. (TOMM) 16(2), 1–18 (2020)

    Article  Google Scholar 

  27. Meyer, S., Wang, O., Zimmer, H., et al: Phase-based frame interpolation for video. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1410–1418 (2015)

  28. Meyer, S., Djelouah, A., McWilliams, B., et al: Phasenet for video frame interpolation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 498–507 (2018)

  29. Ranjan, A., Black, M.J.: Optical flow estimation using a spatial pyramid network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4161–4170 (2017)

  30. Xue, T., Chen, B., Wu, J., et al.: Video enhancement with task-oriented flow. Int. J. Comput. Vis. 127, 1106–1125 (2019)

    Article  Google Scholar 

  31. Saxena, R., Schuster, R., Wasenmuller, O., et al: PWOC-3D: deep occlusion-aware end-to-end scene flow estimation. In: 2019 IEEE Intelligent Vehicles Symposium (IV), pp. 324–331 (2019)

  32. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5–9, 2015, Proceedings, Part III 18, pp. 234–241. Springer (2015)

  33. Islam, M.J., Xia, Y., Sattar, J.: Fast underwater image enhancement for improved visual perception. IEEE Robot. Autom. Lett. 5(2), 3227–3234 (2020)

    Article  Google Scholar 

  34. Islam, M.J., Luo, P., Sattar, J.: Simultaneous enhancement and super-resolution of underwater imagery for improved visual perception. arXiv:2002.01155 (2020)

  35. Li, C., Guo, C., Ren, W., et al.: An underwater image enhancement benchmark dataset and beyond. IEEE Trans. Image Process. 29, 4376–4389 (2019)

    Article  Google Scholar 

  36. Li H, Li J, Wang W: A fusion adversarial underwater image enhancement network with a public test dataset. arXiv preprint arXiv:1906.06819 (2019)

  37. Islam, M.J., Enan, S.S., Luo, P., et al: Underwater image super-resolution using deep residual multipliers. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 900–906 (2020)

  38. Charbonnier, P., Blanc-Feraud, L., Aubert, G., et al: Two deterministic half-quadratic regularization algorithms for computed imaging. In: Proceedings of 1st International Conference on Image Processing, vol 2, pp. 168–172 (1994)

  39. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part II 14, pp. 694–711. Springer (2016)

  40. Pertuz, S., Puig, D., Garcia, M.A.: Analysis of focus measure operators for shape-from-focus. Pattern Recogn. 46(5), 1415–1432 (2013)

    Article  Google Scholar 

  41. Zhou, Y., Yan, K.: Domain adaptive adversarial learning based on physics model feedback for underwater image enhancement. arXiv:2002.09315 (2020)

  42. Sun, H.H., Wang, H.X., Li, M., et al.: The blurring and restoring process of the moving object images. Appl. Mech. Mater. 462, 251–254 (2014)

    Article  Google Scholar 

  43. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv:1412.6980 (2014)

  44. Jin, X., Wu, L., Chen, J., et al: A unified pyramid recurrent network for video frame interpolation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1578–1587 (2023)

  45. Zhang, G., Zhu, Y., Wang, H., et al: Extracting motion and appearance via inter-frame attention for efficient video frame interpolation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5682–5692 (2023)

  46. Singh, A., Satapathy, S.C., Roy, A., et al: Ai-based mobile edge computing for iot: applications, challenges, and future scope. Arab. J. Sci. Eng. 1–31 (2022)

  47. Farooqi, N., Gutub, A., Khozium, M.O.: Smart community challenges: enabling IoT/M2M technology case study. Life Sci. J. 16(7), 11–17 (2019)

    Google Scholar 

  48. Makkah, K.S.A.: Data dissemination and collection algorithms for collaborative sensor devices using dynamic cluster heads. Trends Appl. Sci. Res. 8(2), 55–72 (2013)

    Article  Google Scholar 

  49. Abdul-Aziz, G.A., Aly, A.S.A.: Trialing a smart face-recognition computer system to recognize lost people visiting the two holy mosques. Arab. J. Forensic Sci. Forensic Med. 1(8), 1120–1132 (2018)

    Article  Google Scholar 

  50. Kim, S., Guy, S.J., Hillesland, K., et al.: Velocity-based modeling of physical interactions in dense crowds. Vis. Comput. 31, 541–555 (2015)

    Article  Google Scholar 

  51. Shambour, M.K., Gutub, A.: Progress of IoT research technologies and applications serving Hajj and Umrah. Arab. J. Sci. Eng. 1–21 (2022)

  52. Curtis, S., Zafar, B., Gutub, A., et al.: Right of way: asymmetric agent interactions in crowds. Vis. Comput. 29, 1277–1292 (2013)

    Article  Google Scholar 

  53. Aljarf, A., Zamzami, H., Gutub, A.: Integrating machine learning and features extraction for practical reliable color images steganalysis classification. Soft. Comput. 27(19), 13877–13888 (2023)

    Article  Google Scholar 

  54. Aly, S., Gutub, A.: Intelligent recognition system for identifying items and pilgrims. NED Univ. J. Res. 15(2), 17–23 (2018)

    Google Scholar 

  55. Alharthi, N., Gutub, A.: Data visualization to explore improving decision-making within Hajj services. Sci. Model. Res. 2(1), 9–18 (2017)

    Google Scholar 

  56. Aly, S.A., AlGhamdi, T.A., Salim, M., et al.: Information gathering schemes for collaborative sensor devices. Procedia Comput. Sci. 32, 1141–1146 (2014)

    Article  Google Scholar 

  57. Roy, P.K., Saumya, S., Singh, J.P., et al.: Analysis of community question-answering issues via machine learning and deep learning: State-of-the-art review. CAAI Trans. Intell. Technol. 8(1), 95–117 (2023)

    Article  Google Scholar 

Download references

Acknowledgements

The author disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Natural Science Foundation of Shanghai (No.14ZR1414900) for providing financial support for this work.

Author information

Authors and Affiliations

  1. School of Mechatronic Engineering and Automation, Shanghai University, No. 99 Shangda Road, Shanghai, China

    Zhijie Tang, Congqi Xu & Siyu Yan

Authors
  1. Zhijie Tang
    View author publications

    Search author on:PubMed Google Scholar

  2. Congqi Xu
    View author publications

    Search author on:PubMed Google Scholar

  3. Siyu Yan
    View author publications

    Search author on:PubMed Google Scholar

Corresponding author

Correspondence to Zhijie Tang.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tang, Z., Xu, C. & Yan, S. An adaptive interpolation and 3D reconstruction algorithm for underwater images. Machine Vision and Applications 35, 33 (2024). https://doi.org/10.1007/s00138-024-01518-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Version of record:

  • DOI: https://doi.org/10.1007/s00138-024-01518-2

Keywords