{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2025,10,12]],"date-time":"2025-10-12T02:10:47Z","timestamp":1760235047314,"version":"build-2065373602"},"reference-count":39,"publisher":"MDPI AG","issue":"15","license":[{"start":{"date-parts":[[2021,7,23]],"date-time":"2021-07-23T00:00:00Z","timestamp":1626998400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100004837","name":"Ministerio de Ciencia e Innovaci\u00f3n","doi-asserted-by":"publisher","award":["TEC2016-76038-C3-1-R (HERAKLES)"],"award-info":[{"award-number":["TEC2016-76038-C3-1-R (HERAKLES)"]}],"id":[{"id":"10.13039\/501100004837","id-type":"DOI","asserted-by":"publisher"}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>We use the recent advances in Deep Learning to solve an underwater motion planning problem by making use of optimal control tools\u2014namely, we propose using the Deep Galerkin Method (DGM) to approximate the Hamilton\u2013Jacobi\u2013Bellman PDE that can be used to solve continuous time and state optimal control problems. In order to make our approach more realistic, we consider that there are disturbances in the underwater medium that affect the trajectory of the autonomous vehicle. After adapting DGM by making use of a surrogate approach, our results show that our method is able to efficiently solve the proposed problem, providing large improvements over a baseline control in terms of costs, especially in the case in which the disturbances effects are more significant.<\/jats:p>","DOI":"10.3390\/s21155011","type":"journal-article","created":{"date-parts":[[2021,7,23]],"date-time":"2021-07-23T10:31:44Z","timestamp":1627036304000},"page":"5011","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":3,"title":["Deep Learning for Efficient and Optimal Motion Planning for AUVs with Disturbances"],"prefix":"10.3390","volume":"21","author":[{"ORCID":"https:\/\/orcid.org\/0000-0002-7028-3179","authenticated-orcid":false,"given":"Juan","family":"Parras","sequence":"first","affiliation":[{"name":"Information Processing and Telecommunications Center, E.T.S. Ingenieros de Telecomunicaci\u00f3n, Universidad Polit\u00e9cnica de Madrid, 28040 Madrid, Spain"}]},{"given":"Patricia A.","family":"Apell\u00e1niz","sequence":"additional","affiliation":[{"name":"Information Processing and Telecommunications Center, E.T.S. Ingenieros de Telecomunicaci\u00f3n, Universidad Polit\u00e9cnica de Madrid, 28040 Madrid, Spain"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-9073-7927","authenticated-orcid":false,"given":"Santiago","family":"Zazo","sequence":"additional","affiliation":[{"name":"Information Processing and Telecommunications Center, E.T.S. Ingenieros de Telecomunicaci\u00f3n, Universidad Polit\u00e9cnica de Madrid, 28040 Madrid, Spain"}]}],"member":"1968","published-online":{"date-parts":[[2021,7,23]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1007\/s10846-009-9383-1","article-title":"A survey of motion planning algorithms from the perspective of autonomous UAV guidance","volume":"57","author":"Goerzen","year":"2010","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Friesz, T.L. (2010). Dynamic Optimization and Differential Games, Springer.","DOI":"10.1007\/978-0-387-72778-3"},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"167","DOI":"10.1093\/imamci\/17.2.167","article-title":"An upwind finite-difference method for the approximation of viscosity solutions to Hamilton-Jacobi-Bellman equations","volume":"17","author":"Wang","year":"2000","journal-title":"IMA J. Math. Control Inf."},{"key":"ref_4","first-page":"201","article-title":"An Introduction to Finite Diffference Methods for PDEs in Finance","volume":"Volume 29","author":"Tourin","year":"2011","journal-title":"Book Chapter: Nizar Touzi, Optimal Stochastic Target problems, and Backward SDE, Fields Institute Monographs"},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"3012","DOI":"10.1109\/TAC.2015.2406976","article-title":"Convergence of an upwind finite-difference scheme for Hamilton\u2013Jacobi\u2013Bellman equation in optimal control","volume":"60","author":"Sun","year":"2015","journal-title":"IEEE Trans. Autom. Control"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"603","DOI":"10.1016\/bs.hna.2016.09.018","article-title":"Numerical methods for hamilton\u2013jacobi type equations","volume":"Volume 17","author":"Falcone","year":"2016","journal-title":"Handbook of Numerical Methods for Hyperbolic Problems, Handbook of Numerical Analysis"},{"key":"ref_7","unstructured":"Gombao, S. (2002, January 12\u201316). Approximation of optimal controls for semilinear parabolic PDE by solving Hamilton-Jacobi-Bellman equations. Proceedings of the 15th International Symposium on the Mathematical Theory of Networks and Systems, Notre Dame, IN, USA."},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"1239","DOI":"10.1137\/040610830","article-title":"A curse-of-dimensionality-free numerical method for solution of certain HJB PDEs","volume":"46","author":"McEneaney","year":"2007","journal-title":"SIAM J. Control Optim."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"1339","DOI":"10.1016\/j.jcp.2018.08.029","article-title":"DGM: A deep learning algorithm for solving partial differential equations","volume":"375","author":"Sirignano","year":"2018","journal-title":"J. Comput. Phys."},{"key":"ref_10","unstructured":"Al-Aradi, A., Correia, A., Naiff, D.d.F., Jardim, G., and Saporito, Y. (2019). Applications of the Deep Galerkin Method to Solving Partial Integro-Differential and Hamilton-Jacobi-Bellman Equations. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Chen, J., Du, R., and Wu, K. (2020). A Comparison Study of Deep Galerkin Method and Deep Ritz Method for Elliptic Problems with Different Boundary Conditions. arXiv.","DOI":"10.4208\/cmr.2020-0051"},{"key":"ref_12","unstructured":"Wang, J., Xu, Z.Q.J., Zhang, J., and Zhang, Y. (2020). Implicit bias with Ritz-Galerkin method in understanding deep learning for solving PDEs. arXiv."},{"key":"ref_13","unstructured":"Li, J., Yue, J., Zhang, W., and Duan, W. (2020). The Deep Learning Galerkin Method for the General Stokes Equations. arXiv."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"65","DOI":"10.1109\/48.380245","article-title":"The use of artificial neural networks for the intelligent optimal control of surface ships","volume":"20","author":"Burns","year":"1995","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_15","doi-asserted-by":"crossref","first-page":"1373","DOI":"10.1007\/s10236-014-0757-y","article-title":"Time-optimal path planning in dynamic flows using level set equations: Theory and schemes","volume":"64","author":"Lolla","year":"2014","journal-title":"Ocean. Dyn."},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Takei, R., Tsai, R., Shen, H., and Landa, Y. (July, January 30). A practical path-planning algorithm for a simple car: A Hamilton-Jacobi approach. Proceedings of the 2010 American Control Conference, Baltimore, MD, USA.","DOI":"10.1109\/ACC.2010.5531607"},{"key":"ref_17","doi-asserted-by":"crossref","first-page":"466","DOI":"10.1109\/JOE.2020.2998930","article-title":"Generalized Optimal Control for Autonomous Mine Countermeasures Missions","volume":"46","author":"Kragelund","year":"2020","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_18","doi-asserted-by":"crossref","first-page":"1219","DOI":"10.1109\/JOE.2019.2926822","article-title":"Toward time-optimal trajectory planning for autonomous ship maneuvering in close-range encounters","volume":"45","author":"Li","year":"2019","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Shen, C., Shi, Y., and Buckham, B. (2016, January 12\u201314). Nonlinear model predictive control for trajectory tracking of an AUV: A distributed implementation. Proceedings of the 2016 IEEE 55th Conference on Decision and Control (CDC), Las Vegas, NV, USA.","DOI":"10.1109\/CDC.2016.7799190"},{"key":"ref_20","doi-asserted-by":"crossref","first-page":"108863","DOI":"10.1016\/j.automatica.2020.108863","article-title":"Distributed implementation of nonlinear model predictive control for AUV trajectory tracking","volume":"115","author":"Shen","year":"2020","journal-title":"Automatica"},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"3","DOI":"10.1016\/S0005-1098(01)00174-1","article-title":"The explicit linear quadratic regulator for constrained systems","volume":"38","author":"Bemporad","year":"2002","journal-title":"Automatica"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"3866","DOI":"10.1109\/TCYB.2020.2999556","article-title":"Efficient representation and approximation of model predictive control laws via deep learning","volume":"50","author":"Karg","year":"2020","journal-title":"IEEE Trans. Cybern."},{"key":"ref_23","doi-asserted-by":"crossref","first-page":"131","DOI":"10.1109\/JOE.2013.2278891","article-title":"AUV navigation and localization: A review","volume":"39","author":"Paull","year":"2013","journal-title":"IEEE J. Ocean. Eng."},{"key":"ref_24","unstructured":"Isaacs, R. (1965). Differential Games: A Mathematical Theory with Applications to Warfare and Pursuit, Control and Optimization, Wiley."},{"key":"ref_25","unstructured":"Bertsekas, D.P. (2005). Dynamic Programming and Optimal Control, Athena Scientific."},{"key":"ref_26","unstructured":"Bertsekas, D.P. (2007). Dynamic Programming and Optimal Control, Athena Scientific."},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Sutton, R.S., and Barto, A.G. (1998). Reinforcement Learning: An Introduction, MIT Press Cambridge.","DOI":"10.1109\/TNN.1998.712192"},{"key":"ref_28","doi-asserted-by":"crossref","unstructured":"Todorov, E. (2006). Optimal control theory. Bayesian Brain: Probabilistic Approaches to Neural Coding, MIT Press.","DOI":"10.7551\/mitpress\/9780262042383.003.0012"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1090\/S0002-9947-1983-0690039-8","article-title":"Viscosity solutions of Hamilton-Jacobi equations","volume":"277","author":"Crandall","year":"1983","journal-title":"Trans. Am. Math. Soc."},{"key":"ref_30","unstructured":"Lions, P.L. (1983, January 16\u201324). Hamilton-Jacobi-Bellman equations and the optimal control of stochastic systems. Proceedings of the International Congress of Mathematicians, Warsaw, Poland."},{"key":"ref_31","doi-asserted-by":"crossref","unstructured":"Crandall, M.G. (1997). Viscosity solutions: A primer. Viscosity Solutions and Applications, Springer.","DOI":"10.1007\/BFb0094294"},{"key":"ref_32","unstructured":"Bressan, A. (2011). Viscosity solutions of Hamilton-Jacobi equations and optimal control problems. Lect. Notes, Available online: http:\/\/personal.psu.edu\/axb62\/PSPDF\/hj.pdf."},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1090\/S0273-0979-1992-00266-5","article-title":"User\u2019s guide to viscosity solutions of second order partial differential equations","volume":"27","author":"Crandall","year":"1992","journal-title":"Bull. Am. Math. Soc."},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Bardi, M., and Capuzzo-Dolcetta, I. (1997). Optimal Control and Viscosity Solutions of Hamilton-Jacobi-Bellman Equations, Birkh\u00e4user.","DOI":"10.1007\/978-0-8176-4755-1"},{"key":"ref_35","unstructured":"Fleming, W.H., and Soner, H.M. (2006). Controlled Markov Processes and Viscosity Solutions, Springer."},{"key":"ref_36","first-page":"271","article-title":"Convergence of approximation schemes for fully nonlinear second order equations","volume":"4","author":"Barles","year":"1991","journal-title":"Asymptot. Anal."},{"key":"ref_37","unstructured":"Kingma, D.P., and Ba, J.L. (2015, January 7\u20139). Adam: A method for stochastic gradient descent. Proceedings of the ICLR: International Conference on Learning Representations, San Diego, CA, USA."},{"key":"ref_38","doi-asserted-by":"crossref","first-page":"591","DOI":"10.1007\/s10846-019-01004-2","article-title":"Mapless motion planning system for an autonomous underwater vehicle using policy gradient-based deep reinforcement learning","volume":"96","author":"Sun","year":"2019","journal-title":"J. Intell. Robot. Syst."},{"key":"ref_39","doi-asserted-by":"crossref","first-page":"96549","DOI":"10.1109\/ACCESS.2019.2929120","article-title":"Target Search Control of AUV in Underwater Environment With Deep Reinforcement Learning","volume":"7","author":"Cao","year":"2019","journal-title":"IEEE Access"}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/15\/5011\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,11]],"date-time":"2025-10-11T06:33:57Z","timestamp":1760164437000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/21\/15\/5011"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2021,7,23]]},"references-count":39,"journal-issue":{"issue":"15","published-online":{"date-parts":[[2021,8]]}},"alternative-id":["s21155011"],"URL":"https:\/\/doi.org\/10.3390\/s21155011","relation":{},"ISSN":["1424-8220"],"issn-type":[{"type":"electronic","value":"1424-8220"}],"subject":[],"published":{"date-parts":[[2021,7,23]]}}}