{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,2,28]],"date-time":"2026-02-28T18:17:55Z","timestamp":1772302675273,"version":"3.50.1"},"reference-count":42,"publisher":"MDPI AG","issue":"18","license":[{"start":{"date-parts":[[2024,9,10]],"date-time":"2024-09-10T00:00:00Z","timestamp":1725926400000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"funder":[{"DOI":"10.13039\/501100000266","name":"Royal Academy of Engineering Chairs in Emerging Technologies program and the UK Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/T00097X\/1"],"award-info":[{"award-number":["EP\/T00097X\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Royal Academy of Engineering Chairs in Emerging Technologies program and the UK Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/T00097X\/1"],"award-info":[{"award-number":["EP\/T00097X\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Royal Academy of Engineering Chairs in Emerging Technologies program and the UK Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["QUEST EP\/T021020\/1"],"award-info":[{"award-number":["QUEST EP\/T021020\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Royal Academy of Engineering Chairs in Emerging Technologies program and the UK Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["101097708"],"award-info":[{"award-number":["101097708"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"Royal Academy of Engineering Chairs in Emerging Technologies program and the UK Engineering and Physical Sciences Research Council","doi-asserted-by":"publisher","award":["EP\/Y029178\/1"],"award-info":[{"award-number":["EP\/Y029178\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"EPSRC projects Quantic","doi-asserted-by":"publisher","award":["EP\/T00097X\/1"],"award-info":[{"award-number":["EP\/T00097X\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"EPSRC projects Quantic","doi-asserted-by":"publisher","award":["EP\/T00097X\/1"],"award-info":[{"award-number":["EP\/T00097X\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"EPSRC projects Quantic","doi-asserted-by":"publisher","award":["QUEST EP\/T021020\/1"],"award-info":[{"award-number":["QUEST EP\/T021020\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"EPSRC projects Quantic","doi-asserted-by":"publisher","award":["101097708"],"award-info":[{"award-number":["101097708"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"DOI":"10.13039\/501100000266","name":"EPSRC projects Quantic","doi-asserted-by":"publisher","award":["EP\/Y029178\/1"],"award-info":[{"award-number":["EP\/Y029178\/1"]}],"id":[{"id":"10.13039\/501100000266","id-type":"DOI","asserted-by":"publisher"}]},{"name":"DIFAI ERC Advanced","award":["EP\/T00097X\/1"],"award-info":[{"award-number":["EP\/T00097X\/1"]}]},{"name":"DIFAI ERC Advanced","award":["EP\/T00097X\/1"],"award-info":[{"award-number":["EP\/T00097X\/1"]}]},{"name":"DIFAI ERC Advanced","award":["QUEST EP\/T021020\/1"],"award-info":[{"award-number":["QUEST EP\/T021020\/1"]}]},{"name":"DIFAI ERC Advanced","award":["101097708"],"award-info":[{"award-number":["101097708"]}]},{"name":"DIFAI ERC Advanced","award":["EP\/Y029178\/1"],"award-info":[{"award-number":["EP\/Y029178\/1"]}]},{"name":"UK Horizon","award":["EP\/T00097X\/1"],"award-info":[{"award-number":["EP\/T00097X\/1"]}]},{"name":"UK Horizon","award":["EP\/T00097X\/1"],"award-info":[{"award-number":["EP\/T00097X\/1"]}]},{"name":"UK Horizon","award":["QUEST EP\/T021020\/1"],"award-info":[{"award-number":["QUEST EP\/T021020\/1"]}]},{"name":"UK Horizon","award":["101097708"],"award-info":[{"award-number":["101097708"]}]},{"name":"UK Horizon","award":["EP\/Y029178\/1"],"award-info":[{"award-number":["EP\/Y029178\/1"]}]}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Sensors"],"abstract":"<jats:p>In the field of detection and ranging, multiple complementary sensing modalities may be used to enrich information obtained from a dynamic scene. One application of this sensor fusion is in public security and surveillance, where efficacy and privacy protection measures must be continually evaluated. We present a novel deployment of sensor fusion for the discrete detection of concealed metal objects on persons whilst preserving their privacy. This is achieved by coupling off-the-shelf mmWave radar and depth camera technology with a novel neural network architecture that processes radar signals using convolutional Long Short-Term Memory (LSTM) blocks and depth signals using convolutional operations. The combined latent features are then magnified using deep feature magnification to reveal cross-modality dependencies in the data. We further propose a decoder, based on the feature extraction and embedding block, to learn an efficient upsampling of the latent space to locate the concealed object in the spatial domain through radar feature guidance. We demonstrate the ability to detect the presence and infer the 3D location of concealed metal objects. We achieve accuracies of up to 95% using a technique that is robust to multiple persons. This work provides a demonstration of the potential for cost-effective and portable sensor fusion with strong opportunities for further development.<\/jats:p>","DOI":"10.3390\/s24185865","type":"journal-article","created":{"date-parts":[[2024,9,10]],"date-time":"2024-09-10T03:16:51Z","timestamp":1725938211000},"page":"5865","update-policy":"https:\/\/doi.org\/10.3390\/mdpi_crossmark_policy","source":"Crossref","is-referenced-by-count":5,"title":["AI-Enabled Sensor Fusion of Time-of-Flight Imaging and mmWave for Concealed Metal Detection"],"prefix":"10.3390","volume":"24","author":[{"ORCID":"https:\/\/orcid.org\/0000-0003-4893-6222","authenticated-orcid":false,"given":"Chaitanya","family":"Kaul","sequence":"first","affiliation":[{"name":"School of Computing Science, University of Glasgow, Glasgow G12 8QQ, UK"}]},{"ORCID":"https:\/\/orcid.org\/0009-0005-3433-7167","authenticated-orcid":false,"given":"Kevin J.","family":"Mitchell","sequence":"additional","affiliation":[{"name":"School of Physics and Astronomy, University of Glasgow, Glasgow G12 8QQ, UK"}]},{"given":"Khaled","family":"Kassem","sequence":"additional","affiliation":[{"name":"School of Physics and Astronomy, University of Glasgow, Glasgow G12 8QQ, UK"}]},{"given":"Athanasios","family":"Tragakis","sequence":"additional","affiliation":[{"name":"School of Physics and Astronomy, University of Glasgow, Glasgow G12 8QQ, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4424-2194","authenticated-orcid":false,"given":"Valentin","family":"Kapitany","sequence":"additional","affiliation":[{"name":"School of Physics and Astronomy, University of Glasgow, Glasgow G12 8QQ, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-6656-9123","authenticated-orcid":false,"given":"Ilya","family":"Starshynov","sequence":"additional","affiliation":[{"name":"School of Physics and Astronomy, University of Glasgow, Glasgow G12 8QQ, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-9840-0269","authenticated-orcid":false,"given":"Federica","family":"Villa","sequence":"additional","affiliation":[{"name":"Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Via G. Ponzio 34\/5, 20133 Milano, Italy"}]},{"ORCID":"https:\/\/orcid.org\/0000-0003-4228-7962","authenticated-orcid":false,"given":"Roderick","family":"Murray-Smith","sequence":"additional","affiliation":[{"name":"School of Computing Science, University of Glasgow, Glasgow G12 8QQ, UK"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-8397-334X","authenticated-orcid":false,"given":"Daniele","family":"Faccio","sequence":"additional","affiliation":[{"name":"School of Physics and Astronomy, University of Glasgow, Glasgow G12 8QQ, UK"}]}],"member":"1968","published-online":{"date-parts":[[2024,9,10]]},"reference":[{"key":"ref_1","doi-asserted-by":"crossref","first-page":"1683","DOI":"10.1109\/JPROC.2007.898832","article-title":"Millimeter-Wave and Submillimeter-Wave Imaging for Security and Surveillance","volume":"95","author":"Appleby","year":"2007","journal-title":"Proc. IEEE"},{"key":"ref_2","first-page":"1","article-title":"mSense: Towards Mobile Material Sensing with a Single Millimeter-Wave Radio","volume":"4","author":"Wu","year":"2020","journal-title":"Proc. ACM Interact. Mob. Wearable Ubiquitous Technol."},{"key":"ref_3","doi-asserted-by":"crossref","first-page":"184","DOI":"10.1109\/TTHZ.2015.2392628","article-title":"THz and mm-Wave Sensing of Corneal Tissue Water Content: In Vivo Sensing and Imaging Results","volume":"5","author":"Taylor","year":"2015","journal-title":"IEEE Trans. Terahertz Sci. Technol."},{"key":"ref_4","doi-asserted-by":"crossref","first-page":"78","DOI":"10.1109\/TSP.2010.2086448","article-title":"Adaptive OFDM Radar for Target Detection in Multipath Scenarios","volume":"59","author":"Sen","year":"2011","journal-title":"IEEE Trans. Signal Process."},{"key":"ref_5","doi-asserted-by":"crossref","first-page":"900","DOI":"10.1364\/OPTICA.392465","article-title":"Spatial images from temporal data","volume":"7","author":"Turpin","year":"2020","journal-title":"Optica"},{"key":"ref_6","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1103\/PhysRevLett.126.174301","article-title":"3D Imaging from Multipath Temporal Echoes","volume":"126","author":"Turpin","year":"2021","journal-title":"Phys. Rev. Lett."},{"key":"ref_7","doi-asserted-by":"crossref","first-page":"28616","DOI":"10.1109\/ACCESS.2019.2902315","article-title":"Imaging with WiFi","volume":"7","author":"Vakalis","year":"2019","journal-title":"IEEE Access"},{"key":"ref_8","doi-asserted-by":"crossref","first-page":"171","DOI":"10.1109\/RBME.2022.3156810","article-title":"Contactless WiFi Sensing and Monitoring for Future Healthcare - Emerging Trends, Challenges, and Opportunities","volume":"16","author":"Ge","year":"2023","journal-title":"IEEE Rev. Biomed. Eng."},{"key":"ref_9","doi-asserted-by":"crossref","first-page":"109","DOI":"10.1117\/12.603662","article-title":"A compact, low-cost, passive MMW security scanner","volume":"Volume 5789","author":"Appleby","year":"2005","journal-title":"Proceedings of the Passive Millimeter-Wave Imaging Technology VIII"},{"key":"ref_10","first-page":"198","article-title":"Radiation exposure and privacy concerns surrounding full-body scanners in airports","volume":"7","author":"Accardo","year":"2014","journal-title":"J. Radiat. Res. Appl. Sci."},{"key":"ref_11","first-page":"135","article-title":"Passive Millimeter Wave Imaging: A New Weapon in the Fight Against Crime or a Fourth Amendment Violation?","volume":"9","author":"Rosenberg","year":"1998","journal-title":"Alb. LJ Sci. Tech."},{"key":"ref_12","doi-asserted-by":"crossref","first-page":"654806","DOI":"10.1117\/12.718950","article-title":"High-resolution passive millimeter-wave security screening using few amplifiers","volume":"Volume 6548","author":"Appleby","year":"2007","journal-title":"Proceedings of the Passive Millimeter-Wave Imaging Technology X"},{"key":"ref_13","unstructured":"SSI (2024, August 30). Athena Security Unveils Occupancy Tracking, Concealed Gun Detection. Security Sales and Integration. Available online: https:\/\/www.securitysales.com\/surveillance\/athena-security-occupancy-tracking-concealed-gun-detection\/."},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"1085","DOI":"10.1109\/LAWP.2013.2280461","article-title":"Improvement in 77-GHz radar cross section of road work jacket and side screen by use of planar flexible retrodirective reflectors","volume":"12","author":"Cheng","year":"2013","journal-title":"IEEE Antennas Wirel. Propag. Lett."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Yamada, N., Tanaka, Y., and Nishikawa, K. (2005, January 4\u20136). Radar cross section for pedestrian in 76GHz band. Proceedings of the 2005 European Microwave Conference, Paris, France.","DOI":"10.1109\/EUMC.2005.1610101"},{"key":"ref_16","unstructured":"Baker, J. (2024, July 03). How Can AI Help Speed Up Airport Security?. Available online: https:\/\/www.airport-technology.com\/features\/ai-at-airports-security."},{"key":"ref_17","unstructured":"IPVM (2024, July 08). How Metal Detectors Can Perform Like Evolv at 90% Lower Price. Available online: https:\/\/ipvm.com\/reports\/metal-evolv."},{"key":"ref_18","unstructured":"Sequestim (2024, July 08). Express Security Solutions. Available online: https:\/\/sequestim.com\/technology."},{"key":"ref_19","unstructured":"Clayton, J. (2024, July 10). AI Weapons Scanner Evolv Backtracks on UK Testing Claims. Available online: https:\/\/www.bbc.co.uk\/news\/technology-68547574."},{"key":"ref_20","unstructured":"Gapi\u0144ski, B., Ciszak, O., Ivanov, V., and Machado, J.M. (2024). Managing Pandemics in Airport Security Environments: A Comparative Analysis of Classic Airport Security and Smart Security Approaches. Proceedings of the Advances in Manufacturing IV, Springer."},{"key":"ref_21","doi-asserted-by":"crossref","first-page":"100","DOI":"10.1111\/j.1746-1561.2010.00566.x","article-title":"Impacts of Metal Detector Use in Schools: Insights from 15 Years of Research*","volume":"81","author":"Hankin","year":"2011","journal-title":"J. Sch. Health"},{"key":"ref_22","doi-asserted-by":"crossref","first-page":"569","DOI":"10.1177\/0895904816673735","article-title":"The Impact of Random Metal Detector Searches on Contraband Possession and Feelings of Safety at School","volume":"32","author":"Bhatt","year":"2018","journal-title":"Educ. Policy"},{"key":"ref_23","first-page":"2","article-title":"Beyond Politics: Technology-A Rational Alternative to Prevent School Shootings","volume":"17","author":"Nirgudkar","year":"2024","journal-title":"Intersect Stanf. J. Sci. Technol. Soc."},{"key":"ref_24","doi-asserted-by":"crossref","unstructured":"Mitchell, K., Kassem, K., Kaul, C., Kapitany, V., Binner, P., Ramsay, A., Faccio, D., and Murray-Smith, R. (2023, January 4\u201310). mmSense: Detecting Concealed Weapons with a Miniature Radar Sensor. Proceedings of the ICASSP 2023\u20142023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece.","DOI":"10.1109\/ICASSP49357.2023.10095884"},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Pei, D., Gong, D., Liu, K., Zeng, X., Zhang, S., Chen, M., and Zheng, K. (2024). mmCTD: Concealed Threat Detection for Cruise Ships via Mmwave Radar. IEEE Trans. Veh. Technol., 1\u201318.","DOI":"10.1109\/TVT.2024.3352039"},{"key":"ref_26","doi-asserted-by":"crossref","unstructured":"Mahdipour, H., Laviada, J., and Las-Heras, F. (2024, January 17\u201322). Detection Capability of a CNN-Based Imageless Millimeter Wave System for Static Concealed Objects. Proceedings of the 2024 18th European Conference on Antennas and Propagation (EuCAP), Glasgow, UK.","DOI":"10.23919\/EuCAP60739.2024.10501605"},{"key":"ref_27","doi-asserted-by":"crossref","unstructured":"Chen, D., Schlegel, A., and Nanzer, J.A. (2024). Imageless Contraband Detection Using a Millimeter-Wave Dynamic Antenna Array via Spatial Fourier Domain Sampling. arXiv.","DOI":"10.1109\/ACCESS.2024.3437433"},{"key":"ref_28","unstructured":"Rao, S., and Instruments, T. (2024, July 08). The Fundamentals of Millimeter Wave Radar Sensors (Rev. A). Texas Instruments Incorporated (Dallas, TX, USA) White Paper. Available online: https:\/\/www.ti.com\/lit\/wp\/spyy005a\/spyy005a.pdf."},{"key":"ref_29","unstructured":"Skolnik, M.I. (1999). Introduction to Radar Systems, McGraw-Hill."},{"key":"ref_30","unstructured":"AG, I.T. (2024, July 08). Demo Position2GO Software Guide XENSIV 24 GHz Radar. Infineon Technologies AG (Neubiberg, Germany) White Paper. Available online: https:\/\/www.infineon.com\/dgdl\/Infineon-UM_Demo_Position2Go_Software_Guide-UserManual-v01_20-EN.pdf?fileId=5546d4627762291e017769040a233324."},{"key":"ref_31","unstructured":"Intel (2018). Intel Realsense D400 Series\/SR300 Viewer: User Guide, Intel Corporation. Technical Report."},{"key":"ref_32","doi-asserted-by":"crossref","first-page":"5086","DOI":"10.1364\/OE.21.005086","article-title":"Single-photon pulsed-light indirect time-of-flight 3D ranging","volume":"21","author":"Bellisai","year":"2013","journal-title":"Opt. Express"},{"key":"ref_33","doi-asserted-by":"crossref","first-page":"87270L","DOI":"10.1117\/12.2018661","article-title":"MiSPIA: Microelectronic single-photon 3D imaging arrays for low-light high-speed safety and security applications","volume":"8727","author":"Zappa","year":"2013","journal-title":"Adv. Photon Count. Tech. VII"},{"key":"ref_34","doi-asserted-by":"crossref","unstructured":"Bronzi, D., Member, S., Villa, F., Tisa, S., Tosi, A., Zappa, F., Member, S., Durini, D., Weyers, S., and Brockherde, W. (2014). Array for 2-D Imaging and 3-D Ranging. IEEE J. Sel. Top. Quantum Electron., 20.","DOI":"10.1109\/JSTQE.2014.2341562"},{"key":"ref_35","doi-asserted-by":"crossref","first-page":"17","DOI":"10.1016\/j.inffus.2023.01.015","article-title":"Multi-level feature fusion for multimodal human activity recognition in Internet of Healthcare Things","volume":"94","author":"Islam","year":"2023","journal-title":"Inf. Fusion"},{"key":"ref_36","doi-asserted-by":"crossref","unstructured":"Xaviar, S., Yang, X., and Ardakanian, O. (2023). Robust Multimodal Fusion for Human Activity Recognition. arXiv.","DOI":"10.1109\/JSEN.2024.3388893"},{"key":"ref_37","doi-asserted-by":"crossref","unstructured":"Ryu, J., Rehman, M.U., Nizami, I.F., and Chong, K.T. (2023). SegR-Net: A deep learning framework with multi-scale feature fusion for robust retinal vessel segmentation. Comput. Biol. Med., 163.","DOI":"10.1016\/j.compbiomed.2023.107132"},{"key":"ref_38","doi-asserted-by":"crossref","unstructured":"Kaul, C., Mitchell, K.J., Kassem, K., Tragakis, A., Kapitany, V., Starshynov, I., Albert Villa, F., Murray-Smith, R., and Faccio, D. (2024). AI-Enabled Sensor Fusion of Time of Flight Imaging and Mmwave for Concealed Metal Detection. arXiv.","DOI":"10.3390\/s24185865"},{"key":"ref_39","doi-asserted-by":"crossref","unstructured":"Pearce, A., Zhang, J.A., Xu, R., and Wu, K. (2023). Multi-Object Tracking with mmWave Radar: A Review. Electronics, 12.","DOI":"10.3390\/electronics12020308"},{"key":"ref_40","doi-asserted-by":"crossref","first-page":"6657709","DOI":"10.1155\/2021\/6657709","article-title":"Indoor Detection and Tracking of People Using mmWave Sensor","volume":"2021","author":"Huang","year":"2021","journal-title":"J. Sens."},{"key":"ref_41","doi-asserted-by":"crossref","first-page":"2994","DOI":"10.1109\/TGRS.2020.3019915","article-title":"Multiperson Continuous Tracking and Identification From mm-Wave Micro-Doppler Signatures","volume":"59","author":"Pegoraro","year":"2021","journal-title":"IEEE Trans. Geosci. Remote Sens."},{"key":"ref_42","doi-asserted-by":"crossref","first-page":"311","DOI":"10.1109\/JERM.2022.3143431","article-title":"Simultaneous Monitoring of Multiple People\u2019s Vital Sign Leveraging a Single Phased-MIMO Radar","volume":"6","author":"Xu","year":"2022","journal-title":"IEEE J. Electromagn. Microwaves Med. Biol."}],"container-title":["Sensors"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/18\/5865\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,10,10]],"date-time":"2025-10-10T15:52:40Z","timestamp":1760111560000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1424-8220\/24\/18\/5865"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2024,9,10]]},"references-count":42,"journal-issue":{"issue":"18","published-online":{"date-parts":[[2024,9]]}},"alternative-id":["s24185865"],"URL":"https:\/\/doi.org\/10.3390\/s24185865","relation":{},"ISSN":["1424-8220"],"issn-type":[{"value":"1424-8220","type":"electronic"}],"subject":[],"published":{"date-parts":[[2024,9,10]]}}}