Skip to main content
Springer Nature Link
Log in

ALATT-network: automated LSTM-based framework for classification and monitoring of autism spectrum disorder therapy tasks

  • Original Paper
  • Published:
Save article
View saved research
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

Autism Spectrum Disorder affects the overall growth and development of children by limiting their social and cognitive skills, where a child can have low, medium, and high-functioning autism. Depending upon the level of autism, different applied behavioral and cognitive therapies are performed by therapists that may continue for months or years depending upon the severity of autism in a child. Applied Behavior Analysis (ABA) therapy is provided to improve a child’s social, communicational, and behavioral skills. Nevertheless, locating highly skilled therapists for an extended duration proves challenging in the realm of ABA therapy. Moreover, the progress of the child needs to be monitored at every stage of the therapy, which further increases the overall cost and time of the therapy. During the ABA sessions, various tasks e.g., imitation, joint attention, and turn taking are key measurements to analyze a child's progress. It's very challenging to monitor these tasks manually by the therapist and predict the overall child's progress. To overcome this problem, the paper presents a novel deep learning framework based on a Long Short-Term Memory network for the classification of ABA therapy tasks (ALATT-Network) i.e., Imitation, Joint Attention, and Turn-Taking. The proposed framework is trained on a large-scale skeleton dataset consisting of spatial and temporal information on autism therapy sessions. The DREAM dataset is employed for the experiments which is the largest publically available dataset for autism therapy sessions. The framework assists caregivers, doctors, and therapists to monitor the ongoing therapy sessions and predict the child’s progress. The proposed ALATT network employs different five optimizers and analyzes the performance of the network. In the experiment, the results show that the proposed ALATT-Network with Adam optimizer effectively learns the temporal and spatial features of skeleton movements and provides an accuracy of 79.3% for classification tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+
from €37.37 /Month
  • Starting from 10 chapters or articles per month
  • Access and download chapters and articles from more than 300k books and 2,500 journals
  • Cancel anytime
View plans

Buy Now

Price includes VAT (Netherlands)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
The alternative text for this image may have been generated using AI.
Fig. 2
The alternative text for this image may have been generated using AI.
Fig. 3
The alternative text for this image may have been generated using AI.
Fig. 4
The alternative text for this image may have been generated using AI.
Fig. 5
The alternative text for this image may have been generated using AI.
Fig. 6
The alternative text for this image may have been generated using AI.
Fig. 7
The alternative text for this image may have been generated using AI.
Fig. 8
The alternative text for this image may have been generated using AI.
Fig. 9
The alternative text for this image may have been generated using AI.
Fig. 10
The alternative text for this image may have been generated using AI.

Similar content being viewed by others

Explore related subjects

Discover the latest articles, books and news in related subjects, suggested using machine learning.

References

  1. Sealey, L., et al.: Environmental factors in the development of autism spectrum disorders. Environ. Int. 88, 288–298 (2016)

    Article  Google Scholar 

  2. Tanner, A., Dounavi, K.: The emergence of autism symptoms prior to 18 months of age: a systematic literature review. J. Autism Dev. Disord. 51(3), 973–993 (2021)

    Article  Google Scholar 

  3. El Mouatasim, A., Ikermane, M.: Control learning rate for autism facial detection via deep transfer learning. SIViP 17(7), 3713–3720 (2023)

    Article  Google Scholar 

  4. Uddin, M.Z., Shahriar, M.A., Mahamood, M.N., Alnajjar, F., Pramanik, M.I., Ahad, M.A.R.: Deep learning with image-based autism spectrum disorder analysis: a systematic review. Eng. Appl. Artif. Intell. 127, 107185 (2024)

    Article  Google Scholar 

  5. Ion, S.: Social Assistance for Children and Young People with Autism Spectrum Disorders (ASD), Social Work Review/Revista de Asistenta Sociala, no. 2 (2021)

  6. Carpita, B., et al.: The broad autism phenotype in real-life: clinical and functional correlates of autism spectrum symptoms and rumination among parents of patients with autism spectrum disorder. CNS Spectr. 25(6), 765–773 (2020)

    Article  Google Scholar 

  7. Ozsahin, I., Mustapha, M.T., Albarwary, S., Sanlidag, B., Ozsahin, D.U., Butler, T.A.: An investigation to choose the proper therapy technique in the management of autism spectrum disorder. J. Compar. Eff. Res. 10(5), 423–437 (2021)

    Article  Google Scholar 

  8. Foxx, R.M.: Applied behavior analysis treatment of autism: the state of the art. Child Adolesc. Psychiatr. Clin. N. Am. 17(4), 821–834 (2008)

    Article  Google Scholar 

  9. Thoma, N., Pilecki, B., McKay, D.: Contemporary cognitive behavior therapy: a review of theory, history, and evidence. Psychodyn. Psychiatry 43(3), 423–461 (2015)

    Article  Google Scholar 

  10. Karges, J., Smallfield, S.: A description of the outcomes, frequency, duration, and intensity of occupational, physical, and speech therapy in inpatient stroke rehabilitation. J. Allied Health 38(1), 1E-10E (2009)

    Google Scholar 

  11. Steultjens, E.M., Dekker, J., Bouter, L.M., Van de Nes, J.C., Cup, E.H., Van den Ende, C.H.: Occupational therapy for stroke patients: a systematic review. Stroke 34(3), 676–687 (2003)

    Article  Google Scholar 

  12. Smith, T., Mruzek, D.W., Mozingo, D.: Sensory integration therapy. In: Controversial Therapies for Autism and Intellectual Disabilities, pp. 247–269. Routledge (2015)

  13. Jones, L., Rubin, L.: PT 101: teaching introduction to play therapy at the graduate level. Int. J. Play Ther. 14(1), 117 (2005)

    Article  Google Scholar 

  14. Amonkar, N., Su, W.-C., Bhat, A.N., Srinivasan, S.M.: Effects of creative movement therapies on social communication, behavioral-affective, sensorimotor, cognitive, and functional participation skills of individuals with autism spectrum disorder: a systematic review. Front. Psych. 12, 722874 (2021)

    Article  Google Scholar 

  15. Alves, F.J., De Carvalho, E.A., Aguilar, J., De Brito, L.L., Bastos, G.S.: Applied behavior analysis for the treatment of autism: a systematic review of assistive technologies. IEEE Access 8, 118664–118672 (2020)

    Article  Google Scholar 

  16. Virues-Ortega, J., Pérez-Bustamante, A., Tarifa-Rodriguez, A.: Evidence-based applied behavior analysis (ABA) autism treatments: an overview of comprehensive and focused meta-analyses. In: Handbook of Autism and Pervasive Developmental Disorder: Assessment, Diagnosis, and Treatment, pp 631–659 (2022)

  17. Billing, E.: The DREAM dataset: behavioural data from robot enhanced therapies for children with autism spectrum disorder (2020)

  18. Abbas, H., Garberson, F., Liu-Mayo, S., Glover, E., Wall, D.P.: Multi-modular AI approach to streamline autism diagnosis in young children. Sci. Rep. 10(1), 5014 (2020)

    Article  Google Scholar 

  19. Nogay, H.S., Adeli, H.: Machine learning (ML) for the diagnosis of autism spectrum disorder (ASD) using brain imaging. Rev. Neurosci. 31(8), 825–841 (2020)

    Article  Google Scholar 

  20. Mujeeb Rahman, K., Subashini, M.M.: Identification of autism in children using static facial features and deep neural networks. Brain Sci. 12(1), 94 (2022)

    Article  Google Scholar 

  21. Lu, A., Perkowski, M.: Deep learning approach for screening autism spectrum disorder in children with facial images and analysis of ethnoracial factors in model development and application. Brain Sci. 11(11), 1446 (2021)

    Article  Google Scholar 

  22. Mahapatra, P., Pati, S., Sinha, R., Chauhan, A.S., Nanda, R.R., Nallala, S.: Parental care-seeking pathway and challenges for autistic spectrum disorders children: a mixed method study from Bhubaneswar, Odisha. Indian J. Psychiatry 61(1), 37–44 (2019)

    Google Scholar 

  23. Ismail, N.A.S., Ramli, N.S., Hamzaid, N.H., Hassan, N.I.: Exploring eating and nutritional challenges for children with autism spectrum disorder: parents’ and special educators’ perceptions. Nutrients 12(9), 2530 (2020)

    Article  Google Scholar 

  24. Anagnostopoulou, P., Alexandropoulou, V., Lorentzou, G., Lykothanasi, A., Ntaountaki, P., Drigas, A.: Artificial intelligence in autism assessment. Int. J. Emerg. Technol. Learn. (iJET) 15(6), 95–107 (2020)

    Article  Google Scholar 

  25. Elbattah, M., Guérin, J.-L., Carette, R., Cilia, F., Dequen, G.: Vision-based approach for autism diagnosis using transfer learning and eye-tracking. In: HEALTHINF, pp. 256–263 (2022)

  26. Michelassi, G.C. et al.: Classification of facial images to assist in the diagnosis of autism spectrum disorder: a study on the effect of face detection and landmark identification algorithms. In: Brazilian Conference on Intelligent Systems, pp. 261–275. Springer (2023)

  27. Rakić, M., Cabezas, M., Kushibar, K., Oliver, A., Lladó, X.: Improving the detection of autism spectrum disorder by combining structural and functional MRI information. NeuroImage Clin. 25, 102181 (2020)

    Article  Google Scholar 

  28. Banire, B., Al Thani, D., Qaraqe, M., Mansoor, B.: Face-based attention recognition model for children with autism spectrum disorder. J. Healthc. Inf. Res. 5(4), 420–445 (2021)

    Article  Google Scholar 

  29. Wu, C., et al.: Machine learning based autism spectrum disorder detection from videos. In: 2020 IEEE International Conference on E-health Networking, Application and Services (HEALTHCOM), pp. 1–6. IEEE (2021)

  30. Nogay, H.S., Adeli, H.: Multiple classification of brain MRI autism spectrum disorder by age and gender using deep learning. J. Med. Syst. 48(1), 15 (2024)

    Article  Google Scholar 

  31. Lakhan, A., Mohammed, M.A., Abdulkareem, K.H., Hamouda, H., Alyahya, S.: Autism spectrum disorder detection framework for children based on federated learning integrated CNN-LSTM. Comput. Biol. Med. 166, 107539 (2023)

    Article  Google Scholar 

  32. Ali, S., et al.: An adaptive multi-robot therapy for improving joint attention and imitation of ASD children. IEEE Access 7, 81808–81825 (2019)

    Article  Google Scholar 

  33. Guo, Z., Kim, K., Bhat, A., Barmaki, R.: An automated mutual gaze detection framework for social behavior assessment in therapy for children with autism. In: Proceedings of the 2021 International Conference on Multimodal Interaction, pp. 444–452 (2021)

  34. Portnova, G.V., Ivanova, O., Proskurnina, E.V.: Effects of EEG examination and ABA-therapy on resting-state EEG in children with low-functioning autism. AIMS Neurosci. 7(2), 153 (2020)

    Article  Google Scholar 

  35. Penchina, B., Sundaresan, A., Cheong, S., Martel, A.: Deep LSTM recurrent neural network for anxiety classification from EEG in adolescents with autism. In: International Conference on Brain Informatics, pp. 227–238. Springer (2020)

  36. Qayyum, A., et al.: An efficient 1DCNN–LSTM deep learning model for assessment and classification of fMRI-based autism spectrum disorder. In: Innovative Data Communication Technologies and Application: Proceedings of ICIDCA 2021, pp. 1039–1048. Springer (2022)

  37. Li, J., Zhong, Y., Han, J., Ouyang, G., Li, X., Liu, H.: Classifying ASD children with LSTM based on raw videos. Neurocomputing 390, 226–238 (2020)

    Article  Google Scholar 

  38. Esqueda-Elizondo, J.J., et al.: Attention measurement of an autism spectrum disorder user using EEG signals: a case study. Math. Comput. Appl. 27(2), 21 (2022)

    Google Scholar 

  39. van Otterdijk, M.T., et al.: The effects of long-term child–robot interaction on the attention and the engagement of children with autism. Robotics 9(4), 79 (2020)

    Article  Google Scholar 

  40. Wang, M., Yang, N.: OBTAIN: observational therapy-assistance neural network for training state recognition. IEEE Access 11, 31951–31961 (2023)

    Article  Google Scholar 

  41. Saha, P., Tapotee, M.I., Ahad, M.A.R.: Task detection of ASD children by analyzing robotic enhanced and standard human therapy. In: 2021 Thirteenth International Conference on Mobile Computing and Ubiquitous Network (ICMU) , pp. 1–6. IEEE (2021)

  42. Yu, Y., Si, X., Hu, C., Zhang, J.: A review of recurrent neural networks: LSTM cells and network architectures. Neural Comput. 31(7), 1235–1270 (2019)

    Article  MathSciNet  Google Scholar 

  43. Zhuang, J., et al.: Adabelief optimizer: adapting stepsizes by the belief in observed gradients. Adv. Neural. Inf. Process. Syst. 33, 18795–18806 (2020)

    Google Scholar 

  44. Yao, Z., Gholami, A., Shen, S., Mustafa, M., Keutzer, K., Mahoney, M.: Adahessian: an adaptive second order optimizer for machine learning. Proc. AAAI Conf. Artif. Intell. 35(12), 10665–10673 (2021)

    Google Scholar 

  45. Chandra, K., Xie, A., Ragan-Kelley, J., Meijer, E.: Gradient descent: the ultimate optimizer. Adv. Neural. Inf. Process. Syst. 35, 8214–8225 (2022)

    Google Scholar 

  46. Li, J., Wang, B.: Policy optimization with second-order advantage information. arXiv:1805.03586 (2018)

  47. Huk, M.: Stochastic optimization of contextual neural networks with RMSprop. In: Intelligent Information and Database Systems: 12th Asian Conference, ACIIDS 2020, Phuket, Thailand, March 23–26, 2020, Proceedings, Part II 12, 2020, pp. 343–352. Springer

  48. Reddy, D.J.P., Gunasekaran, M., Sundari, K.S.: An effective approach for the prediction of car loan default based-on accuracy, precision, recall using extreme logistic regression algorithm and K-nearest neighbors algorithm on financial institution loan dataset. In: 2022 International Conference on Cyber Resilience (ICCR), pp. 1–5. IEEE (2022)

  49. Verine, A., Negrevergne, B., Pydi, M.S., Chevaleyre, Y.: Training normalizing flows with the precision-recall divergence. arXiv:2302.00628 (2023)

  50. Yu, W., Kim, I.Y., Mechefske, C.: Analysis of different RNN autoencoder variants for time series classification and machine prognostics. Mech. Syst. Signal Process. 149, 107322 (2021)

    Article  Google Scholar 

  51. Khan, M., Wang, H., Ngueilbaye, A., Elfatyany, A.: End-to-end multivariate time series classification via hybrid deep learning architectures. Pers. Ubiquit. Comput. 27(2), 177–191 (2023)

    Article  Google Scholar 

  52. Zahan, S., Gilani, Z., Hassan, G.M., Mian, A.: Human gesture and gait analysis for autism detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3328–3337 (2023)

Download references

Acknowledgements

This work is supported by Princess Nourah bint Abdulrahman University Researchers Supporting Project number (PNURSP2024R506), Princess Nourah bint Abdulrahman University, Riyadh, Saudi Arabia. The authors extend their appreciation to the Deanship of Research and Graduate Studies at King Khalid University for funding this work through the Large Research Project under Grant Number RGP2/283/45.

Author information

Authors and Affiliations

  1. School of Mechanical and Manufacturing Engineering (SMME), NUST, Islamabad, Pakistan

    Ayesha Kanwal, Kashif Javed & Sara Ali

  2. Department of Artificial Intelligence, College of Computer Engineering and Science, Prince Mohammad Bin Fahd University, Al Khobar, Saudi Arabia

    Muhammad Attique Khan

  3. Information Systems Department, College of Computer and Information Sciences, Princess Nourah Bint Abdulrahman University, P.O. Box 84428, 11671, Riyadh, Saudi Arabia

    Shrooq Alsenan

  4. College of Computer Science, King Khalid University, 61413, Abha, Saudi Arabia

    Areej Alasiry & Mehrez Marzougui

  5. Department of Computer Engineering, College of Computing and Informatics, University of Sharjah, 27272, Sharjah, UAE

    Saddaf Rubab

Authors
  1. Ayesha Kanwal
    View author publications

    Search author on:PubMed Google Scholar

  2. Kashif Javed
    View author publications

    Search author on:PubMed Google Scholar

  3. Sara Ali
    View author publications

    Search author on:PubMed Google Scholar

  4. Muhammad Attique Khan
    View author publications

    Search author on:PubMed Google Scholar

  5. Shrooq Alsenan
    View author publications

    Search author on:PubMed Google Scholar

  6. Areej Alasiry
    View author publications

    Search author on:PubMed Google Scholar

  7. Mehrez Marzougui
    View author publications

    Search author on:PubMed Google Scholar

  8. Saddaf Rubab
    View author publications

    Search author on:PubMed Google Scholar

Contributions

All authors contributed equally in this work. Thank you.

Corresponding author

Correspondence to Muhammad Attique Khan.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Data availability statement

https://github.com/dream2020/data.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kanwal, A., Javed, K., Ali, S. et al. ALATT-network: automated LSTM-based framework for classification and monitoring of autism spectrum disorder therapy tasks. SIViP 18, 9205–9221 (2024). https://doi.org/10.1007/s11760-024-03540-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Version of record:

  • Issue date:

  • DOI: https://doi.org/10.1007/s11760-024-03540-3

Keywords