Journal of Speech Pathology & Therapy
Open Access

Automatic emotion recognition; Children with autism; Systematic literature review; Observation channels; Modalities; Qualitative analysis; Quantitative analysis

Introduction

Autism spectrum disorder (ASD) is a neurodevelopmental disorder that can manifest in various forms and degrees throughout a person’s life [1]. The Diagnostic and Statistical Manual of Mental Disorders (DSM V) identifies deficits in social-emotional reciprocity as a key characteristic of individuals with ASD, ranging from abnormal social approaches and difficulties in engaging in back-and-forth conversations to a complete lack of initiating social interactions [2]. The social communication challenges faced by individuals with autism are partly rooted in deficits in emotion recognition (ER), which contribute to difficulties in processing and interpreting socio-emotional cues [3]. These individuals often struggle with social cognition, including identifying facial expressions, recognizing faces, discriminating between faces, and remembering faces. Consequently, they may experience increased stress and anxiety, exhibit abnormal perceptions of faces, and have impaired emotional processing [4,5]. The project’s primary objective is to explore the feasibility of implementing emotion recognition technologies in robot-supported interventions for children with autism, aiming to establish an affective loop in child-robot interactions. The study presented in this paper specifically focuses on automatic emotion recognition applied to children with autism, not only in the context of child-robot interaction applications but also with the aim of enabling robots to perceive emotions effectively. The objective of this paper is to present the findings of a systematic literature review that focuses on exploring the current state of automatic emotion recognition technologies applied to detect emotions in children with autism. To clarify, the studies of interest in this field pertain to automatically recognizing emotions experienced by autistic children, rather than the capacity of children to recognize emotions in others. As per the definition provided in [6], automatic emotion recognition refers to an interdisciplinary research area that involves algorithmically detecting human affect, such as anger or sadness, using various sources like speech or facial gestures. Three literature reviews [7-9] serve as the basis for our study. In the work by Kowallik A. E. and Schweinberger S. R. [7], the authors review papers related to sensor-based social information processing, with a focus on using sensors to diagnose autism and support intervention. Although some of the mentioned intervention papers touch on emotion recognition, it is not the primary focus. Nevertheless, the modalities discussed align with those used in automatic emotion recognition. Chaidi I. and Drigas A. [8] conduct a literature review on both the expression and understanding of emotions in autism, specifically referring to the perception of emotions by children with autism, not the recognition of their emotions. On the other hand, the study by Rashidan et al. [9] concentrates on emotion recognition applied to children with autism, raising research questions about stimuli used and feature extraction methods, complementing our work. Importantly, the reviews do not include challenges and recommendations for utilizing emotion recognition technologies, which is a crucial aspect of our study. In our paper, when referring to emotion recognition, we specifically mean the automatic recognition of emotions in children. This paper presents a systematic literature review [10,11] on the topic of automatic emotion recognition in children with autism. The study follows the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) [12] standard for reporting the research, and the structure is organized as follows: Section 2 describes the research methods and the execution of the systematic literature review. Section 3 presents the quantitative and qualitative results. Following the results, there is a discussion on research validity, as well as an outline of challenges that future research may address.

Methods

In this study, a systematic literature review was employed as the methodological approach to comprehensively assess the current state of the domain under investigation. The systematic method was chosen to ensure that key studies were identified and reviewed with transparency and rigor, facilitating the replication of the study. Following the PRISMA approach, the following steps were conducted:

1. Setting up research questions: Clear and specific research questions were formulated to guide the review process.

2. Defining keywords and search string: Relevant keywords and a comprehensive search string were established to effectively retrieve relevant literature.

3. Inclusion and exclusion criteria: Criteria were defined to determine which studies would be included or excluded from the review based on their relevance to the research questions.

4. Selection of search engines: The appropriate search engines and databases were chosen to gather relevant scholarly publications.

5. Data extraction: Information from selected papers was systematically extracted for analysis.

6. Multiple-phase selection based on quality criteria and research questions: A multi-phase selection process was carried out, considering both quality criteria and alignment with the research questions to ensure the inclusion of pertinent studies.

7. Final selection of papers and snowballing technique: The final set of papers meeting the inclusion criteria was selected.

Additionally, the snowballing technique was employed to identify any additional relevant studies from the reference lists of the selected papers.

8. Extraction of key findings: Relevant findings from the selected studies were extracted and analyzed to contribute to the overall conclusions of the systematic literature review.

The systematic literature review presented in this paper offers valuable insights into the application of automatic emotion recognition technologies in studying and training children with autism. The findings shed light on various aspects, including observation channels, modalities, methods, challenges, and implications for this rapidly advancing field.

Observation channels, modalities, and methods

The reviewed studies demonstrate a wide range of observation channels and modalities employed for automatic emotion recognition in children with autism. These include facial expressions, prosody of speech, and physiological signals, among others. The prevalence of basic emotions, such as happiness, fear, and sadness, in the recognized emotions is noteworthy. The preference for single-channel approaches in the analyzed studies indicates a common trend in the field [13].However, it is evident that more research is needed to explore the potential benefits of multimodal approaches, which could enhance the accuracy and robustness of emotion recognition systems.

Challenges and limitations

The review uncovers several challenges and limitations associated with applying emotion recognition technologies in studies involving children on the autism spectrum. Disturbances reported in all channels may result in temporary or permanent unavailability of specific emotional symptom information. This highlights the need for developing robust and adaptive algorithms that can account for variations in data quality. Additionally, the creation of appropriate stimuli and labeling methods tailored specifically for children with autism presents a significant challenge. The development of open datasets exclusively focused on this special group can significantly contribute to the advancement of research in this area. The systematic literature review uncovers a number of challenges associated with applying emotion recognition to children on the autism spectrum. Additionally, some commendable practices were identified. The findings have implications for both scientific research and practical applications. Future research may explore diverse stimuli specifically tailored for this special group, adjusted to their reactivity type. Moreover, more extensive investigations into multimodal approaches could yield practical outcomes [14]. Notable concerns for further studies include participant group construction, considering factors such as sex, developmental age, and level of functioning, as well as exploring mixed (compound) labelling approaches. The creation of datasets exclusively focused on children with autism and the investigation of emotions of real interest in the studied interactions, beyond a basic emotions model, are also areas worth considering.

Implications for future research and practical applications

The systematic literature review has significant implications for both scientific research and practical applications. Researchers should consider exploring diverse stimuli tailored to the reactivity type of children with autism to ensure accurate emotion recognition. Furthermore, conducting studies that focus on a broader range of emotions beyond the basic emotions model could lead to a more comprehensive understanding of emotional experiences in these children. The integration of multimodal approaches should be given greater attention as it holds the potential for improving the effectiveness of emotion recognition systems [15]. The findings of this review also have practical implications for therapy and intervention programs for children with autism. Emotion recognition technologies integrated with social robots could facilitate more effective and personalized interventions, supporting the emotional development and social communication skills of these children. However, it is crucial to address the challenges identified in the review to ensure the successful implementation of such technologies in real-world settings.

Conclusion

In conclusion, the systematic literature review on automatic emotion recognition for children with autism provides a comprehensive overview of the current state of the field. The diverse observation channels, modalities, and methods utilized in the reviewed studies showcase the potential of these technologies in enhancing therapy and interventions for children with autism. However, challenges related to data quality, stimuli, and labeling methods must be addressed to maximize the benefits of emotion recognition technologies. Moving forward, researchers should focus on developing multimodal approaches and open datasets tailored to the unique needs of children with autism, to advance our understanding and application of automatic emotion recognition in this population. This paper presents a comprehensive systematic literature review on the challenge of automatic emotion recognition in the context of studying and training children with autism. Initially, over 2000 papers were extracted from 7 search engines, ultimately resulting in the inclusion of 50 papers for qualitative analysis and 27 for quantitative analysis. The study sheds light on crucial observations concerning observation channels, modalities, and methods employed for emotion recognition in children with autism. Qualitative analysis provides valuable insights into participant group construction and the prevailing combinations of modalities and methods. Researchers engaged in emotion recognition or seeking to improve affect classification methods in autism-related studies would find this review particularly relevant.

Acknowledgement

Not applicable.

Conflict of Interest

Author declares no conflict of interest.

1. Jordan R (1999) Autistic Spectrum Disorders: An Introductory Handbook for Practitioners. David Fulton Publishers, London, UK.

Google Scholar, Crossref

2. American Psychiatric Association (2013) Diagnostic and Statistical Manual of Mental Disorders. American Psychiatric Association, Washington, DC, USA.

Google Scholar, Crossref

3. Fridenson-Hayo S, Berggren S, Lassalle A, Tal S, Pigat D, et al. (2016) Basic and complex emotion recognition in children with autism: Cross-cultural findings. Mol Autism 7:52.

Indexed at, Google Scholar, Crossref

4. Corbett BA, Carmean V, Ravizza S, Wendelken C, Henry ML, et al. (2009) A functional and structural study of emotion and face processing in children with autism. Psychiatry Res 173:196-205.

Indexed at, Google Scholar, Crossref

5. Karpus A, Landowska A, Miler J, Pykala M (2020) Systematic Literature Review-Methods and Hints.

Google Scholar

6. Kowallik AE, Schweinberger SR (2019) Sensor-Based Technology for Social Information Processing in Autism: A Review. Sensors 19:4787.

Indexed at, Google Scholar, Crossref

7. Chaidi I, Drigas A (2020) Autism, Expression, and Understanding of Emotions: Literature Review. Int J Online Eng 16:94-111.

Google Scholar, Crossref

8. Rashidan MA, Sidek SN, Yusof HM, Khalid M, Dzulkarnain AAA, et al. (2021) Technology-Assisted Emotion Recognition for Autism Spectrum Disorder (ASD) Children: A Systematic Literature Review. IEEE Access 9:33638-33653.

Google Scholar, Crossref

9. Gu S, Wang F, Patel NP, Bourgeois JA, Huang JH (2019) A Model for Basic Emotions Using Observations of Behavior in Drosophila. Front Psychol 10:781.

Indexed at, Google Scholar, Crossref

10. Ekman P (1984) Expression and the nature of emotion. Approaches to Emotion 3:319–344.

Google Scholar

11. Ekman P (1992) Are there basic emotions? Psychol Rev 99:550-553.

Indexed at, Google Scholar, Crossref

12. Russell JA (1980) A circumplex model of affect. J Personal Soc Psychol 39:1161-1178.

Google Scholar, Crossref

13. Javed H, Jeon M, Park CH (2018) Adaptive Framework for Emotional Engagement in Child-Robot Interactions for Autism Interventions. 15th International Conference on Ubiquitous, US.

Google Scholar

14. Kim JC, Azzi P, Jeon M, Howard AM, Park CH (2017) Audio-based emotion estimation for interactive robotic therapy for children with autism spectrum disorder. 14th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI), Korea.

Google Scholar, Crossref

15. Liu C, Conn K, Sarkar N, Stone W (2007) Online Affect Detection and Adaptation in Robot Assisted Rehabilitation for Children with Autism. RO-MAN 2007–The 16th IEEE International Symposium on Robot and Human Interactive Communication (IEEE), Korea.

Google Scholar, Crossref