Assessment of Executive Functions in Preschool-Aged Children with Autism Spectrum Disorders: Usefulness and Limitation of BRIEF-P in Clinical Practice

The Executive Functions (EF) are a set of mental processes that are responsible for the control and the monitoring of higher-level cognitive, emotional and behavioral skills, characterized by voluntary control processes and organized according to a goal oriented to problem solving. These functional modules of the mind regulate the planning, control and coordination processes of the cognitive system and govern the activation and modulation of cognitive schemes and processes. Welsh and Pennington [1] described the early development of executive functions in terms of ability to maintain a suitable system of problem solving to achieve objectives. Stuss and Benson [2] presented a set of interrelated skills in intentional problem solving that includes the anticipation, the goal selection, the planning, the monitoring and use of feedback.

The hierarchical model proposed by the authors highlights important aspects of executive functions involved in complex cognitive levels as the anticipation, the evaluation, the self-awareness and the decision making. The specific sub-domains, which constitute this set of regulation and control functions, include the ability to start a behavior, to inhibit actions or competing stimuli, to select, plan and organize solutions, flexibly change cognitive and behavioral strategies during the evaluation of the procedures used. It is worth noting that the executive functions are not exclusive of the operation and of cognitive control, but are involved in the regulation of the emotional and behavioral responses so fundamental in the development of the socialcognitive, emotional and relational capacity to adapt. In neurodevelopment, the EF gradually emerge and change across the lifespan of an individual and can be improved or adversely affected at any time by a variety of events. Then, the neurodevelopmental disorders seem to define uneven patterns of functioning in preschool children and adolescents [3,4]. The direct clinical evaluation of the EF has been very problematic because of the variability and dynamic components that distinguish them, especially in the preschool for the limitations in verbal and motor skills in this age group [5]. For this reason the informations coming from parents, teachers and/or therapists who have an ongoing relationship with the child in everyday situations, may be useful in terms of ecological validity, as not intrusive, and allows the observation of current or emergent behaviors.

According to the literature, since the executive functions are crucial in social adaptation processes, in cognitive and linguistic development and interpersonal and communication skills, their assessment could be an important element in defining profiles of individual development in children with Autism Spectrum Disorders (ASDs), where the neurodevelopment is characterized by social and cognitive deficits, with severe socio-emotional reciprocity and communication impairments, in the presence of restricted and repetitive behaviors [6]. In conclusion, the objective of this research was to identify the impairment of the EF assessment of children with ASD, by correlating the scores obtained from parents and professionals, and evaluate the relation between the EF impairment and the Autism Diagnostic Observation Schedule-Second Edition (ADOS-2) scores, with children’s age, IQ and duration of therapy. We would, in fact,investigate the correlations between the development of executive functions and the autistic symptoms, the level of cognitive development, the chronological age and the time of therapy.

Participants

Participants of the study were 46 preschool children, aged between 24 and 71 months (M=42.74 ± 13.4). All children received an Autism Spectrum Disorder (ASDs) diagnosis according both to the ADOS-2 scores and to the clinical assessment based on the DSM-5 diagnostic criteria. The children involved in the research were attending the school on average from 5.8 months (SD=7.7; range in months: 0-33).

The Table 1 describes the demographic characteristics of the sample. As expected from the ADOS-2, according to chronological age and level of expressive language, 30 of 46 children were assessed with ADOS-2 Module 1 and achieved an average score of 18:43 ± 4.9 (range 9-26); 2 children were assessed with the ADOS-2 Module-2 and achieved an average score of 7.50 ± 0.7 (range 7-8), and 14 children were assessed with the Toddler Form of the ADOS-2 obtaining an average score of 18.80 ± 5.2 (range 6-25). In the sample there were 36 males and 10 females consistent with a recent research about ASD indicating a 3.1: 1 male to female prevalence ratio (Nicholas et al. 2008). At the time of the first assessment, in children it was not present a form of verbal language, so their cognitive level was evaluated through the Leiter International Performance Scale (Leiter-R). 28 of the 46 children in the sample have one or more siblings. Of these, 10 children have a sibling with specific needs (4 children have a twin with ASD; 3 children have a sibling with speech disorder; 3 children have a sibling with emotional disorder).

Table 1: Descriptive characteristics of 46 children with autism spectrum disorder.

The autism diagnostic observation schedule, second edition

The Autism Diagnostic Observation Schedule-Second Edition (ADOS-2) [7] is a semi-structured standardized assessment concerning the areas of communication, social interaction, play, and repetitive and restricted behaviors. It is considered a gold standard so being the most commonly standardized measure used in research protocols. It has strong psychometric properties, including good reliability and validity. The ADOS-2 includes five modules, but during the assessment process, just one module is administered and it is selected on the basis of the expressive language level and chronological age of the child:

Toddler Module: For children between 12 and 30 months of age who do not constantly use sentences in their speech. This module allows you to precisely identify children at risk of autism spectrum disorders (ASD);

Module 1: For children aged from 31 months who do not continuously use sentences in their speech;

Module 2: For children of all ages who use sentences in the speech, without being verbally fluent;

Module 3: For verbally fluent children and adolescents;

Module 4: For verbally fluent older adolescents and adults.

In Modules 1 to 4, the algorithm scores are compared to get one of three classifications: Autism, Autism Spectrum, and Not Spectrum. The cut-off for the ADOS-2 classifications ADOS-2 varies according to the module used and to the child's language level. The ADOS-2, in addition to providing an overall score, also measures the domain of Social Affection (SA) and the domain of Repetitive and Restricted Behaviors (RRB). The SA includes the evaluation of aspects related to communication and reciprocal social interaction. The RRB includes unusual sensory interests, mannerisms, repetitive interests and behaviors and stereotyped-idiosyncratic use of words and/or vocalizations.

Leiter-R

The Leiter International Performance Scale-Revised (Leiter-R) [8] is designed for the evaluation of intellectual functions of children and adolescents aged between 2 and 20 years. The Leiter-R is formulated to meet the clinical need of evaluating the non-verbal intelligence through a comprehensive analysis of the strengths and weaknesses of the child, with a view to a diagnosis that includes both neuropsychological and cognitive aspects. Nonverbal cognitive abilities do not require the ability to perceive, manipulate and reason with words and numbers, so that the scale can be administered completely without the use of verbal language, including instructions, because it does not require verbal responses from the subject. The validity coefficients of the IQ score was calculated for each age group (alpha .92 to .93). The Leiter-R has good evidence of its validity, with data from a wide analysis from the criterion related studies, the accuracy of classification of intellectual disability, and various studies related to the construct. The IQ score has a mean of 100 and a standard deviation of 15. The intellectual disability is indicated by a composite score that is two standard deviations or more below the average, so that a score of 70 constitutes the borderline value.

Behavior rating inventory of executive functions-preschool version (BRIEF-P)

The Behavior Rating Inventory of Executive Functions-Preschool Version (BRIEF-P) [9] is a 63-item informant report of Executive Functions in real world situations comprised of five scales: Inhibit, Shift, Emotional Control, Working Memory, Planning/Organization. All subscales are collapsed into three broad indices: Inhibitory Self- Control Index (ISCI), Flexibility Index (FI), and Emergent Metacognition Index (EMI). Inhibit and Emotional Control scales comprise the ISCI, the Shift and Emotional Control scales comprise the FI, and the Working Memory and Planning/Organization scales comprise the EMI. The five scales are then collapsed into a Global Executive Composite (GEC). The BRIEF-P asks parents to assign a rating of "Never", "Sometimes" or "Often" to each item, reflecting how much of a problem the behavior poses in their child. These ratings are then converted to a 1, 2, or 3 and results are reported as T-scores. Higher scores on scales indicated greater impairment; T-scores ≥65 (i.e., 1.5 Standard Deviations [SD] ≥ the mean) show clinically significant ratings [10].

Children with autism spectrum disorder and their families were recruited during the period from January 2014 to January 2016, at the Institute of Ortofonologia (IdO). The IdO works in agreement with the National Health System for taking charge of ASD children and their families. The Autism diagnosis was made by a group of experts (psychologists, neuropsychiatrists, speech therapists and occupational therapists) according to the DSM-5 criteria [6] and confirmed by the ADOS-2 evaluations. Psychologists, psychotherapists and neuropsychiatrists took care of the clinical assessment and of the psychodiagnosis; parents and therapists who deal with the therapy and the home care intervention for at least three months completed the rating scales on the development of executive functions. All children who exceeded the ADOS-2 clinical cut-off for the risk and/or for the autism spectrum disorder diagnosis were included in the study. Children with brain injury, genetic and metabolic disorders, sensory disabilities, children over 72 months and under 24 months were not taken into account in this research.

In the assessment process of their child (from 1 to 4 weeks from the first clinical consultation), parents are informed about the characteristics of the autism spectrum disorder in general and of their child in particular. When the children begin the therapeutic program (immediately after the diagnostic phase) they are involved in a counseling program that provides, according to their needs, also group meetings with other parents or therapy sessions together with their children. This research meets the APA ethic guidelines. This study was carried out in accordance with the recommendations of the 'name of guidelines, name of committee' with written informed consent from all subjects, in agreement with the Declaration of Helsinki.

Data analysis

We used the Statistical Package for Social Sciences (SPSS) version 19 for data analysis. Significance level was in September at 12:05 alpha (two-tailed). Multivariate Analysis of Variance (MANOVA) was used to evaluate differences between respondents on BRIEF-P subscales. Effect sizes were reported as partial eta squared (η2p), A η2p of 0.02 was considered a small effect size, 0.13 a medium effect size and 0.23 a large effect size. Correlation analysis was performed to analyze the relationship between executive function, ASD symptoms, social affection, restricted and repetitive behaviors, cognitive development, chronological age and therapy duration.

Aims

The general objective of this study was to verify the level of agreement in the BRIEF-P completion in the assessment of the Executive Functions (EF) when used by professionals and parents of preschool children with autism spectrum disorder and the possible correlation between the degree of EF impairment and other developmental parameters or disease. Specifically we would:

1. Verify if the scores provided by parents and professionals to the BRIEF-P were related to ADOS-2 and the Leiter-R scores;

2. Verify if the scores provided by parents and professionals to BRIEF-P were related to children's age and therapy duration;

3. Verify if the ASD children presented clinically pathological levels in the different domains of the EF, and in particular whether there were specific domains more compromised than others in parents and professionals’ evaluations;

4. Verify if parents and professionals’ evaluations were mutually consistent.

Descriptive analysis

Children of the research sample were divided, depending on the ADOS-2 cut-off, in AUT children (N=26; ADOS-2 range 12-26), SpD children (N=7; ADOS-2 range 7-14) and RISK children (N=13; ADOS-2 range 6-25). We calculated the average of the IQ, ADOS-2, RRB and SA scores and the descriptive analysis showed that AUT children have an IQ score significantly lower than children from the other two subgroups; AUT and RISK children have a higher ADOS-2 score (thus more severe) than the SpD children, both as regards for the total score that for the SA and the RRB (Table 2).

Table 2: Mean (SD) and between-group differences in IQ, total ADOS-2, SA and RRB scores.

Parents report: The relationship between the BRIEF-P, the ADOS-2, the Leiter-R scores, the chronological age and the therapy duration.

As reported in Table 3, the analysis of correlations between the scores at the BRIEF-P completed by parents and the ADOS-2 scores show that only the Emergent Metacognition (EMI) domain and of its subscale, the Working Memory, positively correlates with the ADOS-2 total score and with the scores of Repetitive Behaviors and Social Affection. As for the relationship with cognitive development, the IQ positively correlates only with the domain of Emerging Metacognition. Increasing children’s chronological age, parents evaluate as compromised the Global Executive Functions (GEC), the Inhibitory Self-Control (ISCI) domain and its subscales Inhibit and Emotional Control, the Flexibility (FI) domain and its Shift subscale. Increasing children’s period of therapy, parents identify as compromised the ISCI, the FI and the Emotional Regulation domains.

Table 3: Correlations between the BRIEF-P scores completed by Parents and total ADOS scores, Social Affective behavior, Restricted Repetitive Behavior, IQ, Chronological age and Months of therapy.

Professionals report: The relationship between the BRIEF-P, the ADOS-2, the Leiter-R scores, the chronological age and the therapy duration.

As reported in Table 4, the analysis of correlations between the scores at the BRIEF-P completed by professionals and the ADOS-2 scores shows that only the Metacognition domain and of its subscale Plan/Organize, positively correlates with the ADOS-2 total score and with scores of Repetitive Behaviors and Social Affection.

Table 4: Correlations between the BRIEF-P scores completed by Professionals and total ADOS scores, Social Affective behavior, Restricted Repetitive Behavior, IQ, Chronological age and Months of therapy.

In addition, the analysis shows that the lower the IQ scores of children are, the most the professionals evaluate as compromised the executive functions, both in the Global Executive Composite (GEC) and in ISCI and FI, as well as in Shift and Emotional Control. As for the chronological age of the children, we found that increasing it, professionals evaluate as more compromised the ISCI and FI domains as well as the Inhibit and Emotional Control areas. Finally, the therapy duration is not correlated with the professionals’ BRIEF-P scores. Since it was found a high correlation between the professionals’ BRIEF-P and IQ scores, we verified the sensitivity and specificity of the questionnaire with respect to its ability to discriminate children above and below a specific IQ cut-off. The cut-off (IQ score 67) has been chosen because it represents the 50° percentile of the normal distribution of the IQ scores in this research sample. From the analysis

Differences between parents and professionals BRIEF-P scores

ANOVA for repeated measures shown that parents give children significantly lower scores than those attributed by professionals. In general, in all the domains measured by the BRIEF-P, parents evaluate children as mostly adequate in the executive functions while professionals as primarily pathological (Table 5). Moreover, the percentage of children that exceed the pathological cut-off score of the of the GEC (that is the BRIEF-P overall score) emerged from the professionals’ evaluations, it appeared that the questionnaire has good sensitivity but low specificity in identifying children with high and low IQ (35% specificity, 85% sensitivity).

Table 5: Differences (mean, SD) between the BRIEF-P scores completed by parents and teachers.

BRIEF-P in the evaluations of the Professional is higher when compared to those of the Parent.

Correlations between parents and professionals BRIEF-P scores

The analysis of the correlations shows that the areas in which parents and operators are in agreement are the Inhibit, the Emotional Control and the Inhibitory Self-Control Index. In the other areas, no correlations were found between parents and professionals evaluations.

In addition, Table 6 shows the Cohen’s Kappa coefficient value, as an index of coherence between the two evaluations.

Table 6: Correlations between the BRIEF-P scores completed by parents and professionals.

The results of our work show that both in parents and professionals BRIEF-P protocols, the lack of correlation between the executive functions evaluations and the ADOS-2 scores, which are indicative of symptoms severity, is coherent with the data found in in other studies [11,12] which showed that the level of autism severity is not related with the executive functions deficits in pre-school age, as well as a lower level of autism severity does not necessarily correspond to children’s good executive functioning skills. Specifically, in the parents evaluation the BRIEF-P domain that is most correlated with the ADOS-2 score is the EMI, which is indicative of greater cognitive rigidity (or less flexibility) and therefore of the tendency to repetitiveness and of a low planning and information manipulation.

In our study, moreover, the correlations between IQ and executive impairment were highly significant when evaluated by the professionals, and this would demonstrate the homogeneity of the two parameters, both "cognitive". In addition, the analysis shows that the lower nonverbal IQ scores of children corresponded to compromised executive functions, both in the overall score (GEC), both in ISCI and FI, and in Inhibit, Shift and Emotional Control. Conversely the BRIEFP domains that are more sensitive with respect to the IQ, or ISCI and FI, denote a lower ability in Emotional Control, Inhibit and Shift.

In professionals’ evaluations the EMI, which is expression of skills related to working memory and the ability to initiate, plan and support problem solving, positively correlates with RBB and the SA. This represents a confirmation of how much metacognitive abilities are impaired in autism spectrum disorders and shows how the presence of mannerisms, stereotypies and sensory seeking behaviors are connected both to the flexibility of the mental strategies, and communication and social responsivity [13-15]. Other results confirmed the relationship between executive functioning and metacognitive skills in autism, although not through the use of the BRIEF-P [16] highlighting difficulties in executive functions also in neurodevelopmental disorders [17,18], but emphasizing the importance of considering the individual differences and the child's level of functioning and adaptation in everyday situations [19], considering that complex cognitiveemotional behaviours have their basis in dynamic coalitions of networks of brain areas, none of which should be conceptualized as specifically affective or cognitive [20].

Finally, in most cases there is a low correlation between the BRIEF-P scores obtained by parents and by therapists of preschool children with ASDs. Especially, parents seem to have a greater difficulty to identify children’s deficit areas, but their accuracy improves with the increasing age of the children and in parallel with the therapeutic process [21,22]. This data can be read both as a function of increased clinical expression connected with the increasing age with evidence of a greater lack in social and cognitive behaviors, both as a result of a greater awareness associated with the parental counseling provided with the therapeutic project.

Our findings show that parents and professionals have a different representation of the child. This could well depend on the specific preparation of the professionals which makes them more accurate and objective. On the other hand it is possible for parents to recognize more easily the deficit areas that determine dysfunctional behaviors in their children, or the emotional control and the inhibitory control (EMI). We agree with other authors who emphasize both the difficulty of the EF evaluation in a such complex and heterogeneous area as autism spectrum disorders, both the multi-source research perspective and approach that would enable a more accurate and comprehensive assessment of the EF in autism, differentiating the clinical characteristics and considering how the various developmental components interact to determine cognitive and social development.

As already mentioned in the introduction, despite between the pathogenetic hypothesis of ASDs has been inserted that of the executive functions impairment, the latter, when present, does not correlate with the severity of autistic symptomatology and, in any case, it does not appear a specific cause. Then, despite the executive functions are crucial in the socio-relational functioning, when this is primarily affected, as in the ASDs, the degree of socio-relational deficit is not directly dependent on the level of executive functioning impairment. The different levels of functioning and socio-cognitive adaptation in autism are in fact related to the ability to integrate the skills possessed and use them to achieve a purpose; thus, even if are found good skills in specific domains, it is possible that the overall functioning appears disharmonic and so disadaptive.

The inhibit, the flexibility, the emotional and attentional control represent capabilities that allow the child to express his cognitive potential which in autism spectrum disorders could be blocked and inhibited by the symptomatic severity as well as by restricted and repetitive behaviors. Other studies highlighted deficits in such areas of the EF, also considering children with phrasal structure and fluent language, all aspects that indicate a different level of severity than the nonverbal sample evaluated in this research and also found no correlation between executive control and restricted and repetitive behaviors. This finding diverges from our results and can be explained by the use of the first version of the ADOS which considers such behaviors but does not codify them in the overall score. The ADOS-2 diagnostic criteria used in our research allow a more accurate assessment of these behaviors and a more reliable classification of the symptomatic severity in the global score.

Few studies have dealt with the executive functions impairment in preschool ASD children until now, although with a different research design, reaches the same conclusion, or that there is a degree of executive impairment in ASD individuals (in our study when the BRIEF-P is compiled by professionals) but that there is not a significant correlation between the degree of autistic symptomatology and executive impairment. To our knowledge, no analysis have been conducted about the level of agreement between parents and therapists, while there are several between parents and teachers. Moreover, while many researches, having a control sample, base their results on the comparison between ASD and typically developing children or with other milder developmental disorders and find a greater degree of impairment in autistic subjects, our objective was to analyze the correlation between altered executive functioning and other dysfunctional levels in a group with the same clinical diagnosis. It seems quite significant, in accordance with Granader and Kimhi, that the more compromised domains in ASDs are Shift/Flexibility and Emergent Metacognition. The fact that in the Italian literature there is no research about the BRIEF-P applied to autism, is for us an important starting point for further studies.

The recent publication in Italy of the BAFE for the EF assessment can be useful to ensure a more objective and valid assessment context to get an accurate profile of the executive functioning of the child and assess the developmental trajectory of individual competencies and their integration in pre-school age. Directly assessing executive functioning through performance tests will allow to integrate the evaluation results with the informations derived from indirect sources such as BRIEF-P compiled by parents and/or professionals. In our opinion, the BRIEF-P questionnaire usefulness lies in the possibility to obtain information in everyday contexts and actually learn about the representation that parents have of the skills expressed by their autistic children. Despite the reliability is not always adequate for the definition of a development profile, it is therefore important in the therapeutic process as a measure of their awareness of the executive dysfunctioning of the child and for the evaluation of child’s potential, having as a comparison parameter the one compiled by the professionals. The clinical implication of these findings suggests the importance to support parents of children with ASD in a path that will lead them to greater awareness of the potential and the fragility of their children, in order to create greater compliance in the therapeutic project.

In our experience the BRIEF-P compiled by the professionals is a useful tool in defining the individual development profile in preschool, but it is not indicative of the severity of autistic symptoms. In the ASDs could therefore be used to define the "specifier" of the executive functioning, in line with the suggestion made by the DSM-5. From these considerations the assessment of the EF cannot be left solely to a questionnaire like the BRIEF-P, although compiled by professionals, but must be supported by a clinical diagnosis made by autism trained and experienced professionals in a multidisciplinary team.