Assessing the Possible Impact of Child and Family Teams in Arizona Using Propensity Scores

Arizona Department of Health Services, Division of Public Health and Preparedness, Bureau of Public Health Statistics, USA

*Corresponding Author:

Khaleel S. Hussaini
Arizona Department of Health Services
Division of Public Health and Preparedness
Bureau of Public Health Statistics
150 N. 18th Avenue, Suite 550, Phoenix, AZ 85007-3242, USA
Tel: 602-542-4452
Fax: 602-364-0082
E-mail: khaleel.hussaini@azdhs.gov

Received Date: August 07, 2014; Accepted Date: September 15, 2014; Published Date: September 18, 2014

Citation: Hussaini KS (2014) Assessing the Possible Impact of Child and Family Teams in Arizona Using Propensity Scores. J Child Adolesc Behav 2:160. doi:10.4172/2375-4494.1000160

Copyright: ©Hussaini KS. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use,distribution, and reproduction in any medium, provided the original author and source are credited.

Visit for more related articles at Journal of Child and Adolescent Behavior

Abstract

The study assessed the impact of Child and Family Teams (CFT) on functional outcomes, for children enrolled in Arizona’s public behavioral health system. The current study extends the existing evidence on wraparound approach to a public behavioral health setting where randomization may not be feasible and/or ethical. The study was a quasiexperimental non-equivalent post-test only design comprising of 3,950 children with an AXIS V diagnoses of ‘severe impairment’ in Arizona who were eligible and received Medicaid funded services. Multivariable propensity weights were used to estimate the odds of successful functioning among children who voluntarily participated in a CFT compared to those who did not. Children who participated in CFT had better odds of avoiding delinquency (95% CI, 1.06-1.3, p<0.01) and succeeding in school (95% CI, 1.29-1.55, p<0.01) when compared to the children that did not participate in CFT even after adjusting for other covariates.

Keywords

Children; Adolescents; Mental health; Framework

Introduction

Nearly 21 percent of children and adolescents aged 9-17 are diagnosed with a mental health or addictive disorder that causes at least minimal impairment in daily functioning in the U.S. [1]. According to the Center for Mental Health Services, the type of mental health disorders experienced by youth varies considerably, and that approximately half of the disorders relate to disruptive behaviors [1]. In treating mental illness, there is an over-reliance on hospitalization, residential treatment, and other restrictive placements and there are very few community-based treatments [2]. Available services are often perceived as hostile to parents and other caregivers, discouraging their participation in treatment planning and follow through [3,4]. Further, even when they ‘successfully’ complete treatment and return to their communities, most therapeutic gains are not maintained. Overall, there is weak evidence for the effectiveness of services delivered in institutional settings [1]. One promising practice to address this issue has been the development of wraparound or individualized services [5]. Bruns, Rast, Peterson, Walker, and Bosworth note that wraparound is a collaborative planning process where family members, service providers, and other members of the family’s support systems develop a comprehensive individualized plan of care guided by a wraparound facilitator [5,6]. Approaches to wraparound differ widely in their scope, implementation, processes, structures, and underlying theories [7]; however, common to various approaches is the emphasis for community-based integrated and coordinated care [8]. Suter and Bruns in their meta-analytic study of wraparound services note that the evidence surrounding wraparound services is mixed. Their conclusion is that wraparound’s acceptance and/or dismissal are not yet evidence-based [9]. One of the components of the wraparound services is Child and Family Teams (CFTs).

CFTs are an important component of wraparound practice and are comprised of people important to an individual child’s development. Child and family teams in the U.S. appeared in the early 1990’s, and since then have emerged as a strategy to meet the safety and mental health needs of children and families involved in child serving systems [10,11]. They also have led to some systemic change efforts in the US [12,13]. CFTs are typically family-centered promoting family’s rights to define their own needs, strengths and capacities [14,15]. CFTs are comprised of people important to an individual child’s development by promoting family group decision-making that is culturally relevant [15]. CFTs utilize a team approach with a view that the involvement of multiple systems is critical in planning the development of academic and behavioral plans [15]. In the U.S., the evidence on the impact of family meetings on outcomes such as decreased additional contact with CPS, decreased future maltreatment, better placement stability, and increased placement with relatives is also mixed [16].

There is currently little-to-no literature on how CFTs influence children’s functional outcomes in public behavioral health settings. The present evaluation study focuses on the efficacy of CFTs in Arizona by examining differences in children’s functional outcomes of those who voluntarily participated in a CFT to those who did not. All children who are encountered in the public behavioral health system and who are Title XIX and XXI eligible, receive appropriate behavioral health services, which include medications, access to and care from psychiatrists/clinicians, and counseling services based on their clinical diagnoses and need. Apart from these standardized behavioral health services, children also have a choice of participating in a CFT voluntarily; and thus, children who do not participate in a CFT provide an internal comparison group. This study assessed the impact of CFT on two outcomes: avoiding delinquency and success in school. It adds to limited evidence available in public behavioral health settings on CFT using statewide data.

Methods

Evaluation study design

In general, randomized control trials (RCTs) outweigh quasi-experimental evaluation studies because random allocation minimizes bias in estimating ‘treatment effects.’ However, organizational realities often restrict conducting randomized studies [17] and the ‘gold standard’ of RCTs may not always be feasible in evaluation studies, especially in public health settings due to a variety of reasons [18]. While there are several methods available to reduce the group differences, one method is the use of propensity score weighting. Propensity score analysis utilizes a counterfactual framework and a counterfactual is defined as a potential outcome that happens in the absence of cause [19]. Thus, for the participants in the treatment, a counterfactual is the potential outcome under the control condition and vice-versa [20]. Because participation was voluntary, a counterfactual in this study would imply what the outcome of a ‘non-participating’ child in a CFT would look like had they participated. Propensity score weights were utilized to estimate ‘treatment effects’ of participation in CFT on functional outcomes.

Rosenbaum and Rubin define a propensity score as the conditional probability of assignment to a particular treatment (intervention) given a vector of observed covariates [21]. While there are several methods within propensity score analysis, propensity score weighting is one technique that does not resample the data and reduces the potential loss of participants by using weights in a weighted regression of the outcome on treatment and covariates [20,22-24]. This study utilized propensity score weighting technique. First, propensity scores were estimated using logistic regression using available covariates at the time of enrollment, and these variables were: age, gender, race and ethnicity, educational attainment, household income, and household size. These variables were used to predict the odds of participating in a CFT (i.e. ‘receiving treatment’ and/or ‘intervention’). In the equation form,

Ln CFT = ß_o + ß₁X_age + ß₂X_female + ß₃X_Hispanic+ ß₄X_{educational attainment} + ß₅X_{household income} + ß₆X_{household size} ----> (1)

The predicted probabilities are essentially the propensity scores (PS) and propensity weights were calculated as the inverse of propensity scores as indicated in equation (2).

w = 1/PS for those participating in CFT

w = 1/(1-PS) for those whose not participating in CFT ---> (2)

Evaluation Data and Sample

The study utilized the behavioral health administrative data available for children 9 to 17 years of age. At the time of enrollment and/or at intake an individual's overall level of functioning and his/her ability to carry out the activities of daily living is assessed by a licensed clinician using Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision (DSM-IV-TR) criteria. This overall functioning also known as the Global Assessment of Functioning Scale (GAF) is recorded as an AXIS V diagnosis using a 100-point scale. This measures an individual’s overall level of psychological, social, and occupational functioning on a hypothetical continuum of mental health-illness. Researchers usually group GAF scores into three categories according to their clinical meaningfulness: 1 to 40 as pervasive impairment; 41 to 60 as severe impairment; 61 to 100 as mild to minimal impairment [25-27]. While the current DSM approach is limited by the fact that it is purely descriptive, DSM diagnoses are reliable across trained raters [27]. The analytic sample for this study comprised of 3,950 children enrolled during May 2006 and May 2007 of Hispanic and non-Hispanic White race and ethnic background. The AXIS V diagnoses of severe impairment (i.e., a GAF score of 41 to 60) was chosen due to its clinical relevance for treatment planning. The average GAF score for the sample was M = 55.77 (SD = 4.33).

Measures

Our outcome variable of interest related to functional outcomes. Functioning is a critical outcome indicator in treating children and adolescents [28] and in determining outcomes, interactive and collaborative approach is critical; although “clinicians and clients may measure improvement differently” [29,30]. Functional outcomes analyzed in this study utilize a clinician’s observations and reports, and the outcome measures utilized in evaluating children were: (a) avoiding delinquency; and (b) success in school. These functional outcomes carry specific guidelines and have specific ‘descriptors’ for clinicians and are audited through chart reviews by an independent ADHS clinical team. For instance, descriptors for avoiding delinquency are: (i) decreased contact with law enforcement and/or no contact with law enforcement; (ii) compliance of the child with the terms of his/her probation; (iii) decrease in illegal behavior; and (iv) decreased and/or no use of illegal substances. Similarly, descriptors for success in school are: (i) increased attendance; (ii) maintaining or improved grades; (iii) completing homework; (iv) obeying school rules; (v) improved peer relations; (vi) participating in extra-curricular activities. These ‘descriptors’ are not collected in administrative data for conducting any reliability analysis and/or creation of a continuous measure. Using these descriptors as a guide, a licensed clinician records a functional outcome such as avoiding delinquency and school success as a dichotomy. ‘Achieved the outcome’ is coded as ‘1’ or ‘did not achieve the outcome’ is coded ‘0.’ The hypothesized independent variable was a dummy variable that captured whether or not the child participated in a CFT (0=Non-participant; 1=Participant). Out of the 3,950 children, approximately 38 percent (n=1,491) participated (intervention/treatment group) in a CFT and approximately 62 percent (n = 2,459) did not participate (comparison/control group).

Apart from the hypothesized independent variable, other confounders such as age, gender, race and ethnicity, household size, household income, and education available at enrollment and assessment process was also included. Age was computed using client's date of birth at the time of intake and the average age of the client was 13.26 years (SD=2.33 years). The gender variable was a dummy variable coded ‘1’ for females and ‘0’ for males. Approximately 60 percent (n=2,368) of the sample were males and 40 percent (n=1,582) were females living with their family. A dummy variable (1=Hispanic; 0 = non-Hispanic White) of race and ethnicity variable was created from the standard race ethnic categories. Average household size was a continuous measure with M = 4.08 (SD = 2.00), which indicated that on an average client’s enrolled in the behavioral health system had four family members. Median gross monthly household income in dollars was a continuous measure with Mdn=$416 (IQR=$1,500). Average years of education was also a continuous measure with M=3.61 (SD=1.02), which indicated that on average a client had at least four to six years of grade school.

Analytic procedures

Bivariate and multivariable statistical procedures were conducted using SAS v9.2 software 2008. Chi-square tests were used for nominal variables and independent t-tests with correction for unequal variances and/or Wilcoxon-Mann-Whitney test for medians were used for continuous variables to test for differences in the two groups on covariates (see table I). Separate logistic regression models utilizing propensity weights were estimated for each of the functional outcomes using CFT group as a predictor adjusting for age, gender, educational attainment, household income, household size, and race and ethnicity variables. An interaction term was created to assess if the outcomes varied for those participating in CFT. Each interaction term was separately evaluated with the final logistic regression model for age, gender, race and ethnicity, household size, and household income.

Results

Table 1 compares key baseline demographics and covariates of interest by their participation status in the CFT prior to estimation of propensity scores and weights. It is evident from the table that those who participated in the CFT (intervention/treatment group) and those who did not participate in the CFT (comparison/control) group were similar with respect to gender composition, race and ethnicity, age composition, and educational levels. However, children who participated in the CFT (Mdn=$650; IQR=$1,500) differed from those who did not participate in the CFT (Mdn=$333; IQR=$1,493) with respect to monthly gross household income. Children who participated in the CFT (Mdn=4; IQR=2) also differed from those children who did not participate in the CFT (Mdn=4; IQR=2) with respect to average household size (p<0.01). While the difference in average household size was statistically significant it is perhaps due to large sample size and is perhaps spurious.

Table 1: Demographic characteristics of clients 9 to 17 year old with an AXIS V rating of ‘severe impairment’ by participating (treatment) and non-participating (control) groups prior to estimating propensity scores

Because the groups are non-equivalent and there could be potential selection bias, propensity weights were estimated using all the available covariates discussed earlier. Table 2 presents weighted point estimates (odd ratios) of logistic regressions for the two outcomes with confidence intervals in parentheses. In interpreting odds ratio it is important to note that odds ratio greater than 1 indicates a positive association, less than one indicates a negative association, and an odds ratio equal to 1 indicates no relationship. The weighted estimates had lower standard errors and narrower confidence intervals compared to unweighted point estimates (not presented here). Children participating in a CFT had better odds of avoiding delinquency (OR=1.17; p<0.01) and succeeding in school (OR=1.41; p<0.01) when compared to those children who did not participate in a CFT even after adjusting for other covariates. Hypothesis regarding efficacy of participating and non-participating children in CFT was confirmed. No significant interaction effects were found in regards to participation in CFT and age, gender, race and ethnicity, household size, and household income.

Table 2: Propensity weighted logistic regression point estimates and
confidence intervals (CIs) assessing effects of CFTs

Discussion

A key outcome of this evaluation study is that children participating in a CFT had better functional outcomes compared to children who did not participate; however, there was no evidence to suggest that outcomes differed by age, gender, race and ethnicity, household size and household income and participation in CFT. This evaluation study adds to the existing literature on efficacy of wraparound in several ways. First, the finding is limited to the population of 9 to 17 year olds with severe impairment living with family rather than all children encountered in the public behavioral health system. Second, it specifically examines the impact of children with severe impairment, i.e., individuals with a GAF score range of 41 to 60 with a relatively large sample in a public behavioral health setting, while other quasi-experimental evaluation studies on wraparound had limited sample size. Third, this study utilized propensity weighted logistic regressions to minimize any selection bias. While there was no evidence of differential impact in outcomes by age, gender, race and ethnicity, household size, and household income, Walton in her doctoral work examined the outcomes among seriously emotionally disturbed youth in Indiana, and found that while youth in systems of care had better outcomes, Hispanic youth did not [31]. Other than Walton’s study till date there are no specific outcome studies examining differences in Hispanic and non-Hispanic youth in wraparound research.

While there is some evidence to suggest that children participating in CFTs have better functional outcomes (i.e., avoiding delinquency and achieving success in school) compared to those who are not, the study also has limitations. The study is limited in regard to its design (non-equivalent posttest only), lack of randomization, mono-operation, mono-method bias, and lack of availability of fidelity measures. Ideally, multiple versions of CFT could reduce the threat of mono-operation bias. However, costs in public health setting preclude from implementing multiple versions of a program. Mono-method bias relating to the outcome measures avoiding delinquency and success in school was less plausible as they were several ‘descriptors’ utilized to determine functional outcome. Because these descriptors are not accessible in administrative data for conducting any reliability analysis and/or creation of a continuous measure, the veracity of the ‘clinical judgment’ on the two functional outcomes avoiding delinquency and achieving success in school needs to be taken at face value, and is still subject to bias. Lack of availability of fidelity data on the quality of the meetings, composition of the teams, cultural competence of the teams, the number of meetings, and number of collaborating partners that may influence the outcomes were not available at the time of the study; and thus, limit the scope of the findings. Finally, it is important to note that while propensity score analysis minimizes the threat to selection bias, propensity scores cannot account for hidden biases such as factors that were not measured and/or collected at the time of enrollment and assessment of a child entering the behavioral health system. Given its methodological limitations, and the limited number of outcome studies specific to Child and Family Teams, the current study, however, provides only suggestive rather than conclusive evidence. In conclusion, this study presents one approach to assess the impact of CFT and functional outcomes among children with severe impairment in a public behavioral health setting where randomization was not feasible with limited administrative data. The findings of this study provide a basis and most importantly accentuate the need for further exploration of efficacy of child and family teams using more rigorous designs (i.e., RCTs) and better data measures when feasible.

References

1. Perou R, Bitsko RH, Blumberg SJ, Pastor P, Ghandour RM, et al. (2013) Mental health surveillance among children--United States, 2005-2011.  MMWR SurveillSumm 62 Suppl 2: 1-35.
2. (2003)New Freedom Commission on Mental Health Achieving the promise: Transforming mental health care in America: Final report Rockville.
3. Koroloff NM, Friesen, BJ (1997) Challenges in conducting family-centered mental health services research. Journal of Emotional Behavioral Disorders 5:130-137.
4. Kruzich JM, Jivanjee P, Robinson A, Friesen BJ (2003) Family caregivers' perceptions of barriers to and supports of participation in their children's out-of-home treatment.  PsychiatrServ 54: 1513-1518.
5. VanDenBerg JE (1993) Integration of individualized mental health services into the system of care for children and adolescents. Administration and Policy in Mental Health 20:247-257.
6. Bruns EJ, Rast J, Peterson C, Walker J, Bosworth J (2006) Spreadsheets, Service Providers, and the Statehouse: Using Data and the Wraparound Process to Reform Systems for Children and Families. Am J Community Psychol 38:202-212.
7. Clark HB, Clarke RT (1996) Research on the wraparound process and individualized services for children with multisystem needs. Journal of Child and Family Studies 5:1-5.
8. Mears SL, Yaffe J, Harris NJ (2009) Evaluation of Wraparound Services for Severely Emotionally Disturbed Youths. Research on Social Work Practice 19:678-685.
9. Suter JC, Bruns EJ (2009) Effectiveness of the wraparound process for children with emotional and behavioral disorders: a meta-analysis.  Clin Child FamPsychol Rev 12: 336-351.
10. Burchard JD, Burchard SN (2000) The wraparound process with children and families. In: Family Group Conferencing: New Directions in Community-Centered Child and Family Practice (G Burford, J Hudsonedn.,) Aldine de Gruyter, Hawthorne, NY.
11. Merkel-Holguin L (2000) Diversions and departures in the implementation of family group conferences in the United States. In: Family Group Conferencing: New Directions in Community-Centered Child and Family Practice (G Burford, J Hudsonedn.,) Aldine de Gruyter, Hawthorne, NY.
12. Doolan M, Phillips P (2000) Conferencing in New Zealand: child protection. In: Family Group Conferences: New Directions in Community-Centered Child and Family Practice (G Burford, J Hudsonedn.,) Aldine De Gruyter, New York, NY pp: 93–197.
13. Pennell J, Anderson G (2005) Widening the Circle: The Practice and Evaluation of Family Group Conferencing with Children, Youths, and their Families. Washington, DC, NASW Press.
14. Dunst CJ,Trivette CM (1994) Aims and principles of family support programs. In: Supporting and Strengthening Families: Methods, Strategies and Practices (CJDunst, CM Trivette, AG Dealedn.,) Cambridge, MA, Brookline Books, pp: 30–48.
15. Taliaferro JD, DeCuir-Gunby J, Allen-Eckerd K (2009) ‘I can see parents being reluctant’: perceptions of parental involvement using Child and Family Teams in schools. Child and Family Social Work 14:278-288.
16. Snyder EH, Lawrence CN, Dodge KA (2012) The impact of system of care support in adherence to wraparound principles in Child and Family Teams in child welfare in North Carolina.  Child Youth Serv Rev 34: 639-647.
17. Wohl MJ, Sztainert T (2011) Where did all the pathological gamblers go? Gambling symptomatology and stage of change predict attrition in longitudinal research.  J Gambl Stud 27: 155-169.
18. Oakley A, Strange V, Toroyan T, Wiggins M, Roberts I et al. (2003) Using Random Allocation to Evaluate Social Interventions: Three Recent U.K. Examples. Annals of the American Academy of Political and Social Science, 589:170-189.
19. Shadish WR, Cook TD, Campbell DT (2002) Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.
20. Guo S, Fraser MW (2010) Propensity score analysis: statistical methods and applications. Los Angeles, USA: Sage.
21. Rosenbaum P R, Rubin DB (1983) The central role of the propensity score in observational studies for causal effects. Biometrika, 70:41-55.
22. Hirano K, Imbens GW (2001) Estimation of causal effects using propensity score weighting: An application to data on right heart catheterization. Health Services and Outcome Research Methodology, 2:259-278.
23. Hirano K, Imbens GW, Ridder G (2003) Efficient estimation of average treatment effects using the estimated propensity score. Econometrica, 71:1161-1189.
24. McCaffrey DF, Ridgeway G, Morral AR (2004) Propensity score estimation with boosted regression for evaluating causal effects in observational studies.  Psychol Methods 9: 403-425.
25. Lindsey MA, Joe S, Muroff J, Ford BE (2010) Social and clinical factors associated with psychiatric emergency service use and civil commitment among African-American youth. Gen Hosp Psychiatry, 32:300-309.
26. Moos RH, Nichol AC, Moos BS (2002) Global Assessment of Functioning ratings and the allocation and outcomes of mental health services.  PsychiatrServ 53: 730-737.
27. Moos RH, McCoy L, Moos BS (2000) Global assessment of functioning (GAF) ratings: determinants and role as predictors of one-year treatment outcomes.  J ClinPsychol 56: 449-461.
28. Parens E, Johnston J (2008) Understanding the agreements and controversies surrounding childhood psychopharmacology.  Child Adolesc Psychiatry Ment Health 2: 5.
29. Hodges K, Wotring J (2000) Client typology based on functioning across domains using the CAFAS: implications for service planning.  J Behav Health Serv Res 27: 257-270.
30. O'Malia L, McFarland BH, Barker S, Barron NM (2002) A level-of-functioning self-report measure for consumers with severe mental illness.  PsychiatrServ 53: 326-331.
31. Walton B (2006) Predictors of improvement for children served in developing systems of care (Doctoral dissertation) Available from Dissertation and Theses database.