Implementing Evidence-Based Suicide Prevention Training in Communities: Implications for Quality Improvement

¹RAND Corporation, 1776 Main Street, P.O. Box 2138, Santa Monica, CA 90407-2138, USA

²RAND Corporation, 1200 South Hayes Street, Arlington, VA 22202-5050, USA

*Corresponding Author:

Karen Chan Osilla
RAND Corporation, 1776 Main St., PO Box 2138
Santa Monica, CA 90407-2138, USA
Tel: 310-393-0411 extn. 6074
Email: karenc@rand.org

Received date: September 03, 2015 Accepted date: October 21, 2015 Published date: October 25, 2015

Citation: Osilla KC, Ramchand R, Seelam R, Barnes-Proby D, Gilbert ML (2015) Implementing Evidence-based Suicide Prevention Training in Communities: Implications for Quality Improvement. J Community Med Health Educ 5:371. doi:10.4172/2161-0711.1000371

Copyright: © 2015 Osilla KC, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution and reproduction in any medium, provided the original author and source are credited.

Visit for more related articles at Journal of Community Medicine & Health Education

Abstract

Suicide prevention trainings are implemented to equip the public’s ability to intervene with those who are at-risk, but their implementation is not often monitored for quality. In this study, we propose a quality improvement model to improve trainer skill, demonstrate evidence of knowledge uptake, and document the quality of training workshop implementation. We collected participant data (N=2006) from over 127 Applied Suicide Intervention Skills Training (ASIST) training workshops that evaluated workshop satisfaction, confidence to intervene, and likelihood to intervene and refer immediately post-training. We also collected trainer data by measuring fidelity and adherence to the ASIST protocol at five live ASIST workshops. Training participants reported improved confidence and likelihood to intervene and refer after the workshop. Participants also reported high satisfaction. In three of the five workshops, newly trained trainers covered 75% or more of the fidelity items demonstrating thorough review of the training. Trainers generally adhered to one of four competencies specific to ASIST and five of the 11 general competencies relating to group management. Trainers may need to improve their efforts to tailor content to specific audiences, promote cultural competence, and manage time.

Keywords:

Suicide prevention; Health education; Fidelity monitoring; California

Introduction

Suicide is the tenth leading cause of death in the United States and in their efforts to prevent this tragic loss of life, communities have increasingly emphasized the importance of disseminating suicide prevention trainings to broaden the public’s knowledge of suicide, its risk factors, and intervention strategies. Unfortunately, the existence of evidence-based trainings is scant [1]. One notable study exception is the Applied Suicide Intervention Skills Training (ASIST) [2]. Evaluated behavior changes in telephone crisis responders who participated in ASIST workshops, and outcomes of their crisis line calls. Compared to responders not trained in ASIST, those who were trained exhibited specific behaviors (e.g., ability to explore callers’ reasons for living, ambivalence about dying, and informal support contacts; increased length of calls) that were positively associated with improved caller outcomes (e.g., the appearance that callers were less depressed, suicidal, and overwhelmed [3]. Though not as rigorous, other research studies that have evaluated ASIST support these findings, indicating that participants in ASIST workshops had self-reported increased knowledge and skills in pre-post surveys [4] however, we acknowledge one study that found no changes in similar outcomes [5].

ASIST workshops are designed to span two days and teach participants “suicide first aid” so they can recognize those at-risk of suicide and intervene accordingly. Participants are taught how to recognize suicide risk factors, to respond in a manner that increases immediate safety, and to link suicidal persons to appropriate resources [6]. Workshops are typically led by at least two trainers who take turns presenting the ASIST curriculum. Throughout both days, the trainers present activities geared to the whole group and in smaller breakout groups. Both training days explore participant’s attitudes about suicide, reactions to video scenarios, and discuss the Suicide Intervention Model (SIM) framework and its three phases – connecting, understanding and assisting. Connecting is the process of establishing rapport and engaging the person at risk to continue the interaction; the understanding phase is active listening to reasons for living and dying through the perspective of the person at risk; and assisting is providing an opportunity for the person at risk to consider resources and a plan to stay safe.

Although shown to be effective in a trial, in order to ensure that such effects are replicated in other communities it is important that trainings be administered with high quality [7]. Specifically, it is important to monitor the fidelity of trainings to understand whether trainers were delivering workshops with high fidelity (i.e., covering all aspects of the manual-based training) and adherence (i.e., whether the training followed the recommended style of presenting). In addition to monitoring hotline responder behaviors and caller outcomes, the findings of [6] were complemented by a study [8] that assessed ASIST fidelity during workshops with crisis hotline responders. Video and audio segments of 66 ASIST workshops were coded. Results suggest that about 35% of trainers delivered ASIST with high fidelity as measured by more than 75% of the workshop content delivered, but less than 18% of trainers were adherent and able to demonstrate solid competence on the competency scales (e.g., presentation style/delivery [7]. These results suggest that trainers might easily present workshop content, but may have a more challenging time delivering the content within the recommended facilitation style. This is an important finding because the style of information delivery is just as important as training content and may have a direct effect on whether participants learn the information [8]. By documenting both fidelity and adherence, research can provide evidence of knowledge uptake and that training was implemented as intended and with high quality (e.g., best practice). Collecting data on adherence and fidelity help ensure that past results evaluating ASIST may be translatable to the new trainings in which ASIST is being delivered, providing a way of accountability for monitoring how existing funds are being spent.

In addition to fidelity and adherence, it is also important to monitor whether trainings are associated with improved outcomes for training participants and the clients they serve. With respect to trainings like ASIST, a new conceptual model states that such courses can change participants’ knowledge about suicide, beliefs and attitudes about whether suicide can be prevented, reluctance to intervene, and self-efficacy to intervene; changes in these domains can then change intervention behaviors [9].

The current study extends past research by evaluating both the participant (N=2006) satisfaction and outcomes across 127 ASIST workshops and the quality (fidelity and adherence) of five ASIST workshops in California. The current study has three goals: (1) to examine participant outcomes for ASIST workshops delivered in California, (2) to provide a preliminary assessment of whether trainers were delivering workshops with high fidelity and adherence, and (3) to provide a quality improvement model for ongoing ASIST evaluation that could be applied to other communities.

The delivery of ASIST trainings in California occurred as a result of voter-approved Proposition 63, the Mental Health Services Act (MHSA). Proposition 63 is funded by levying a 1% tax on personal income above $1 million. A portion of Proposition 63 funds were dedicated to prevention and early intervention (PEI) services, including for suicide prevention. As part of this effort, LivingWorks, was awarded a contract to provide Train-the-Trainer (T4T) trainings in ASIST with the intent that these trainers would disseminate ASIST workshops in their community. Sixteen T4T trainings were conducted to train Californians to become registered ASIST trainers. To date, 327 people have been trained at these T4Ts, and they have subsequently trained 4,887 people at 267 ASIST workshops in California [10].

Methods

Participant post-training surveys

Participants: A total of 2006 workshop trainees were surveyed across 127 different workshops in California (Table 1). Details about how information was collected from trainees are provided below; overall, 73% of participants reported being female, 46% reported White race/ethnicity, and 30% reported Latino race/ethnicity. About half (53%) of participants were health professionals.

Table 1: Training participant demographics (N=2006).

Procedure and measures: At the end of the two-day ASIST workshops, participants were asked to complete an anonymous short survey. Surveys were then collected by the trainers and sent to LivingWorks to be data entered. Post-training surveys captured both participant satisfaction and outcomes, respectively. Satisfaction was captured in seven domains (1 item per domain): Quality (How would you rate the quality of the ASIST workshop?); Recommendation (Would you recommend ASIST to others?); Personal Practicality (This workshop has practical use in my personal life.); Professional Practicality (This workshop has practical use in my work life.); Helpfulness (This workshop was helpful.); Cultural Tailoring (This workshop meets the unique needs of the people I work with, e.g., diverse ethnic/language groups, LGBTQ, low income); and Importance (It is important for people to attend workshops like this one to support others in the prevention and/or intervention of suicide. ). All responses ranged from 1 (Definitely no) to 10 (Definitely yes), though for Quality, scale anchors were 1 (Did not like at all) to 10 (Liked a lot).

Participant outcomes were captured in three domains: Confidence to intervene (3 items; I feel prepared to help a person at risk of suicide); Confidence to refer (2 items; I can identify the places or people where I should refer others at risk of suicide); and Likelihood to intervene (2 items; e.g. If a person’s words and/or behaviors suggest the possibility of suicide, I would ask directly if he/she is thinking about suicide). Participants were asked on the survey to think about their confidence and likelihood before the workshop and then after the workshop retrospectively in one sitting. We took the mean of the items within each domain.

Fidelity and adherence monitoring

Fidelity coder training: Two coders with a background in social work and no experience in ASIST received 10 hours of training which included attending a training on the fidelity monitoring observational protocol, discussing questions about the rating scales, and reaching consensus on the scoring criteria. The coders also attended a two-day ASIST workshop and practiced coding using various examples of the ASIST workshops to increase inter-rater reliability. Coders met regularly to discuss their ratings and review their codes with a third RAND research staff member trained in ASIST. Thereafter, coders independently coded the trainings for the current study.

Measures

Our study conducted live observation of full ASIST workshops versus a portion, and we therefore had to expand existing ASIST fidelity protocols [7]. We reviewed the ASIST trainer workbook and itemized each program section of the workbook. We worked closely with Living Works Education to ensure these fidelity items were comprehensive and appropriate. We then adapted adherence items that measured trainer competencies (e.g., the style in which the content is delivered). These items were adapted from the ASIST trainer competencies in the trainer workbook and additional general trainer competencies from other research [6,8,9]. The ASIST fidelity and adherence monitoring protocol will be available for free download at http://www.rand.org/health/surveys_tools.html.

Fidelity

The fidelity checklist consisted of a total of 59 items with 40 items corresponding to the manual-based training activities on day one and 19 items with the activities on day two of the workshop. Of the 59 items, 5 items were logistical items (e.g., introduce self), 40 items referred to specific training content, 8 referred to role-plays trainers are supposed to conduct, and 6 referred to open-ended discussions to be held throughout the training. For each item, coders checked off “yes”, “only in part”, or “no” based on whether the activity was presented. To measure fidelity, observers counted the number of sections the trainer covered in the ASIST training.

Adherence

The adherence checklist consisted of 16 items to measure trainer presentation style corresponding to ASIST trainer competencies and other general facilitator proficiencies. These skills included whether the trainer was collaborative, engaging, organized, and able to manage the group, and an additional four items assessed whether the trainer demonstrated cultural sensitivity (see items 13-16 in Table) [2,8-13]. Items also queried whether the trainer conveyed empathy and the overall participation level of the group. Coders rated trainers based on the ASIST guidelines trainers learn in the ASIST T4T. These four ASIST trainer competencies included whether trainers talked about suicide directly, provided positive feedback to participants, provided no negative feedback to participants, and presented the Suicide Intervention Model (SIM) framework. Ratings were on a scale of 0-3 (e.g., ‘not at all adherence’ to ‘fully adherent’) on 14 items, 0-2 on one item, and 0-4 on one item.

Workshop selection

Five ASIST training workshops were selected for fidelity monitoring. We first used a convenience sample of individuals who had received Proposition 63-funded ASIST T4T training, but who were not yet registered trainers. We asked trainers in different regions of California who served diverse populations if RAND could observe their trainings. Of the trainers asked, all agreed to be observed. Trainers who were observed were both male (n=3) and female (n=5), and had between six months to a year of ASIST training experience. For all sites, there was generally one main trainer who was selected to be observed by the coders based on being recently trained under CalMHSA funds (e.g., we purposely did not want to observe a senior registered trainer not trained under CalMHSA funds). The main trainer was assisted by an additional one to three trainers. Coders observed the main trainer in breakout groups.

Workshops were held in northern California (n=1), central California (n=1), and southern California (n=3). One of the trainings was conducted in Spanish at a community-based multi-cultural counseling center. Two of the trainings were for military and/or law enforcement organizations. One training was held for a suicide hotline organization, and another was for a mental health service organization. The number of participants per training ranged from seven to 35 people. There were a total of 115 participants (39 males and 76 females) across all five trainings.

Observation procedures

We carefully considered how to conduct live observation of ASIST trainings so that observations would not disrupt the existential nature of the training (e.g., participants feeling comfortable sharing their personal experiences with suicide) and that this model could be adapted later for quality improvement. We collaborated with LivingWorks Education to develop a verbal consent script the trainer read that emphasized the importance of trainer-participant privacy,that coders were only interested in the process of an ASIST workshop, that notes would only be taken on the second day, and notes would only document the delivery of the training. Training participants were introduced to the coders at the beginning of the training and were offered the option to not be observed by participating in a different group. No participants declined observation. Two coders attended four trainings, and one coder fluent in Spanish attended the Spanish language training. To encourage neutrality and minimize bias and disruption of the training process, coders sat in the back of the training room(s) and did not interact with participants either during the training or on breaks. All procedures were approved by RAND’s Institutional Review Board.

Statistical analyses

Post-training surveys: Participant satisfaction data were heavily skewed with most participants reporting 10’s on all outcomes. To account for these ceiling effects, we estimated two-part regression models. First, we reverse coded each outcome such that 10s became 0s and 1s became 9s. We then estimated regression models in which participants’ gender (male/female), profession (health professional/administrator/educator/law enforcer/student/other), race (White/Latino/Black/Asian/Other), and the region in which they work (Southern region, Los Angeles, Bay Area, Central Region, and Superior Region) were used to predict each of the 7 outcomes. These models yield recycled prediction mean values for each covariate group of interest, as well as comparisons between each group category (i.e., Whites relative to all other race groups; Latinos relative to Blacks, Asians, Other; Blacks relative to Asians and Other; Asians relative to Other). Significant differences between groups are indicated by confidence intervals that do not contain the null value (=0).

Participant outcome data were analyzed using linear regressions, controlling for the same predictors as in the two-part models – gender, profession, region, and race – as well as the pre-training scores. Clustering effects due to trainers conducting multiple trainings were also accounted for in the models. By including pre-training scores, the models yield estimates of the change in confidence to intervene, confidence to refer, and likelihood to intervene from prior to the workshop to after the training, relative to the reference category in each group. An alpha of 0.05 was used to determine significant differences between groups.

Fidelity and adherence: Fidelity data were analyzed descriptively examining the percentage of items covered at each of the five workshops to better understand the amount of material adequately covered during the workshops. Adherence data were analyzed by examining the average ratings across trainers on each of the adherence items to assess which areas trainers adhered more or less to.

Results

Participant satisfaction

For each of the seven items, predicted probabilities indicated that participants were extremely satisfied with the training (in most cases, scores were between 0, indicating extreme satisfaction, and 1 on a scale from 0 to 9). However, there were some differences by group, presented in Table 2 and summarized below.

Table 2: Participant satisfaction – predicted means from a two-part regression model.

Gender differences

For two outcomes (culturally tailored and importance), there was evidence that females were more satisfied than males.

Professional differences

In general, students expressed strongest satisfaction with the workshops and educators expressed the lowest levels of satisfaction. Students were more likely to recommend the workshop and found it more helpful relative to health professionals, educators, and law enforcers, and reported greater satisfaction with the workshop’s personal practicality relative to health professionals, administrators, educators, and law enforcers. On the other hand, educators rated lower overall workshop quality (relative to health professionals, law enforcers, administrators, students, and “other” professions), lower satisfaction with professional practicality (relative to health professionals and law enforcers), and found the workshops less helpful (relative to administrators, students, and “other” professions).

Race differences

Across outcomes, two patterns emerged with respect to race differences. First, White participants tended to rate the workshops as less satisfying, and Latinos tended to rate the workshops as more satisfying. There was evidence of Whites being less satisfied with respect to workshop quality (relative to Latinos and Blacks), whether they would recommend the workshop (relative to Latinos), personal practicality (relative to Latinos and Blacks), professional practicality (relative to Latinos), whether they found the workshop helpful (relative to Latinos), the cultural tailoring of the workshop (relative to Latinos), and the importance of the workshop (relative to Latinos). In addition to these differences, Latinos also reported greater satisfaction on overall workshop quality and whether they would recommend the workshop relative to Asians and members of “other” race groups. Second, Asians tended to rate the workshops as less satisfying than members of other races. They reported less satisfaction with overall workshop quality and the cultural tailoring of the workshops relative to all other race groups. They were also less likely to recommend the workshop to others (relative to Latinos and Blacks), and reported lower satisfaction with personal practicality (relative to Blacks), professional practicality (relative to Latinos), whether they found the workshops Helpful (relative to Latinos, Blacks, and members of other race groups), and overall importance (relative to Latinos).

Regional differences

Participants from Northern California were, in general, more satisfied with the trainings than those in Southern California. There was evidence that participants from Los Angeles reported lower satisfaction with respect to workshop quality, whether they would recommend the workshop to others, whether they found it helpful, and the workshops’ importance relative to all three Northern regions. Participants form Los Angeles also reported less satisfaction with cultural tailoring of the workshop, but relative only to participants from the Central region. Participants from the Southern region also reported lower quality and whether they found the workshop helpful relative to all three Northern regions, and whether they would recommend the workshop relative to the Central and Superior regions, The Southern region also reported less satisfaction with both personal and professional practicality, but relative only to the Superior region.

Participant outcomes

Overall, participants reported statistically significant improvements on all three outcomes after training (Table 3); mean ratings of confidence to intervene increased from 3.14 to 4.54, confidence to refer from 3.42 to 4.52, and likelihood to intervene from 3.53 to 4.76. We next examined whether participant’s ratings of training outcomes varied by gender, profession, race, and region. In general, pre-training values of each construct was significantly associated with post-training scores. Participants did not statistically differ in their outcomes by race, and region. However, females reported higher levels of likelihood to intervene, and students reported higher post-training values relative to health professionals in their confidence to refer and intervene, and their likelihood to intervene.

Table 3: Training outcomes – coefficients from linear regression models.

Fidelity monitoring

Inter-rater agreement: Across the four trainings attended by both coders, coders agreed in their scoring on 98% of the fidelity items and 87% of the adherence items.

Fidelity: As shown in Table 4, of the five trainings, 3 of 5 trainings had 75% or more of the fidelity items covered demonstrating thorough review of the training content. The other two trainings had 63 and 68 percent of the fidelity items covered, respectively. Of the 59 fidelity items, there were 9 items that tended to be covered partially or not at all in three or more of the trainings. These included three role-plays, three items related to reviewing training material, one open-ended discussion to occur after two videos of ASIST intervention demonstrations, and two logistical items (i.e., introduction of suicide checkpoints and brainstorm lists of resources).

Table 4: Means (SD) scores on adherence items.

Adherence: On a scale from 0 to 3, trainers ranged in their adherence scores averaging about 1.80 (min 1.33; max 2.75; see Table 4), which for most items is between “occasionally adherent=1” to “most of the time=2” adhered to. Trainers generally scored 2 or higher (“most of the time” to “almost all of the time”) on one ASIST competency item (talking about suicide directly) and five general competencies related to the group management (organized simulations, control of group, participation across both days, participation in role-plays, accepting of diverse cultural differences). Trainers scored the lowest on items related to tailoring intervention content to the target population, being knowledgeable about cultural beliefs around suicide, providing positive feedback, and managing time.

Discussion

The current study evaluated the implementation of the ASIST suicide prevention training by examining training participant satisfaction and outcomes, and trainer fidelity and adherence, respectively. With respect to satisfaction, these results indicate that participants attending ASIST trainings were generally very satisfied with the trainings they attended, indicating that they were helpful and of high quality. Although most participants were highly satisfied, there were some differences. Of note, students reported higher satisfaction than other professionals. Because students may be more likely to come into daily contact with a range of students with various strengths and needs, such a finding suggests the trainings may have a positive impact on the support of students with mental health problems. Interestingly, when outcomes were examined (confidence to intervene, confidence to refer, and likelihood to intervene) students also reported greater post-training values even after controlling for pre-training values. This suggests that ASIST trainings may be particularly well-received by students and promoted specifically to this group. In contrast, educators reported less satisfaction than other occupations, though readers should keep in mind that, overall, all participants were highly satisfied with the training overall. Finally, we note that respondents who self-identified as Latino reported greater participant satisfaction compared with White respondents. We do not know to what extent these differences are the result of cultural differences, variations in the characteristics of trainee participants or hosts, or other factors, but training organizers should be aware that the impact of trainings may vary across different racial/ethnic groups.

With regards to trainer fidelity and adherence, results indicate some variability in trainer fidelity and adherence across trainings. Three of the five trainings covered 75% or more of the items on the fidelity measure. Of these items, some were more significant than others. For example, three role-plays were partially or not covered, which may be an important omission as they are part of demonstrating the ASIST curriculum. This is important because it means that any evidence of the effectiveness of the ASIST training [11] may not be generalizable to trainings that do not cover all of the prescribed program elements. Regarding adherence, trainers tended to score on the lower end of the scale indicating less proficiency in ASIST competencies and general facilitator style. This may not be surprising as trainers were newly trained in ASIST (i.e., a year or less of experience) and may have difficulty managing the large amount of training content in a collaborative style. In fact, previous research has shown that ASIST training participants tend to exhibit higher fidelity ratings and lower adherence ratings [7]. Providing adherence rating feedback to newly trained trainers may facilitate stronger adherence in future trainings, though longitudinal research is needed to evaluate this.

Anecdotally, we know from our previous research that it takes time for clinicians to learn a protocol. Initially, clinicians tend to read the protocol from their manuals and are not as engaging and collaborative with training participants because they are focused on covering the material. However, time to practice delivering more trainings may not be sufficient to increase fidelity and adherence [7], and trainers may need standardized and concrete feedback to hone in on specific skills. The tool developed for this study can facilitate this. For example, in this study, we found that adherence scores were lowest in time management and the frequency for which positive feedback was given. These are two concrete points of feedback that trainers could focus on improving so that their presentation style is more consistent with other ASIST trainers. Trainers could also be provided with examples on how to tailor training material to diverse groups. For example, if the ASIST workshop took place in a Spanish-speaking agency, trainers could ask questions such as “How do these videos match the experiences your clients go through?” or “How would you describe suicide in Spanish?”

Trainers did score high with regards to talking about suicide directly, which may be one of the most important ASIST trainer competencies. Thus, feedback can be prioritized and tailored to what adherence items may be most important to convey and future research could systematically dismantle which of these items may be associated with client outcomes and training participants’ improvements in skills, knowledge, and attitudes.

Limitations

We evaluated only one suicide prevention training without a comparison group. Thus, we do not know if our results are due specifically to the training or due to other factors we do not know about. Our satisfaction and outcome variables were also subjective (e.g., attitudes) and we do not know to what extent they would correlate with more-objective assessments of training outcomes (e.g., intervention skill) or client outcomes. Also, while prior studies report the validity of retrospective surveys [14-18], a “true” baseline survey was not administered prior to the training. Finally, we only trained two coders to monitor fidelity and adherence. Future studies could examine the inter-rater reliability across several coders.

Conclusions

Providing feedback to trainers as a quality improvement tool could be done within the current training infrastructure with few additional resources. For example, LivingWorks Education requires that an individual deliver three trainings within a year they were trained in order to advance as a registered trainer. This offers time to hone in on skill development. Trainers, for example, often deliver ASIST training workshops with a more experienced co-trainer. Thus, feedback could be efficiently delivered during post-training debriefing where their co-trainer provides constructive feedback about fidelity (e.g., the number of items that they completed versus omitted) and adherence (e.g., how their style could be improved upon). Similarly, post-training evaluation forms could be completed by training participants to ensure material was covered. Information can then be relayed back to trainers. By providing this feedback to trainers, trainers have the opportunity to directly work on concrete things such as missing content or enhancements of presentation style.

Proposition 63 has provided the opportunity for community agencies to train on ASIST and other prevention models. However, an important element to determining the success of training efforts is to understand whether trainers are delivering the material consistently and in the manner that it was intended (i.e., best practice). Without this assurance, investments in training are questionable because trainers are not left accountable for delivering intervention content with high fidelity. This heightens the need for train-the-trainer trainings to invest in their participants by including standards that must be met before advancing them to registered trainer status. Efforts like these could be done within existing resources by requiring a process in which participants are followed by a mentor or co-trainer and provided standardized feedback post-training. This investment is needed to ensure high quality training dissemination, which may have lasting effects as more individuals are trained to deliver trainings with high fidelity.

We propose a quality improvement model that recommends that trainers be given standardized feedback on their fidelity and adherence prior to becoming a registered trainer, and have created one available to the public available for free on RAND’s website. This process can be done at low-cost or within existing resources. As noted in our study, the coder does not need extensive experience in suicide prevention in order to correctly decipher whether content and proficiency in ASIST workshops are met. As more initiatives are established, accountability for whether training efforts are being adequately implemented is important.

References

1. Breitenstein SM, Gross D, Garvey CA, Hill C, Fogg L, et al. (2010) Implementation fidelity in community-based interventions.Res Nurs Health 33: 164-173.
2. Burnette C, Ramchand R, Ayer L (2015) Gatekeeper Training for Suicide Prevention: A Theoretical Model and Review of the Empirical Literature. Santa Monica, CA: RAND Corporation.
3. Carroll C, Patterson M, Wood S, Booth A, Rick J, et al. (2007) A conceptual framework for implementation fidelity.Implement Sci 2: 40.
4. Clark W, Welch SN, Berry SH, Collentine AM, Collins R, et al. (2013) California's historic effort to reduce the stigma of mental illness: the Mental Health Services Act.Am J Public Health 103: 786-794.
5. Cross WF, West JC (2011) Examining implementer fidelity: Conceptualizing and measuring adherence and competence.J Child Serv 6: 18-33.
6. Cross WF, Pisani AR, Schmeelk-Cone K, Xia Y, Tu X, et al. (2014) Measuring trainer fidelity in the transfer of suicide prevention training.Crisis 35: 202-212.
7. Fernandez A, Schillinger D, Grumbach K, Rosenthal A, Stewart AL, et al. (2004) Physician language ability and cultural competence: An exploratory study of communication with spanish-speaking patients. Journal of General Internal Medicine 19: 167-174.
8. Gould MS, Cross W, Pisani AR, Munfakh JL, Kleinman M (2013) Impact of Applied Suicide Intervention Skills Training on the National Suicide Prevention Lifeline.Suicide Life Threat Behav 43: 676-691.
9. Howard GS (1980) Response-Shift Bias a Problem in Evaluating Interventions with Pre/Post Self-Reports, Evaluation Review 4(1): 93–106.
10. Lam TC, Bengo P (2003) A Comparison of Three Retrospective Self-Reporting Methods of Measuring Change in Instructional Practice, American Journal of Evaluation24(1): 65-80.
11. Kumas-Tan Z, Beagan B, Loppie C, MacLeod A, Frank B (2007) Measures of cultural competence: examining hidden assumptions.Acad Med 82: 548-557.
12. Peets AD, Cooke L, Wright B, Coderre S, McLaughlin K (2010) A prospective randomized trial of content expertise versus process expertise in small group teaching.BMC Med Educ 10: 70.
13. Pratt CC, McGuigan WM, Katzev AR (2000) Measuring Program Outcomes: Using Retrospective Pretest Methodology, American Journal of Evaluation 21(3): 341-349.
14. Rockwell SK, Kohn H (1989) Post-Then-Pre Evaluation: Measuring Behavior Change More Accurately,” Journal of Extension27(2).
15. Sareen J, Isaak C, Bolton SL, Enns MW, Elias B, et al. (2013) Gatekeeper training for suicide prevention in First Nations community members: a randomized controlled trial.Depress Anxiety 30: 1021-1029.
16. Smith AR, Silva C, Covington DW, Joiner TE (2014) An assessment of suicide-related knowledge and skills among health professionals.Health Psychol 33: 110-119.
17. Swanner J (2014) CalMHSA Suicide Prevention Training. Panel presentation at the 47th Annual Conference of the American Association of Suicidology, Los Angeles, CA.
18. Taub DJ, Servaty-Seib HL, Miles N, Lee J, Morris CAW, et al. (2013) The impact of gatekeeper training for suicide prevention on university resident assistants. Journal of College Counseling 16: 64-78.