Development and Validation of A New Tool for Assessing Risk of Falls in Acute Psychiatric Settings

¹University of Milan, Italy

²Tutor nurse, San Paolo Bachelor School of Nursing, San Paolo Hospital, Milan, Italy

³Associate Professor of Nursing, University of Milan, Italy

*Corresponding Author:

Paolo Ferrara
Tutor nurse, San Paolo Bachelor School of Nursing
San Paolo Hospital, Milan
Italy
Email: paolo.ferrara@asst-santipaolocarlo.it

Visit for more related articles at International Journal of Emergency Mental Health and Human Resilience

Abstract

Background: Falls in hospitals are a global problem, due to their frequency and the consequences for the person, the operators, the organization. The literature shows that patients suffering from mental disorders, especially inacute settings, have specific characteristics that increase their risk. Two scales are available in literature for these patients, but one is not fully validated, and the other has unsatisfactory predictive validity. Furthermore, no tools exist in Italian.

Aim: To create and validate a tool, in English and in Italian, to assess risk to fall in patients hospitalized acute psychiatric settings. Materials and Methods: upon literature review, a new scale was created and administered to the patients of two psychiatric services in different Italian hospitals. Validity and reliability of the scale were assessed by means of factor analysis, content validity index, and Cronbach’s alpha coefficient.

Results: The scale consists of 11 items (CVI-S = 90.9%); 7 clinical experts have positively judged comprehensibility and the uniqueness of the items. 123 patients were screened (at admission, and after 24 and 48 hours), 10 of them reported falls; with a cut-off of 11 points, the scale showed 100% sensitivity and 80.2% specificity. Inter-rater reliability was high (Pearson's r = 0.93).

Conclusions: the new scale seems easy to use and capable of predicting falls in psychiatric patients admitted to acute hospital units.

Keywords

Fall, Risk, Psychiatry, Nursing, Assessment

Background

Falls in hospital are relevant adverse events, due to their frequency and potential consequences, for both patients and healthcare facilities (Saccomano & Ferrara, 2015; Cattelani et al., 2015). In literature, their incidence ranges between 10 and 17 for 1,000 patient bed days, with physical consequences in 30% of the cases (Hill et al., 2015). Furthermore, the increasing length of stay due to diagnostic and therapeutic activities contributes to raising costs. (Centers for disease control and prevention – Home and recreational safety., 2015) (Virdis et al., 2012).

Most falls are preventable, because they are related to environmental and human risk factors that can be taken into consideration to implement preventive measures (Morse, 2002).

Based on these aspects, in Italy death or severe damage due to falls is considered a sentinel event; in 2011, the Ministry of Health has published a recommendation aiming to promote the implementation of programmes targeted at improving understanding and reduction of risks in all healthcare facilities, according to the criteria defined by the Joint Commission International (The Joint Commission, 2009). Nursing assessment, as a core part of wider prevention policies, is geared to identify patients at risk of falling and consequently to implement valid preventive strategies; as regards the detection of intrinsic risk factors, such activity can be supported by one of the many screening tools available in the literature (Kim, Mordiffi, Bee, Devi & Evans, 2007) (Lovallo et al., 2010).

The person suffering from mental disorders, especially during hospitalization and therefore in the acute stage, presents specific risk factors related to higher risk of falling, such as altered mental status, use of psychotropic drugs, gait and balance impairment, and sleep disorders (Allen et al., 2012). Incidence of falls in this population ranges from 13.1 to 25 per 1,000 patient bed days. (Blair, 2005)

Despite these considerations, few papers are available on this topic; only in recent times, two scales have been published, which assess the risk of falls in the psychiatric population, the Edmonson Psychiatric Fall Risk Assessment Tool - (EPFRAT) and the Wilson-Sims Fall Risk Assessment Tool (WSFRAT). The first has better predictive values than generic scales, but has unsatisfactory sensitivity (0.63). (Edmonson et al., 2011). The WSFRAT has only undergone preliminary validation so far, consisting in content validity analysis (Billeen et al., 2013). Furthermore, no specific tools validated in Italian currently exist, and the two abovementioned scales investigate different aspects of the problem. In particular, both scales investigate age, mental status, sleep disorders, nutritional behavior disorders, previous falls, urinary and faecal elimination, drug therapy, and balance impairment. The EPFRAT investigates, psychiatric diagnosis, nutrition and hydration, while the WSFRAT includes gender and physical status. The WSFRAT investigates the influence of different drug types while the EPFRAT only includes the overall number of drugs taken by the patient.

Based on these considerations, it makes sense to merge the items of the scales, in order to obtain a validated tool in Italian. The purpose of this study is to create and validate a screening tool to support nurses in assessing the risk of falling of the person hospitalized in Psychiatric inpatient units.

Material And Methods

We conducted a literature search on PubMed, CINAHL, and Cochrane Library, to retrieve data on the main risk factors in the psychiatric population, as well as to identify dedicated tools in this context. Articles published between 2000 and 2015 in English and Italian were taken into consideration. This literature search confirmed the lack of dedicated tools in Italian.

After this first step, we analyzed the content of the EPFRAT and WSFRAT. This analysis pointed out an overlap of some items.

Based on the findings of these two steps, we created a new tool by including all the items of the two abovementioned scales. The new instrument consisted of 14 items; we chose to name it IPFRAT, which stands for Italian Psychiatric Fall Risk Assessment Tool.

The literature review aimed to investigate the presence of additional risk factors not included in the two original instruments.

The "gait / balance" factor was identified and added to the new scale. Moreover, in accordance with the results of the study by Chan 2013 (Chan et al., 2013), changing dosage of medication is considered influential if it occurs during the 24 hours before risk assessment. This item was included as well.

The content validity of the new instrument was tested by calculation of the Content Validity Index (CVI) for every single item (CVI - I) and for the scale as a whole (CVI - S). (Burns & Grove, 2009; Secginli, 2012a). The new scale was assessed by 5 nurses and 2 psychiatrists from a major Milan hospital. Of the 14 initial items, 3 had unsatisfactory CVI - I (taking mood-stabilizing medications, taking antidepressants, intake of drugs active on the cardiovascular system) and were therefore eliminated. For each of the 11 items, we defined specific scores, according to the CVI-I of the item itself (CVI-I 100%: 6 points, CVI-I 85.71%: 5 points, CVI-I 71.43%. 4 points (the percentages come from the number of nurses and psychiatrists who agreed on the item). Each score was divided into three levels, and resulted in the final version of the scale shown in Table 1 (English) and Table 2 (Italian). The Content Validity of the scale was then calculated, with a result of 90.9%, which is a high value. The final version of the tool allows investigating the risk of falling of the patients admitted to the inpatients psychiatry unit, through direct observation (items no. 3, 4, and 6 ), interview (item 1, 5, 7, and 8 ) and medical history (items 2, 9, 10, and 11 ).

Table 1: The new scale – English version

Table 2: The new scale – Italian version

We conducted a longitudinal, multicentric study, at the San Paolo teaching Hospital and the Hospital of Vizzolo Predabissi, Lombardy, North-Western Italy. The study followed the principles of the Declaration of Helsinki and the Italian law on data protection. The institutional review board approved the study; we complied with the rules of the local ethical committee.

The scale was administered to all patients at the admission, in the period from April to July 2015. Risk of falling was also assessed at 24 and 48 hours from admission and after each fall; the person was simultaneously assessed with the Conley-scale, in order to test concurrent validity of the new instrument. This aspect of methodology is of particular relevance, because it allows comparison between a new tool, which is still being tested, and a validated scale which is already considered reliable. Moreover, since we based this research on the hypothesis that non-dedicated scales can be inappropriate for psychiatric patients, this comparison may lead to highlighting the real capacity of both tools to identify patients at risk.

The survey was conducted by a nurse, previously trained by the project leader about the aims and methods of data collection. The first 15 evaluations were conducted simultaneously and independently by the nurse and by a nursing student in her final year of bachelor course, in order to assess interrater reliability. Data were entered in Microsft Excel and analyzed with SAS University Edition for MacOS-X.

Cronbach's alpha coefficient and and Pearson's r coefficient were used, to assess internal consistency and interrater reliability respectively. Area under the ROC curve (Receiver Operator Characteristic) as well as sensitivity, specificity, positive and negative predictive value were calculated. Multivariate analysis was performed with logistic model, to study the association between risk of falling and all possible risk factor taken into account in the new scale. Factor analysis was finally performed; sample adequacy was evaluated with the criterion of Kaiser-Mayer-Olkin and Bartlett's test of sphericity. Kaiser’s criterion was applied to retain factors, which were rotated with the Varimax algorithm.

Results

We enrolled 123 patients in 3 units (76 at the San Paolo Hospital in two units, 47 at Vizzolo Predabissi), 64 males and 59 females. Their mean age was 45 ± 15 years ; 97 (78.8%) were were aged 16 to 50, 21 between 60 and 70 (17.1%), the remaining 5 had more than 70 years (4.1%). Table 3 shows the medical diagnoses.

Table 3: Diagnoses

Only 11 patients had history of falls, 5 others declared they were afraid to fall. During the observation period, 10 different patients fell, once each. Of these, 2 patients had history of falls, and 1 was afraid of falling; 4 were oriented. 7 patients in the overall sample were psychotic, one had manias, 2 were depressed; their age had an average of 48 ± 16 years. 7 patients were females, 3 males.

Only 2 persons in the sample did not have lack of muscle weakness, fatigue, dizziness, or cachexia; only 3 were able to ambulate autonomously. 5 had sleeping disorders, already known before admission. 4 used to take benzodiazepines before admission, 2 started taking benzodiazepines during hospitalization, or to associate multiple drugs. 2 patients were taking narcotics, one began such therapy during hospitalization. 6 were already receiving antipsychotics before the hospitalization, one initiated them during hospitalization.

Validity, Reliability and Structure of the Scale

The scale was reviewed by 5 nurses and 2 psychiatrists who gave positive evaluation of face validity. The literature suggests that a value of the Content Validity Index Instrument (CVI-S) equal to or greater than 80% is the minimum standard to guarantee adequate content validity. (Burns & Grove, 2009; Secginli, 2012). The CVI-S of the scale with the 11 items which had satisfactory CVI-I (>70%) was 90.9%.

Cronbach's alpha was 0.63. This value was not unexpected, because the variables that contribute to the risk of falling are heterogeneous and do not necessarily have a strong correlation with each other.

Factor analysis investigated the structure of the scale; the sample was adequate (KMO = 0.78, Bartlett p < 0.001 test). During the construction of the scale, we did not deem necessary to introduce any sub-scales, considering the nature of the clinical problem; factor analysis conformed the existence of a single scale.

Among the 4 factors extracted, in fact, only the first showed factorial loads above the cutoff defined by the criterion of Stevens, equal to 0.301 (Table 4).

Table 4: Factor loadingscorrectly

All the items showed relevant loads, except for sleep disorders, which will be discussed later in the article.

Results on Patients

In agreement with the hypothesis of this study, according to which the generic scales are not adequate identification of fall risk in these patients, the Conley scale gave scores corresponding to absence of any risk in 5 patients out of the 10 fallen (50%).

Since all patients at risk of falling had a score greater than or equal to 11, we initially chose this value as the cut-off and calculated sensitivity and specificity. According to this cut-off, in 69 observations our patients were at risk. The sensitivity according to the cut-off 11 is obviously equal to 100%; the specificity is 80.72%, thus good, with a negative predictive value of 100% and a very low positive predictive value (14.5%). Actually, patients wrongly judged at risk and therefore potentially susceptible to preventive interventions, were actually a few (59 surveys of patients not at risk of falling out of 316 total surveys). The ROC curve (Receiver Operating Characteristics) is equal to 0.90, 95% CI [0.88; 0.92] as shown in Figure 1.

Figure 1: ROC curve with cut-off set to 11

We also performed the same analysis after arbitrarily raising the cut-off to 12. With the such cutoff, 48 patients instead of 69 were at risk. The sensitivity decreased to 90.0%, and specificity to 80.7% (area under ROC curve 0.89, 95% CI [0.79; 0.98]). This cut-off therefore sacrifices sensitivity by 10%, but improves specificity by 7 points specificity. As a conclusion, in our sample setting the cutoff to 11 is preferable, but it is possible that, by repeating the analysis with a larger number of events, the cut-off of 12 becomes a convenient compromise.

Overall, from a clinical point of view, the new IPFRAT scale correctly identifies all cases in which the patient actually falls. No fallen patients had benn identified as “not at risk”. On the other hand, in 59 cases out of 306 (19.3%) patients that did not fall had been identified as "at risk." From a practical standpoint, however, the classification of the patient as "at risk of falling" does not necessarily imply a substantial burden of care. In order to quantify the strength of the association between each variable and risk of falling, we dichotomized all items; Table 5 summarizes the results of the analysis.

Table 5: Association between risk of falling and single items

Sleep disturbances seems to show a protective association, which is actually biased by the fact that such disorders are present in 24 observations out of 69 at risk; this finding needs confirmation on larger samples. Apart from statistical significance, from a clinical point of view the hypothesis of a close relationship between those disorders of sleep and the risk of falling is not sustainable. This variable should therefore be studied in a multivariate model.

When repeating the analysis looking for an association with the event fall instead of the risk, the statistical significance disappears; this makes sense from a clinical point of view, since the scale of the item are risk factors, which by definition are neither necessary nor sufficient for the event.

In the real world, hardly ever do patients present with only one of the characteristics investigated by this scale; for this reason, it was considered useful to conduct a multivariate analysis using a logistic model. Table 6 shows the outcome of the analysis.

Table 6: Multivariate analysis

As seen in the table, the lack of significance regarding sleep disorders remains; However, since such a characteristic was present in the majority of cases (146 observations of 149) already before admission, it is reasonable to think that ost of these patients were already in therapy. The drugs used for these disorders are tipically benzodiazepines, which in the model are strongly linked to the risk of falling. In the multivariate model, closer to clinical reality of patients, the statistical significance of the association between risk of falling and benzodiazepines or narcotics is clear. With regard to alterations of elimination, an analysis was conducted to only consider the patients with the presence of nocturia and incontinence: of the 15 cases (corresponding to the same observations) with such characteristics, 12 were at risk. Among the observations in which patients presented taking diuretics or laxatives, however, only 3 of 13 corresponded to a fall hazard. The presence of this variable appears reasonable from a clinical point of view, and deserves study on larger samples. The two nurses have agreed on all the observation which were considered at high risk by using the score 11 as a cutoff. 221 out of 247 patients were considered as “not at risk” by both nurses (89.5%) as shown in figure 2 (Cohen’s kappa 0.92).

Figure 2: Interrater reliability

Prior to the analysis, the scores underwent square root transformation for normalization. The robust linear regression model suggests of a strongly significant correlation (Pearson’s r = 0.93, p< 0.0001, R-square = 0.87).

Conclusions

The problem of falls has been extensively studied and debated in the literature; however, to date there are few detailed information on the phenomenon regarding psychiatric inpatients in acute care settings. The absence of studies in the Italian has also been the starting point for the creation of a new scale aimed at exploring the event in our context.

Our tool, named IPFRAT, comprises 11 items, and is easy to understand and use in a real setting. The 10 fall episodes observed during the study period confirm the data found in literature about the frequency and therefore the importance of the event (Blair, 2005; Kerzman et al., 2004).

The current cutoff set to 11 guarantees a sensitivity of 100%, a specificity of 80.72% and a 100% negative predictive value; the positive predictive value is low (14.7%) but actually patients mistakenly considered at risk of falling and thus potentially subject to preventive interventions, are actually a few (n = 59).

In our study, some of the patients at risk did not actually fall; in our opinion, this is due to the clinical judgement of nurses, which leads to preventive actions. For example, nurses would pay special attention during mobilization of a patient who appeared disoriented because of drugs, thus reducing the actual risk of falls.

Some of the odds ratios regarding the association between falls and risk factors investigated by the scale showed large confidence intervals. From a statistical point of view, it seems reasonable to adopt a conservative approach, thus considering the lower limit of the interval to have an idea of how strong the association might be in the overall population. Such values suggest that the variables investigated by the new scale increase the odds of falls by a minimum of 9% (diagnosis) to a maximum of 11.56 times (walk/step/gait/balance). This finding suggests that the items of the scale all represent important characteristics that nurses should consider during patient assessment.

Finally, in this research we chose to work exclusively on intrinsic risk factors, which are patients’ characteristics. However, a thorough multifactor risk assessment is always needed, and other aspects, such as environmental factors or other patients’ behavior, should be taken into consideration as potential additional risk factors.

Nurses play an important role in prevention activities, contributing to the promotion of the best of the assisted security and participating in clinical risk management initiatives and monitoring of adverse events, such as falls in hospital. Obviously, the creation of a screening tool such as the one proposed in this paper plays a major role but that only represents the first step to be included within any management path. Despite the excellent properties demonstrated by the instrument and the promising results, further studies are needed, aimed at confirming the data obtained and exploring the validity of the scale on larger samples.

References

1. Allen, D.E., De Nesnera, A.&Robinson, D.A. (2012). Psychiatric patients are at increased risk of falling and choking. Journal of American Psychiatryand Nurses Association,18(2), 91-95
2. Blair, E. (2005). Falls in an inpatient geriatric psychiatric population. Journal of American Psychiatry and Nurses Association, 11, 351��?354.
4. Burns, N.&Grove, S.K. (eds.)(2009). The practice of nursing research appraisal, synthesis, and generation of evidence
5. Cattelani, L., Palumbo, P., Palmerini, L., Bandinelli, S., Becker, C., Chesani, F., et al. (2015). FRAT-up, a Web-based fall-risk assessment tool for elderly people living in the community. Journal of Medical Internet Research,17, e41.
7. Centers for disease control and prevention ��? Home and recreational safety (2015). Costs of falls among older adults. Retrieved from www.cdc.gov on January 8, 2016
8. Chan, C.H., Gau, S.S.F., Chan, H.Y., Tsai, Y.J., Chiu, C.C., Wang, S.M., et al. (2013). Risk factors for falling in psychiatric inpatients: a prospective, matched case-control study. Journal of Psychiatry Resiliance, 47, 1088��?1094.
10. Edmonson, D., Robinson, S.& Hughes, L. (2011). Development of the Edmonson psychiatric fall risk assessment tool. The Journal of Psychosocial Nursing and Mental Health Services, 49, 29��?36.
12. Hill, A.M., McPhail, S.M., Waldron, N., Etherton-Beer, C., Ingram, K., Flicker, L., et al. (2015). Fall rates in hospital rehabilitation units after individualised patient and staff education programmes: a pragmatic, stepped-wedge, cluster-randomised controlled trial. Lancet London England, 385, 2592��?2599.
14. Kerzman, H., Chetrit, A., Brin, L. &Toren, O. (2004). Characteristics of falls in hospitalized patients. Journal of Advanced Nursing 47, 223��?229.
16. Kim, E.A.N., Mordiffi, S.Z., Bee, W.H., Devi, K. &Evans, D. (2007). Evaluation of three fall-risk assessment tools in an acute care setting.Jounral of Advanced Nursing, 60(4), 427-35
17. Lovallo, C., Rolandi, S., Rossetti, A.M. & Lusignani, M. (2010). Accidental falls in hospital inpatients: evaluation of sensitivity and specificity of two risk assessment tools. Jounral of Advanced Nursing,66, 690��?696.
19. Mary, B., Billeen, Kruszewski, A., Sims, K., Fettes, S.& Wilson, S. (2013). Content validity of the Wilson-Sims falls risk assessment tool to measure fall risk of psychiatric inpatients
20. Morse, J.M. (2002). Enhancing the safety of hospitalization by reducing patient falls. American Journal of Infectious Control, 30, 376��?380
21. Polit, D.F. & Beck, C.T. (2006). The content validity index: Are you sure you know what’s being reported? critique and recommendations. Research in Nursing and Health, 29, 489â�?�?497.
23. Saccomano, S.J. & Ferrara, L.R. (2015). Fall prevention in older adults. Nursing Practise, 40, 40��?48
25. Secginli, S. (2012). Mammography self-efficacy scale and breast cancer fear scale: Psychometric testing of the Turkish Versions. Cancer Nursing,35, 365��?373.
27. Secginli, S. (2012b). Mammography self-efficacy scale and breast cancer fear scale: Psychometric testing of the Turkish Versions. Cancer Nursing, 35, 365��?373.
29. The Joint Commission (2009). Implementation guide for the NQF endorsed nursing-sensitive care measure set.
31. Van Dyke, D., Singley, B., Speroni, K.G. &Daniel, M.G. (2014). Evaluation of fall risk assessment tools for psychiatric patient fall prevention: a comparative study. Journal of Psychosocial Nursing in Mental Health Service, 52, 30��?35.
33. Virdis, A., Mara, E., Javarone, M.A. & Zanda, S.I. (2012). Prevention of patient’s falls in University Hospital to Sassari, Italy. Annalali e idigene MedicinaPreventia e Comunit�? , 24(3), 249â�?�?259.