The Shoulder Symptom Modification Procedure (SSMP): A Reliability Study

Hilla Sarig Bahat^* and Orna Kerner

Department of Physical Therapy,University of Haifa, Israel.

*Corresponding Author:

Hilla Sarig Bahat
Lecturer, University of Haifa
Physical Therapy, Mt. Carmel
Haifa, Israel
Tel: +972545380483, +972773506060
Fax: +97248288140
E-mail: hbahat@physicalvirtue o l

Received Date: May 16, 2016; Accepted Date: July 26, 2016; Published Date: August 08, 2016

Citation: Sarig Bahat H, Kerner O (2016) The Shoulder Symptom Modification Procedure (SSMP): A Reliability Study. J Nov Physiother S3:011. doi: 10.4172/2165-7025. S3-011

Copyright: © 2016 Sarig Bahat H. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Visit for more related articles at Journal of Novel Physiotherapies

Abstract

Study design: Cross section reliability study. The study assessed inter-tester (n=90) and intra-tester reliability (n=25) of the Shoulder Symptom Modification Procedure (SSMP) in adults with localized shoulder pain. Objective: To evaluate the inter- and intra-tester reliability of the SSMP for shoulder assessment in patients with shoulder pain. Background: Diagnosing the underlying structures responsible for shoulder symptoms is difficult and commonly does not advance treatment. Current orthopedic assessment techniques are mostly provocative, and do not necessarily direct towards an effective strategy. The SSMP was developed to provide a treatment-oriented assessment. Methods: Patients underwent an initial functional test used as a reference for their shoulder symptoms. The SSMP aimed to find a technique that eases these symptoms. It comprised four modification categories: Thoracic kyphosis, scapular position, humeral head position, and neuro-modulation. Each modification resulted as positive if symptoms improved more than 30%. Inter-tester reliability was assessed by comparing findings by 2 physiotherapists (arranged in 3 pairs of 3 blinded testers). Intra-tester reliability evaluation included repeated testing by one physiotherapist. A washout period of 20-40 minutes was allowed in between repeated tests, during which intra-tester assessed/treated others and patients rested. Cohen’s kappa was used for statistical analysis. Results: Inter-tester reliability showed moderate to almost perfect agreement in the thoracic kyphosis and humeral head position categories (Ð�=0.77, 0.86 and Ð�=0.78, Ð�=0.74 respectively). Intra-tester reliability showed moderate agreement in the humeral head category (Ð�=0.66). Other categories showed poor to moderate agreement. Conclusion: Findings showed overall moderate reliability, with good reliability achieved for thoracic and humeral head modifications alone. Reliability was possibly limited due to the changeable nature of the variable being assessed. Symptom intensity may have changed due to modifications, altering the response to the provocation in the second test. Further research should investigate whether higher reliability can be achieved with procedures addressing the limitations identified.

Keywords

Shoulder; Assessment; Reliability; Pain; Rotator cuff

Introduction

Shoulder pain is a common and serious musculoskeletal issue associated with high morbidity rates due to inability in affectively using the upper limb [1]. The causes of shoulder pain include rotator cuff (RC) disorders, glenohumeral joint (GHJ) disorders, acromioclavicular joint (ACJ) disorders, and referred neck pain [1]. The most frequent of the four is RC disorders comprising 30% of all shoulder diagnoses [2]. RC disorder is a broad definition that comprises a variety of pathologies such as tendinopathy, tendinosis, tendinitis, bursitis, and partial to full thickness tears in the rotator cuff [3].

Diagnosing the underlying structures responsible for shoulder symptoms is difficult as clinical features are similar and current assessment techniques have poor reliability [4-7]. Physical assessment of the shoulder is based on the premise that isolation of specific anatomical structures is possible during the physical exam using orthopedic tests that apply compression, stretching or isolated contraction to selected tissues. However, clinical tests designed to assess structural integrity and pain response are unlikely to selectively isolate an individual tissue from adjacent structures, confounding the ability to determine which structure(s) are involved in the patient’s symptoms [8-10]. More specifically, clinical shoulder assessment procedures cannot isolate individual tendons or other structures to inform an accurate diagnosis due to morphology of the RC or the position and innervation of the subacromial bursa (SAB) [3]. Moreover, a number of studies showed poor correlation between symptoms and imaging methods currently used in shoulder assessment [11-14].

Recently, three systematic reviews consistently concluded that no currently available single shoulder physical examination can be recommended to establish a pathological diagnosis, due to lack of accuracy and insufficient likelihood ratios [15-17]. Thus, further hindering clinical decision-making based on these potentially misleading clinical assessment procedures and imaging data. Due to the poor validity and lack of diagnostic value of existing tests, and difficulty in concluding a definitive structural diagnosis in shoulder pain, alternative methods of assessment have been suggested, however their reliability and validity require assessment before being adopted [3,18-21].

The Shoulder Symptom Modification Procedure (SSMP) was developed by Lewis [3] to potentially provide an alternative approach for shoulder assessment. This process involves identifying the movement, posture, or activity that most appropriately reproduces the patient’s symptoms. Once the movement or activity has been identified and agreed upon, the SSMP is applied. The SSMP is a series of mechanical techniques that are applied while the patient performs the activity or movement with the aim of identifying a mechanical change that will alleviate symptoms or improve range of motion [3]. The SSMP involves four principal procedures, which include: (1) the influence of the thoracic posture on symptoms, (2) the influence of scapular position on symptoms, (3) the influence of humeral head procedures, and (4) procedures to neuromodulate symptoms (Appendix A). Test results are based on the response to the modification performed in the assessment, and positive responses may guide treatment [3]. The objective of this study was to investigate the intra- and inter-tester reliability of the SSMP in patients with shoulder pain. Investigating the reliability of the SSMP would evaluate its ability to be used by clinicians for assessment of shoulder pain.

Methods

Participants

This study recruited patients experiencing shoulder symptoms eligible by the following inclusion and exclusion criteria:

Inclusion criteria: Age over 18 years; self-reported pain or symptoms localized around the shoulder, and not referred below the elbow, with or without a restriction in shoulder movement. No restriction on symptom duration was defined.

Exclusion criteria: Irritable shoulder pain that did not settle after provocation; systemic problems such as rheumatic, metabolic, and neurologic conditions; inability to communicate or provide informed consent; unstable fracture/dislocation; and pregnancy. The study was approved by IRB of the University of Haifa, and Maccabi healthcare, Israel. Patients’ assessments were conducted at Maccabi healthcare physiotherapy outpatient clinics in Herzelia and Ramat hasharon.

Informed consent was received from the patients before commencing the study’s procedure, and the rights of the patients’ rights were protected throughout the study.

SSMP assessments

Assessments were conducted by three fully qualified physiotherapists with 15 years or more of clinical experience in the musculoskeletal field. Additionally, assessors underwent a 3-day (24 hours) workshop on SSMP. Prior to the study, the assessors ran SSMP assessments to resolve any discrepancies in the performance of the procedure.

Design

In the intra-tester reliability study, one physiotherapist (tester 1) assessed 25 subjects twice, while the inter-tester reliability study involved three qualified physiotherapists, assessing 90 participants with shoulder symptoms. Inter-tester assessments were conducted in three blocks of 30 patients; each underwent two assessments by two physiotherapists: tester 1 and 2, 1 and 3, and 2 and 3. In all interand intra-tester assessments, each evaluation took approximately 40 minutes, with a wash out period of 20-40 minutes to allow for symptoms to return to baseline levels between the sessions. During this wash out period, participants were encouraged to relax and not perform exercise, adopt extreme postures, or perform unaccustomed activities.

As for the intra-tester assessor, she examined or treated other patients during the wash-out period, to reduce recollection of the first assessment findings. Accordingly, the intra-tester assessor could have not been blinded, but the inter-tester testers were blinded to the other tester’s findings. In addition, patients were instructed to avoid any cues or comments regarding their first assessment results during their repeated test, to prevent bias. In-between testers order was random.

The SSMP commences with a search for a functional test that provokes the symptoms and is then used during the SSMP as a reference for their modification. This provocative functional test may include active shoulder motion, resisted activity, or reproduction of a reported function, such as hand above head, hand behind back, or shoulder elevation. A positive test result was defined as greater than 30% improved symptoms as a result of the modification. The SSMP comprises four primary modification categories that are applied in the following order: 1) Thoracic kyphosis, 2) scapular position, 3) humeral head position, and 4) neuro-modulation. The SSMP assessment form is presented in Appendix A.

Thoracic Kyphosis modification: It was performed by active extension with instruction to “place your finger on your sternum and push it forward” (Figure 1). When positive, this active extension was complimented with thoracic taping both aiming to maintain reduced thoracic kyphosis. The effect of these modifications on symptom provocation was evaluated by self-reported percent of symptom improvement and then recorded.

Figure 1: Kyphosis modification by active thoracic extension following therapist instruction.

Scapular position modification: It includes evaluation of the effect of manual scapular elevation, depression, protraction, retraction, and tilt (posterior/anterior) on shoulder symptoms and mobility. An example of manual scapular elevation is presented in Figure 2. When relevant, a combination of 2-3 positive scapular movements was also evaluated to assess its effect on symptoms. In cases when scapular winging was identified, the effect of manual stabilization and prevention of winging was evaluated.

Figure 2: Scapular position modification: manually moving the scapula into elevation to assess its effect on the provocative test.

Humeral head position modification: It involves the influence of manual techniques on the humeral head. Humeral head depression could be tested in different positions including sitting, supine, and during assisted elevation (flexion/abduction) of the arm pulled by a thera-band connected above shoulder level. Other humeral head modifications included external rotation, anterior-posterior (AP) pressure, or posterior-anterior (PA) pressure. Figure 3 presents examples of humeral head position modification: A) Self- resisted depression; B) Therapist-resisted depression; C) Thera-band resisted humeral head depression, and assisting elevation (could be performed into flexion or abduction). External rotation consisted of 3 repetitions of light Thera-band-resisted external rotation, after which the provocative test was re-evaluated. Assisted elevation included 3 repetitions of Thera-band-resisted arm extension, followed by assisted arm elevation (flexion/abduction), after which the provocative test was re-evaluated.

Figure 3: Humeral Head Position modification: A) self- resisted depression; B) Therapist-resisted depression; C) Thera-band resisted humeral head depression, and assisting elevation (could be performed into flexion or abduction).

Neuromodulation modification

It includes four manual therapy techniques aiming to modify the shoulder symptoms (Figure 4).

These Neuromodulation techniques were performed by fixed order:

1. Manual cervical distraction in sitting.

2. Mulligan’s cervical mobilization with movement.

3. Soft tissue techniques (such as stripping, friction) over the Supraspinatus muscle region.

4. Taping techniques (note these were not kinesio-taping techniques).

Figure 4: Neuromodulation modification included various Cervical/Thoracic spine manual therapy techniques aiming to modify the shoulder symptoms. Presented are examples of A) cervial distraction; B) cervical sustained mobilisation with movement.

Data analysis

Following each component of the SSMP, a reported percentage of symptom improvement was recorded by the tester for each component. This value was interpreted as positive if ≥30% threshold was achieved in two consistent responses by the patient within the testing session. A negative value indicated a change < 30% in symptoms or no change.

In cases of total alleviation of shoulder symptoms, the tester recorded 100% improvement on the SSMP, the procedure was terminated, and the tester did not continue to the next category. In all assessments and categories, the percentage of change was recorded and then interpreted as positive or negative. Agreement was established when two repeated tests achieved a positive or negative value in each category.

Statistical analysis

The inter- and intra-tester reliability was analysed using Cohen’s kappa [22]. Significance was determined at p<0.05. Kappa results were interpreted by Landis and Koch’s [22], who have proposed the following standards for strength of agreement for the kappa coefficient: 0=poor, .01–.20=slight, .21–.40=fair, .41–.60=moderate, .61–.80=substantial, and .81–1=almost perfect. JMP^® and SAS^® software were used for statistical analysis (SAS Institute, Cary, NC,).

Results

From a total of 146 referrals of patients to the study, 30 did not meet the inclusion criteria and one patient was unable to participate. Therefore, 115 patients with shoulder pain participated in the study. Participants’ demographics and characteristics for the intra-tester and inter-tester reliability data are presented in Table 1.

Table 1: Demographic and clinical characteristics for intra-tester patient study.

Intra-tester reliability

Twenty-five subjects participated in the intra-tester investigation (15 males and 10 females). Duration of symptoms ranged from 3 to 100 weeks with an average of 26.2 weeks, (no acute patients participated in the study). Normal distributions were found for all dependent variables. All four categories of the SSMP assessment reached moderate to good kappa agreement (ÐÂ?=0.42-0.66). Table 2 presents the results of the agreement between the first and second test for the four main categories and their corresponding subcategories.

Table 2: Intra-tester Reliability results from the Kappa analysis for each category (n=25).

As seen in Table 2, scapular combinations had the best agreement with a kappa of 0.67 followed by humeral head position (ÐÂ?=0.66), both representing substantial agreement [23]. Moderate agreement was found for thoracic kyphosis (ÐÂ?=0.50); various humeral head techniques: depression in sitting (ÐÂ?=0.54), in supine (ÐÂ?=0.47), and assisted elevation (ÐÂ?=0.47); and neuromodulation (ÐÂ?=0.49). The lowest statistically significant agreement for the main categories was of ÐÂ?= 0.42 for scapular stabilization. Although full agreement was found in winging scapula manual stabilisation (ÐÂ?=1.0), it was based on one case and therefore cannot be generalised.

Inter-tester results

Ninety patients participated in the inter-tester investigation (39 male and 51 female) as presented in Table 1. Similar to the intra-tester population, the inter-tester study included a population with a wide age range between 25 and 84, and BMI ranging between 18 and 35 (Table 1). Normal distributions were found for all dependent variables. Results of overall agreement for the main four categories and their subcategories are presented in Table 3.

Table 3: Inter-tester overall agreement for the main categories and their subcategories in all three pairs (n=90).

The agreements for the inter-tester results ranged widely from almost perfect (ÐÂ?=0.86) to poor (ÐÂ?=0.1). Best agreement was found between assessors 2 and 3 for Thoracic Kyphosis (ÐÂ?=0.86), a category which showed substantial agreement also between assessors 1 and 2 (ÐÂ?=.77), but not for assessors 1 and 3. Substantial agreement was found for Humeral head position modification with a Kappa of 0.78 (2 and 3), and 0.75 (1 and 3). Scapular position modification categories that were found statistically significant showed moderate agreement between all three assessors. All other categories had mixed agreements. Assessors 1 and 2 and 2 and 3 had overall higher agreement than between assessors 1 and 3.

In summary, the intra-tester reliability of the SSMP test for the 4 main categories ranged between moderate to substantial (ÐÂ?=0.42- ÐÂ?=0.66), and the reliability for inter-tester ranged between poor to almost perfect (ÐÂ?=0.18-ÐÂ?=0.86). In the intra-tester study, substantial agreement was found in the humeral head category (ÐÂ?=0.66), and in the inter-tester study, two out of three of the pairs had substantial to almost perfect agreement for the thoracic kyphosis, and humeral head position categories (ÐÂ?=0.77, 0.86 and ÐÂ?=0.78, ÐÂ?=0.74 respectfully).

Discussion

This study investigated the intra- and inter-tester reliability of the SSMP, a new evaluation procedure for patients experiencing shoulder pain introduced by Lewis [3]. The SSMP is designed to identify positive effects on symptoms by changes in position of the glenohumeral or scapulothoracic joints. Mechanical modifications are performed based on the assessment findings, and these modifications are then used to direct treatment.

Best intra-tester agreement was found in scapular combination and humeral head position (ÐÂ?=0.66-0.67). Best inter-tester agreement was found for Thoracic Kyphosis (ÐÂ?=0.86 tester 2 and 3; ÐÂ?=0.77 tester 1 and 2), and Humeral head position (ÐÂ?=0.78 tester 2 and 3, ÐÂ?=0.75 tester 1 and 3).

The moderate to substantial level of agreement found here is in line with previous studies examining reliability of commonly used shoulder assessments. Fair to moderate values were found for shoulder tests such as the Hawkins- Kennedy (ÐÂ?=0.39), the Painful Arc (ÐÂ?=0.45), the Empty Can (Jobe) (ÐÂ?=0.47), the External Rotation Resistance (ÐÂ?=0.67), and Neer’s impingement test (ÐÂ?=0.40) [23]. Moderate agreement was found for the majority of the main categories (ÐÂ?=0.41- 0.57) implying they can also be used in clinics, but may require further development and refinement. Furthermore, unlike the nature of the above orthopedic tests, which are mostly provocative, the SSMP also searches for an alleviating technique. While the SSMP is designed to direct treatment, most orthopedic tests aim to identify a structural source of symptoms. This study showed somewhat better inter-tester than intra-tester reliability. However, the intra-tester results may be of higher relevance for clinical utility since in most cases the same clinician follows up on each patient. It is less common for different clinicians to assess the same patient.

The inter-tester results showed lack of consistency among the three pairs of assessors. A possible explanation for these differences may be the different level of use of the SSMP prior to the study. Only one pair of assessors (tester 2 and 3) had moderate to very good agreement on the main categories. As physiotherapists working together on a daily basis, may have increased their similarity in skills and working habits compared to those of the other two assessors. Nevertheless, in an attempt to avoid such bias prior to data collection, all assessors underwent a short training by performing an SSMP evaluation of 20 patients each and of 3 patients together. However, possibly this amount of training was insufficient and more experience might be needed to improve inter-tester reliability.

The lower than expected reliability found in the current study may be a result of the limited amount of training, the lengthy procedure of the SSMP assessment with its multiple categories, and the therapeutic nature of the technique which would have changed the assessed parameter with each modification. The lengthy SSMP evaluation (at least 30 minutes) may have led to increased performance error by the assessors. There was an attempt to shorten the procedure by ending it when 100% alleviation in symptoms was achieved. Nonetheless, further attempts to shorten and refine the procedure should be considered in the future.

The numerous SSMP subcategories may have increased the variability of the results and lowered the ÐÂ? value [24]. Unlike other assessments, the SSMP breaks down each condition into different categories allowing for increased precision and direction of treatment. Unlike the mechanical diagnosis and therapy classification, for example, where a condition can be classified into one category alone, [25] the SSMP can categorize one patient into a number of categories. The SSMP is comprised of multiple subcategories leading to a higher chance of disagreement between repeated procedures. The procedure assessment defined that when 100% improvement in symptoms was reported, the assessment was to end. Although this helped to shorten the process, this definition ignored the changeability of the assessed parameter and may have reduced reliability.

Study limitations

The changeability of the assessed parameter seems to be the main limitation of the studied procedure. The effect from the first assessment could have altered the patient’s symptoms in the second evaluation, thus lowering the agreement between the two tests. Future studies can control these limitations by assessing all categories in all patients. In addition, video recording of the first test can be later re-evaluated by another tester or the same tester without physically reassessing the patient to avoid the therapeutic effect.

The intra-tester reliability was evaluated by repeated assessments, with a 20-minute wash out period. A longer wash out period of a few days may decrease the potential for tester’s bias due to recollection of the first assessment, and may be closer to the clinical scenario where clinicians re-assess patients after a few days.

Moreover, assessing the therapeutic effectiveness of SSMP-based management is more relevant than assessing its reliability, especially in its traditional design. Further research is needed to investigate the added value of the SSMP as compared to traditional physiotherapy.

Conclusion

The Shoulder Symptom Modification Procedure was developed to potentially provide an alternative approach for shoulder assessment that can guide treatment. The current study found moderate to substantial intra- and inter-tester reliability, comparable to that found for commonly used orthopedic shoulder assessments. Findings showed lower than expected reliability of the SSMP which can be explained by the changeable nature of the variable being assessed. Possibly a change in the symptom’s intensity occurred due to the modifications, altering the response to the second provocative test. Further research should investigate whether higher reliability can be achieved with an amended procedure addressing the limitations identified. Moreover, the novelty of the SSMP as a treatment-oriented procedure warrants further research assessing effectiveness of SSMP-directed therapy.

Key Points

Findings

Our evaluation of the Shoulder Symptom Modification Procedure (SSMP) found moderate to substantial level of agreement between assessors, in line with previous studies examining reliability of commonly used shoulder assessments. Moderate agreement was found for the majority of the main categories (ÐÂ?=0.41- 0.57).

Implication

SSMP can be used in clinics, but requires further development and refinement.

Caution

The numerous SSMP subcategories may have increased the variability of the results and lowered the ÐÂ? value. Reliability was possibly limited due to the changeable nature of the variable being assessed. Symptom intensity may have changed due to modifications, altering the response to the provocation in the second test.

References

1. Luime JJ, Koes BW, Hendriksen IJ, Burdorf A, Verhagen AP, et al. (2004) Prevalence and incidence of shoulder pain in the general population; a systematic review. Scand J Rheumatol 33: 73-81.
2. van der Windt DA, Koes BW, de Jong BA, Bouter LM (1995) Shoulder disorders in general practice: incidence, patient characteristics, and management. Ann Rheum Dis 54: 959-964.
3. Lewis JS (2009) Rotator cuff tendinopathy/subacromial impingement syndrome: is it time for a new method of assessment? Br J Sports Med 43: 259-264.
4. Dinnes J, Loveman E, McIntyre L, Waugh N (2003) The effectiveness of diagnostic tests for the assessment of shoulder pain due to soft tissue disorders: a systematic review. Health Technol Assess 7: 1-178.
5. van der Windt DA, Koes BW, Boeke AJ, Deville W, Jong DBA, et al. (1996) Shoulder disorders in general practice: prognostic indicators of outcome. Br J Gen Pract 46: 519-523.
6. Hegedus EJ, Goode A, Campbell S, Morin A, Tamaddoni M, et al. (2008) Physical examination tests of the shoulder: a systematic review with meta-analysis of individual tests. British journal of sports medicine 42: 80-92
7. Lewis J, Tennent D (2007) How Effective are Diagnostic Tests for the Assessment of Rotator Cuff Disease of the Shoulder? Evidence-based Sports Medicine: John Wiley & Sons, Blackwell Publishing, USA.
8. Hawkins RJ, Kennedy JC (1980) Impingement syndrome in athletes. Am J Sports Med 8: 151-158.
9. Jobe FW, Jobe CM (1983) Painful athletic injuries of the shoulder. ClinOrthopRelat Res 173: 117-124.
10. O'Brien SJ, Pagnani MJ, Fealy S, McGlynn SR, Wilson JB (1998) The active compression test: a new and effective test for diagnosing labral tears and acromioclavicular joint abnormality. Am J Sports Med 26: 610-613.
11. Connor PM, Banks DM, Tyson AB, Coumas JS, D’Alessandro DF (2003) Magnetic resonance imaging of the asymptomatic shoulder of overhead athletes a 5-year follow-up study. The American journal of sports medicine 31: 724-727.
12. Worland R, Lee D, Orozco C, SozaRex F, Keenan J (2003) Correlation of age, acromial morphology, and rotator cuff tear pathology diagnosed by ultrasound in asymptomatic patients. Journal of the southern orthopaedic association 12: 23-26.
13. Frost P, Andersen JH, Lundorf E (1999) Is supraspinatus pathology as defined by magnetic resonance imaging associated with clinical sign of shoulder impingement? Journal of Shoulder and Elbow Surgery 8: 565-568.
14. Milgrom C, Schaffler M, Gilbert S, van Holsbeeck M (1995) Rotator-cuff changes in asymptomatic adults. The effect of age, hand dominance and gender. J Bone Joint Surg Br 77: 296-298.
15. Alqunaee M, Galvin R, Fahey T (2012) Diagnostic accuracy of clinical tests for subacromial impingement syndrome: a systematic review and meta-analysis. Archives of physical medicine and rehabilitation 93: 229-236.
16. Hegedus EJ (2012) Which physical examination tests provide clinicians with the most value when examining the shoulder? Update of a systematic review with meta-analysis of individual tests. Br J Sports Med 46: 964-978.
17. Hughes PC, Taylor NF, Green RA (2008) Most clinical tests cannot accurately diagnose rotator cuff pathology: a systematic review. The Australian journal of physiotherapy 54: 159-170.
18. Carter T, Hall H, McIntosh G, Murphy J, MacDougall J, et al. (2012) Intertester reliability of a classification system for shoulder pain. Physiotherapy 98: 40-46.
19. Exelby L (1996) Peripheral mobilisations with movement. Man Ther 1: 118-126.
20. May S, Ross J (2009) The McKenzie classification system in the extremities: a reliability study using Mckenzie assessment forms and experienced clinicians. Journal of manipulative and physiological therapeutics32:556-563.
21. Sahrmann S (2002) Diagnosis and Treatment of Movement Impairment Syndromes. MosbyElsevier, USA.
22. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33: 159-174.
23. Michener LA, Walsworth MK, Doukas WC, Murphy KP (2009) Reliability and diagnostic accuracy of 5 physical examination tests and combination of tests for subacromial impingement. Archives of physical medicine and rehabilitation90:1898-1903.
24. Tooth LR, Ottenbacher KJ (2004) The kappa statistic in rehabilitation research: an examination. Arch Phys Med Rehabil 85: 1371-1376.
25. May S, Littlewood C, Bishop A (2006) Reliability of procedures used in the physical examination of non-specific low back pain: a systematic review. Aust J Physiother 52: 91-102.