Validity and reliability of the 12-item Berg Balance Scale in an Italian population with Parkinson’s disease: A cross sectional study

ABSTRACT

Background: The Berg Balance Scale is widely used to measure balance ability in clinical practice. Recently, the original version was redefined from 14 into 12 items. Its psychometric properties were investigated for different populations. However, for Parkinson disease the new version has not been validated yet. Objective: The purpose of the present study was to evaluate psychometric properties of the 12-item Berg Balance Scale (BBS-12) in a population with Parkinson disease. Methods: Internal consistency was evaluated with Cronbach’s alpha coefficient, whereas reliability was assessed with the intraclass correlation coefficient. For validity analysis, the Pearson correlation coefficient of the BBS-12 was evaluated with the Tinetti Scale and the Physical Activity Scale for the Elderly. Results: The BBS-12 was applied to 50 individuals with a mean age of 65.6 years (SD 11.8). The internal consistency showed a good value (Cronbach's alpha 0.886) and reproducibility reveled very high performances for both inter-rater and intra-rater reliabilities (ICC 0.987 and 0.986, respectively). The validity study demonstrated good linear correlation with the Tinetti Scale (p<0.01) and with the Sport and Home Subscales of the Physical Activity Scale for the Elderly (p<0.01). Conclusions: The present findings revealed the BBS-12 as a reliable and valid assessment tool to measure balance ability in Parkinson disease. Italian health professionals can now use it with more confidence.

RESUMO

Introdução: A Escala de Equilíbrio de Berg é amplamente usada para medir a capacidade de equilíbrio na prática clínica. Recentemente, a versão original foi redefinida de 14 para 12 itens. Suas propriedades psicométricas foram investigadas para diferentes populações. No entanto, para a doença de Parkinson, a nova versão ainda não foi validada. Objetivo: O objetivo do presente estudo foi avaliar as propriedades psicométricas da Escala de Equilíbrio de Berg de 12 itens (Berg Balance Scale - BBS-12) em uma população com doença de Parkinson. Métodos: A consistência interna foi avaliada pelo coeficiente alfa de Cronbach, ao passo que a confiabilidade foi avaliada pelo coeficiente de correlação intraclasse. Para análise de validade, foi avaliado o coeficiente de correlação de Pearson da BBS-12 com a escala de Tinetti e a escala de atividade física para idosos. Resultados: A BBS-12 foi administrada a 50 indivíduos com idade média de 65,6 anos (DP 11,8). A consistência interna mostrou um valor bom (alfa de Cronbach 0.886) e a reprodutibilidade revelou desempenhos muito altos para a confiabilidade inter e intra-avaliadores (ICC 0.987 e 0.986, respectivamente). O estudo de validade demonstrou boa correlação linear com a Escala de Tinetti (p<0,01) e com as Subescalas de Esporte e Doméstica da Escala de Atividade Física para Idosos (p<0,01). Conclusões: Os resultados encontrados revelaram a BBS-12 como uma ferramenta de avaliação válida e confiável para medir a capacidade de equilíbrio na doença de Parkinson. Os profissionais de saúde italianos agora podem usá-la com mais confiança.

Keywords:

Parkinson Disease - Postural Balance - Psychometrics - Validation Studies - Outcome Measure

Palavras-chave:

Doença de Parkinson - Equilíbrio Postural - Psicometria - Estudos de Validação - Medida de Resultado

INTRODUCTION

Over the past generation, the number of individuals with Parkinson’s disease (PD) globally has more than doubled to over 6 million[1]. In 2016, the estimated regional incidence rate of PD in Italy was 0.28 new cases/1000 person-years, with a prevalence of 3.89/1000 persons[2]. The disease incidence increases with older age and is more common among males than in females[3],[4]. Impairment of postural control increases the risk of falls[5],[6], and people with PD experience two to three times more falls than healthy older adults[7].

Over the last few years, the interest for evaluating both motor and non-motor symptoms of PD is increasing in Italy. Italian health professionals and researchers can now use different assessment tools, such as the Parkinson’s Disease Questionnaire 39[8], the Non Motor Symptoms Scale[9], the Parkinson's Disease-Cognitive Rating Scale[10], Freezing of Gait Questionnaire[11] and the Geriatric Depression Scale[12].

In relation to balance evaluation, different assessment tools are used. The Berg Balance Scale (BBS)[13] is widely used in different settings and the psychometric properties for PD population have been validated in different languages, such as Brazilian Portuguese[14], Persian[15] and American English[16].

The BBS is a 14-item tool to assess balance ability[13]. The items examine the subject’s ability to maintain positions or movements of increasing difficulty by diminishing the base of support from sitting and standing to single-leg stance[13],[17].

In 2005, Franchignoni and colleagues applied the BBS on a sample of 57 individuals with PD[18]. In 2012, a Rasch Analysis study verified internal validity and reliability of the BBS in individuals with different neurological diseases, redefining the questionnaire from 14 to 12 items. The study[19] supported clinimetric properties of the 12-item BBS (BBS-12) as a measurement tool independent from the etiology of the neurologic disease causing balance impairment.

Considering that in the study validating the BBS-12 there were no people with extrapyramidal diseases, the primary objective of the present investigation was to evaluate validity and reliability of the BBS-12 in a PD population.

METHODS

A cross-sectional study was performed to investigate the psychometric properties of the BBS-12. A research group of Sapienza University of Rome and Rehabilitation & Outcome Measures Assessment (ROMA) association conducted the study. The research group dealt with the validation of different outcome measures in Italy[20],[21],[22],[23]. The institutional review board of Sapienza Università di Roma approved the study and guaranteed ethical standards and procedures. All procedures performed were in accordance with the 1964 Helsinki declaration, and its later amendments or comparable ethical standards. Informed consent was obtained from all individual participants included in the study. The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

The pre-established sample size was determined by analyzing other validation studies on PD population (sample range 38–53), thus a probable non-convenience sample with a minimum of 46 individuals was required. To be enrolled in the study, participants had to fit the following inclusion criteria: be clients of the Department of Human Neurosciences of Sapienza University of Rome, have diagnoses of PD according to the clinical diagnostic criteria of the Movement Disorder Society for PD[24], be ≥40 years old, achieve a Mini-Mental State Examination score ≥23 points.

Data measurements

The BBS-12 is a specific tool for evaluating balance. Items 2 and 3 of the BBS (Sitting unsupported and Standing unsupported, respectively) were deleted from its original version[19]. The BBS-12 results, therefore, composed as follow: Transfers, from standing to sitting; from sitting to standing; standing with eyes closed; turning trunk (feet fixed); standing with feet together; reaching forward while standing; retrieving an object from floor; tandem standing; standing on one leg; turning 360°; placing alternate foot on a stool. The total score of the BBS-12 ranged from 0 to 35. For further information on calibrations and rescoring pattern refer to the study by La Porta and colleagues[19].

The Physical Activity Scale for the Elderly (PASE) (25) consists of 10 items that focus on three domains of activity over a period of seven days: leisure (5 components), household (4 components), and work-related (1 component) activities. Participation in leisure activities is recorded by frequency (e.g., never, seldom, sometimes, and often) and duration (e.g., less than an hour, two-four hours, or more than four hours); paid or unpaid work is recorded by total hours of work per week; and housework, lawn work, home repair, outdoor gardening, and care for others are recorded with yes or no answers[25],[26]. For the present study, the Italian version of PASE (PASE-I)[27] was used.

The Tinetti test is a performance-oriented assessment of mobility problems. It consists of nine components of initiation of gait, step height and length, step symmetry and continuity, path deviation, trunk stability, walking stance, and turning while walking[28]. Each component was scored as 1 (normal) or 0 (abnormal).

Procedures and data analysis

First, the researchers (a neurologist, a physical therapist and an occupational therapist) assessed participants according to the inclusion criteria. Recruitment strategies included the use of brochures and the organization of face-to-face meetings within the department for both inpatient and outpatient care. After explaining the objectives of the study, an informed written consent form was obtained from people who agreed to participate in it. All participants were asked to complete a socio-demographic questionnaire followed by the administration of BBS-12, PASE-I and Tinetti scale; data collected regarded age, sex, education and employment status.

Internal consistency is a measure based on the correlations between different items on the same test. Internal consistency was examined using Cronbach’s Coefficient Alpha; as recommended by Nunually[29] the significant coefficient was set as ≥0.70.

The intraclass correlation coefficient (ICC) was calculated to assess reproducibility. To evaluate intra-observer reproducibility the same participant was evaluated twice by the same rater; to ensure that no clinical changes occurred, the second evaluation was scheduled within seven days after the first evaluation. To assess inter-observer reproducibility, two raters assessed participants at the same time. The two raters were blindfolded. Two-way random ICC for absolute agreement was adopted to evaluate reproducibility. ICC ranges from 0 (no agreement) to 1 (perfect agreement) and was interpreted as follows: 0.00-0.25=little, if any, correlation; 0.26-0.49=low correlation; 0.50-0.69=moderate correlation; 0.70-0.89=high correlation; and 0.90-1=very high correlation[30].

To evaluate concurrent validity, the BBS-12 score was compared to the values of PASE-I and Tinetti. The three assessment tools were applied together and the Pearson Correlation Coefficient (PCC) was calculated. PCC can be interpreted as follow: 0 indicates no linear relationship; +1/-1=perfect positive/negative linear relationship; between 0 and ±0.3=weak relationship; between ±0.3 and ±0.7=moderate relationship; between ±0.7 and ±1.0=strong relationship[31]. All statistical analyses were carried out using Statistical Package for Social Sciences (SPSS).

RESULTS

Participants were recruited in the period from March 1^st to December 31^st, 2018, by the Department of Human Neurosciences of Sapienza University of Rome. The BBS-12 was applied to 50 people, together with PASE-I and Tinetti. The demographic characteristics of the sample are summarized in [Table 1].

Table 1 Sample sociodemographic characteristics (n = 50).

Age - mean years (SD)	65.6	11.8
Sex	n	%
Female	12	(24)
Male	38	(76)
Education	n	%
Secondary School	10	(20)
High School	33	(66)
University	7	(14)
Employment	n	%
Office worker	19	(38)
Freelance professional	14	(28)
Housewife	6	(12)
Unemployed	11	(22)

BBS-12: Berg Balance Scale 12 Items; SD: standard deviation.

The internal consistency showed an alpha coefficient of 0.886. Item-total correlation analysis revealed positive and statistically significant values (range 0.872-0.889), as reported in [Table 2].

Table 2 Total-item correlation analysis of the BBS-12.

	Scale mean if item deleted	Scale variance if item deleted	Corrected item-total correlation	Squared multiple correlation	Cronbach's alpha if item deleted
Item1	24.20	17.551	0.788	0.783	0.866
Item2	23.22	17.971	0.589	0.643	0.877
Item3	23.22	18.583	0.593	0.557	0.877
Item4	25.14	18.531	0.625	0.663	0.875
Item5	24.46	18.988	0.547	0.671	0.879
Item6	24.66	17.535	0.602	0.503	0.876
Item7	25.12	18.271	0.701	0.759	0.872
Item8	23.40	15.755	0.704	0.763	0.873
Item9	25.52	18.826	0.516	0.712	0.881
Item10	25.48	19.928	0.315	0.533	0.889
Item11	25.16	18.015	0.589	0.510	0.877
Item12	24.78	18.175	0.576	0.423	0.877

BBS-12: Berg Balance Scale 12 Items.

The reliability study showed an ICC of 0.986 and 0.987 for intra-observer and inter-observer reliability, respectively. Results for each item are reported in [Table 3].

Table 3 BBS-12 reliability results for 50 people with Parkinson’s disease.

BBS12	Intra-Rater Reliability			Inter-Rater Reliability
Item	Rater 1 (SD)	Rater 2 (SD)	ICC (lower-upper bund)	Rater 1 (SD)	Rater 2 (SD)	ICC (lower-upper bund)
1	2.56 (0.54)	2.52 (0.54)	0.966 (0.938–0.980)	2.56 (0.54)	2.56 (0.50)	0.961 (0.931–0.978)
2	3.54 (0.61)	3.52 (0.61)	0.987 (0.976–0.992)	3.54 (0.61)	3.52 (0.54)	0.918 (0.856–0.954)
3	3.54 (0.50)	3.54 (0.50)	1.00 (1.00–1.00)	3.54 (0.50)	3.52 (0.50)	0.936 (0.888–0.964)
4	1.62 (0.49)	1.64 (0.48)	0.979 (0.962–0.988)	1.62 (0.49)	1.64 (0.48)	0.979 (0.962–0.988)
5	2.30 (0.46)	2.28 (0.45)	0.976 (0.957–0.986)	2.30 (0.46)	2.32 (0.47)	0.977 (0.959–0.987)
6	2.10 (0.68)	2.14 (0.67)	0.978 (0.961–0.987)	2.10 (0.68)	2.18 (0.75)	0.867 (0.766–0.925)
7	1.64 (0.48)	1.64 (0.48)	1.00 (1.00–1.00)	1.64 (0.48)	1.64 (0.48)	0.955 (0.920–0.974)
8	3.36 (0.87)	3.34 (0.89)	0.994 (0.989–0.996)	3.36 (0.87)	3.34 (0.89)	0.980 (0.965–0.989)
9	1.24 (0.52)	1.22 (0.54)	0.982 (0.968–0.990)	1.24 (0.52)	1.28 (0.50)	0.916 (0.851–0.952)
10	1.28 (0.45)	1.30 (0.54)	0.887 (0.801–0.936)	1.28 (0.45)	1.30 (0.50)	0.876 (0.782–0.930)
11	1.60 (0.61)	1.60 (0.67)	0.857 (0.748–0.919)	1.60 (0.61)	1.54 (0.61)	0.745 (0.550–0.855)
12	1.98 (0.59)	2.40 (0.90)	0.785 (0.621–0.878)	1.98 (0.59)	1.94 (0.43)	0.700 (0.471–0.830)
Tot	26.76 (4.63)	27.14 (5.14)	0.986 (0.976–0.992)	26.76 (4.63)	27.12 (5.11)	0.987 (0.978–0.993)

BBS-12: Berg Balance Scale 12 Items; ICC: intraclass correlation coefficient.

The Pearson Correlation Coefficient analysis showed good linear correlation with the Tinetti (p<0.01) and with the Sport (p<0.01) and Home (p<0.01) subscales of the PASE-I. Values are synthetized in [Table 4].

Table 4 Results for the concurrent validity: Pearson correlation coefficient.

	BBS12	Tinetti	PASE-I TOT	PASE-I SPORT	PASE-I HOME	PASE-I WORK
BBS-12	1	0.817**	0.273	0.967**	0.922**	0.021
Tinetti		1	0.266	0.796**	0.755 **	0.129
PASE-I TOT			1	0.285*	0.213	0.382*
PASE-I SPORT				1	0.884*	0.100
PASE-I HOME					1	0.005
PASE-I WORK						1

BBS12: Berg Balance Scale 12-items; Tinetti; PASE-I: Italian version of the Physical Activity Scale for Elderly; **p<0.01; *p<0.05.

DISCUSSION

The study investigated the psychometric properties of the BBS-12 (19) in a PD sample. Internal consistency analysis revealed an alpha coefficient of 0.866 (0.872-0.889), slightly lower than that of the original study (0.972)[19]. Internal consistency measures whether several items that propose to measure the same general construct produce similar scores. Our findings demonstrated a good internal consistency of the scale.

The reliability study showed high significant values for both inter-observer (0.987) and intra-observer (0.986) reproducibility. Values of the BBS-12 indicate high stability over the time and between raters, as in the original version[19].

As expected, a strong relationship (0.817) of the BBS-12 with the Tinetti score was found (p<0.01). This happens because both measure balance abilities. On the other hand, this correlation was not found for PASE-I. Despite results having shown a correlation with Sport and Home sub-scales of PASE-I, the total score is probably influenced by a very poor correlation (0.021) with the Work sub-scale.

Although these are encouraging results, the present study has some limitations. In fact, the absence of similar studies that use the BBS-12 hinder comparisons. A second limit is due to the small sample size, which does not allow understanding differences between people with heterogeneous levels of impairment. It would be useful to investigate how the BBS-12 works at different stages of PD.

In conclusion, our finding demonstrates preliminary evidence on validity and reliability of the BBS-12 in PD population. Now, Italian healthcare professionals can use it with more confidence.

Conflicts of interest:

There is no conflict of interest to declare.

• References

• 1 Ray Dorsey E, Elbaz A, Nichols E, Abd-Allah F, Abdelalim A, Adsuar JC, et al. Global, regional, and national burden of Parkinson’s disease, 1990-2016: a systematic analysis for the Global Burden of Disease Study 2016. Lancet Neurol. 2018 Nov;17(11):939-53. https://doi.org/10.1007/s10072-018-3273-x
• 2 Valent F, Devigili G, Rinaldo S, Del Zotto S, Tullio A, Eleopra R. The epidemiology of Parkinson’s disease in the Italian region Friuli Venezia Giulia: a population-based study with administrative data. Neurol Sci. 2018 Apr;39(4):699-704. https://doi.org/10.1007/s10072-018-3273-x
• 3 Baldacci F, Policardo L, Rossi S, Ulivelli M, Ramat S, Grassi E, et al. Reliability of administrative data for the identification of Parkinson’s disease cohorts. Neurol Sci. 2015 May;36(5):783-6. https://doi.org/10.1007/s10072-015-2062-z
• 4 Pringsheim T, Jette N, Frolkis A, Steeves TD. The prevalence of Parkinson’s disease: a systematic review and meta-analysis. Mov Disord. 2014 Nov;29(13):1583-90. https://doi.org/10.1002/mds.25945
• 5 Carpenter MG, Allum JHJ, Honegger F, Adkin AL, Bloem BR. Postural abnormalities to multidirectional stance perturbations in Parkinson’s disease. J Neurol Neurosurg Psychiatry. 2004 Sep;75(9):1245-54. https://doi.org/10.1136/jnnp.2003.021147
• 6 Kerr GK, Worringham CJ, Cole MH, Lacherez PF, Wood JM, Silburn PA. Predictors of future falls in Parkinson disease. Neurology. 2010 Jul 13;75(2):116-24. https://doi.org/10.1212/WNL.0b013e3181e7b688
• 7 Pickering RM, Grimbergen YAM, Rigney U, Ashburn A, Mazibrada G, Wood B, et al. A meta-analysis of six prospective studies of falling in Parkinson’s disease. Mov Disord. 2007 Oct;22(13):1892-900. https://doi.org/10.1002/mds.21598
• 8 Galeoto G, Colalelli F, Massai P, Berardi A, Tofani M, Pierantozzi M, et al. Quality of life in Parkinson’s disease: Italian validation of the Parkinson’s Disease Questionnaire (PDQ-39-IT). Neurol Sci. 2018 Nov;39(11):1903-9. https://doi.org/10.1007/s10072-018-3524-x
• 9 Cova I, Di Battista ME, Vanacore N, Papi CP, Alampi G, Rubino A, et al. Validation of the Italian version of the Non Motor Symptoms Scale for Parkinson’s disease. Parkinsonism Relat Disord. 2017 Jan;34:38-42. https://doi.org/10.1016/j.parkreldis.2016.10.020
• 10 Santangelo G, Barone P, Abbruzzese G, Ferini-Strambi L, Antonini A. Validation of the Italian version of Parkinson’s Disease-Cognitive Rating Scale (PD-CRS). Neurol Sci. 2014 Apr;35(4):537-44. https://doi.org/10.1007/s10072-013-1538-y
• 11 Tambasco N, Simoni S, Eusebi P, Ripandelli F, Brahimi E, Sacchini E, et al. The validation of an Italian version of the Freezing of Gait Questionnaire. Neurol Sci. 2015 May;36(5):759-64. https://doi.org/10.1007/s10072-014-2037-5
• 12 Massai P, Colalelli F, Sansoni J, Valente D, Tofani M, Fabbrini G, et al. Reliability and validity of the Geriatric Depression Scale in Italian subjects with Parkinson’s disease. Parkinsons Dis. 2018 Aug 1;2018:7347859. https://doi.org/ 10.1155/2018/7347859
• 13 Berg KO, Wood-Dauphinee SL, Williams JI, Maki B. Measuring balance in the elderly: validation of an instrument. Can J Public Health. 1992 Jul-Aug;83 Suppl 2:S7-11.
• 14 Scalzo PL, Nova IC, Perracini MR, Sacramento DRC, Cardoso F, Ferraz HB, et al. Validation of the Brazilian version of the berg balance scale for patients with Parkinson’s disease. Arq Neuropsiquiatr. 2009 Sep;67(3B):831-5. https://doi.org/10.1590/s0004-282x2009000500010
• 15 Babaei-Ghazani A, Mohammadi H, Shahidi GA, Habibi SAH, Forogh B, Ahadi T, et al. Reliability and validity of the Persian translation of Berg Balance Scale in Parkinson disease. Aging Clin Exp Res. 2017 Oct;29(5):857-862. https://doi.org/10.1007/s40520-016-0682-7
• 16 Qutubuddin AA, Pegg PO, Cifu DX, Brown R, McNamee S, Carne W. Validating the Berg Balance Scale for patients with Parkinson’s disease: a key to rehabilitation evaluation. Arch Phys Med Rehabil. 2005 Apr;86(4):789-92. https://doi.org/10.1016/j.apmr.2004.11.005
• 17 Ottonello M, Ferriero G, Benevolo E, Sessarego P, Dughi D. Psychometric evaluation of the Italian version of the Berg Balance Scale in rehabilitation inpatients. Eura Medicophys. 2003;39(4):181-9.
• 18 Franchignoni F, Velozo CA, Qutubuddin AA, Cifu DX, Carne W. Use of the Berg Balance Scale in rehabilitation evaluation of patients with Parkinson’s disease [2] (multiple letters). Arch Phys Med Rehabil. 2005 Nov;86(11):2225-6. https://doi.org/10.1016/j.apmr.2005.09.006
• 19 La Porta F, Caselli S, Susassi S, Cavallini P, Tennant A, Franceschini M. Is the berg balance scale an internally valid and reliable measure of balance across different etiologies in neurorehabilitation? A revisited rasch analysis study. Arch Phys Med Rehabil. 2012 Jul;93(7):1209-16. https://doi.org/10.1016/j.apmr.2012.02.020
• 20 Galeoto G, Colucci M, Guarino D, Esposito G, Cosma E, De Santis R, et al. Exploring validity, reliability, and factor analysis of the Quebec user evaluation of satisfaction with assistive technology in an Italian population: a cross-sectional study. Occup Ther Health Care. 2018 Oct;32(4):380-92. https://doi.org/10.1080/07380577.2018.1522682
• 21 Dattoli S, Colucci M, Soave MG, De Santis R, Segaletti L, Corsi C, et al. Evaluation of pelvis postural systems in spinal cord injury patients: outcome research. J Spinal Cord Med. 2020;43(2):185-92. https://doi.org/ 10.1080/10790268.2018.1456768
• 22 Savona A, Ferralis L, Saffioti M, Tofani M, Nobilia M, Culicchia G, et al. Evaluation of intra- and inter-rater reliability and concurrent validity of the Italian version of the Jebsen-Taylor Hand Function Test in adults with rheumatoid arthritis. Hand Ther. 2019;24(2):48-54. https://doi.org/10.1177/1758998319843554
• 23 Berardi A, Biondillo A, Màrquez MA, De Santis R, Fabbrini G, Tofani M, et al. Validation of the short version of the Van Lieshout Test in an Italian population with cervical spinal cord injuries: a cross-sectional study. Spinal Cord. 2019 Apr;57:339-45. https://doi.org/10.1038/s41393-018-0226-4
• 24 Litvan I, Bhatia KP, Burn DJ, Goetz CG, Lang AE, McKeith I, et al. Movement disorders society scientific issues committee report: SIC task force appraisal of clinical diagnostic criteria for Parkinsonian disorders. Mov Disord. 2003 May;18(5):467-86. https://doi.org/10.1002/mds.10459
• 25 Washburn RA, Smith KW, Jette AM, Janney CA. The physical activity scale for the elderly (PASE): development and evaluation. J Clin Epidemiol. 1993 Feb;46(2):153-62. https://doi.org/10.1016/0895-4356(93)90053-4
• 26 Washburn RA, McAuley E, Katula J, Mihalko SL, Boileau RA. The Physical Activity Scale for the Elderly (PASE): Evidence for validity. J Clin Epidemiol. 1999 Jul;52(7):643-51. https://doi.org/10.1016/s0895-4356(99)00049-9
• 27 Covotta A, Gagliardi M, Berardi A, Maggi G, Pierelli F, Mollica R, et al. Physical activity scale for the elderly: Translation, cultural adaptation, and validation of the Italian version. Curr Gerontol Geriatr Res. 2018 Aug;2018:8294568. https://doi.org/10.1155/2018/8294568
• 28 Tinetti ME. Performance-oriented assessment of mobility problems in elderly patients. J Am Geriatr Soc. 1986 Feb;34(2):119-26. https://doi.org/10.1111/j.1532-5415.1986.tb05480.x
• 29 Nunnally JC. Psychometric theory. Psychometric theory. New York: McGraw-Hill, 1979.
• 30 Hallgren KA. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor Quant Methods Psychol. 2012;8(1):23-34. https://doi.org/10.20982/tqmp.08.1.p023
• 31 Ratner B. The correlation coefficient: Its values range between +1/−1, or do they? J Targeting, Meas Anal Mark. 2009;17(2):139-42. https://doi.org/10.1057/jt.2009.5

Publication History

Received: 21 January 2020

Accepted: 26 February 2020

Article published online:
13 June 2023