Evaluation of the Modified Naranjo Criteria for Assessing Causal Attribution of Clinical Outcome to Homeopathic Intervention as Presented in Case Reports

Abstract

Objectives The objective of this study was to establish the reliability and content validity of the “Modified Naranjo Criteria for Homeopathy—Causal Attribution Inventory” as a tool for attributing a causal relationship between the homeopathic intervention and outcome in clinical case reports.

Methods Purposive sampling was adopted for the selection of information-rich case reports using pre-defined criteria. Eligible case reports had to fulfil a minimum of nine items of the CARE Clinical Case Reporting Guideline checklist and a minimum of three of the homeopathic HOM-CASE CARE extension items. The Modified Naranjo Criteria for Homeopathy Inventory consists of 10 domains. Inter-rater agreement in the scoring of these domains was determined by calculating the percentage agreement and kappa (κ) values. A κ greater than 0.4, indicating fair agreement between raters, in conjunction with the absence of concerns regarding the face validity, was taken to indicate the validity of a given domain. Each domain was assessed by four raters for the selected case reports.

Results Sixty case reports met the inclusion criteria. Inter-rater agreement/concordance per domain was “perfect” for domains 1 (100%, κ = 1.00) and 2 (100%, κ = 1.00); “almost perfect” for domain 8 (97.5%, κ = 0.86); “substantial” for domains 3 (96.7%, κ = 0.80) and 5 (91.1%, κ = 0.70); “moderate” for domains 4 (83.3%, κ = 0.60), 7 (67.8%, κ = 0.46) and 9 (99.2%, κ = 0.50); and “fair” for domain 10 (56.1%, κ = 0.38). For domains 6A (46.7%, κ = 0.03) and 6B (50.3%, κ = 0.18), there was “slight agreement” only. Thus, the validity of the Modified Naranjo Criteria for Homeopathy tool was established for each of its domains, except for the two that pertain to direction of cure (domains 6A and 6B).

Conclusion The Modified Naranjo Criteria for Homeopathy—Causal Attribution Inventory was identified as a valid tool for assessing the likelihood of a causal relationship between a homeopathic intervention and clinical outcome. Improved wordings for several criteria have been proposed for the assessment tool, under the new acronym “MONARCH”. Further assessment of two MONARCH domains is required.

Keywords

validation - Modified Naranjo Criteria - causal relationship - homeopathic intervention - clinical outcome - case reports

Introduction

Clinical case reports have a time-honored and rich tradition in medicine and science.[1] Their significance for research and educational purposes is still thriving[2] as it allows in-depth, multi-faceted exploration of complex issues in their real-life settings. Clinical case reports can, for instance, be considered for review purposes when an experimental design is either inappropriate to answer the research questions posed or impossible to undertake. Research of case reports, if carefully conceptualized and thoughtfully undertaken and reported, can yield powerful insights into many important aspects of healthcare delivery.[3]

Case reports are often insufficiently rigorous to be combined for data analysis, inform research design, or guide clinical practice. To improve the transparency and quality of clinical case reports, Gagnier et al developed the CARE reporting guideline for use by healthcare stakeholders around the world.[4] Case reports, when written as per reporting guidelines, have the potential to offer evidence from the point of care that can be useful for clinical research, inform clinical practice guidelines, and improve medical education. Though clinical case reports are justifiably graded as weak evidence for assessing causal relationships, they are often the first “signals” of potential adverse drug reactions, and sometimes even single cases can contribute significantly to establishing a causal link.[5]

In modern medicine, three approaches are used to assess the causal relationship between drug treatment and the occurrence of adverse events: expert judgement (global introspection), probabilistic approaches, and algorithms.[6] The Naranjo Criteria algorithm is one that has been utilized to classify the probability that an adverse event is related to drug therapy based on a list of weighted questions.[7] The Modified Naranjo Criteria for Homeopathy (Causal Attribution Inventory) tool, originally adapted by Rutten and further developed over several years by the Clinical Data Working Group of the Homoeopathic Pharmacopoeia Convention of the United States (HPCUS), is proposed to assess the likelihood of causal attribution of the clinical outcome to the medicine(s) prescribed in homeopathic cases and case reports.[8] [9] [10]

When homeopathy is tested in clinical trials, understanding and appraisal are likely to be improved if published reports contain details of prescribing strategies and treatment as indicated in the homeopathy-specific extension (RedHot) of the CONSORT guidelines.[11] Likewise, a homeopathy extension of the CARE clinical case reporting guideline (HOM-CASE) recommends use of the Modified Naranjo Criteria for Homeopathy, which enables assessment of the likelihood of assigning causal relationship between a homeopathic intervention and a clinical improvement.[10] Although randomized controlled trials (RCTs) are considered the gold standard for establishing causality, a pool of good-quality case reports published using HOM-CASE CARE guidelines would offer an important additional contribution to knowledge.

The present study was undertaken to assess the reliability and validity of the currently formulated Modified Naranjo Criteria for Homeopathy,[10] and proposes improved domain-by-domain wording based on the findings.

Methods

A purposive sampling strategy, a non-probability sampling wherein the selection process involves identifying themes, concepts, and indicators through observation and reflection,[12] was adopted to identify information-rich cases from accessible publications. Published case reports were searched in the Central Council for Research in Homoeopathy (CCRH) Library (Janakpuri, New Delhi, India) and on the Web. Case reports were identified using the following criteria: single case report, published in Medline-listed or non-indexed journal or as a dissertation.

Preliminary evaluation of identified case reports as per HOM-CASE CARE guidelines indicated that most of the cases covered around 9 items out of 30 on the generic CARE checklist (all domains and sub-domains numbered from 1 to 30) and a minimum of 3 out of 6 as per the HOM-CASE extension items (main and the sub-domains numbered from 1 to 6, except domain 10h3, which is the possible causal attribution of changes explicitly assessed/discussed). Therefore, these criteria were chosen as a minimum threshold for inclusion of cases. Case reports with poorly described prescribing symptoms, and homeopathic patent medicines or compound formulations or proprietary products or combinations where more than one medicine was administered simultaneously, were excluded.

The eligible case reports were independently evaluated as per the Modified Naranjo Criteria for Homeopathy ([Table 1]) by four raters (VKG, CDL, NM, and AMM) using a specifically designed electronic Case Recording Format. Their evaluations were sent to LR and RvH for compilation and blinding of the raters during analysis, which was done to prevent bias. During analysis, the face validity was assessed with a view to determining if the items of each domain were sensible, appropriate, and relevant.

Table 1 Modified Naranjo Criteria

Domains	Yes	No	Not sure or N/A
1. Was there an improvement in the main symptom or condition for which the homeopathic medicine was prescribed?	+2	–1	0
2. Did the clinical improvement occur within a plausible timeframe relative to the drug intake?	+1	–2	0
3. Was there an initial aggravation of symptoms?	+1	0	0
4. Did the effect encompass more than the main symptom or condition (i.e., were other symptoms ultimately improved or changed)?	+1	0	0
5. Did overall well-being improve? (suggest using validated scale)	+1	0	0
6A Direction of cure: did some symptoms improve in the opposite order of the development of symptoms of the disease?	+1	0	0
6B Direction of cure: did at least two of the following aspects apply to the order of improvement of symptoms:  –from organs of more importance to those of less importance?  –from deeper to more superficial aspects of the individual?  –from the top downwards?	+1	0	0
7. Did “old symptoms” (defined as non-seasonal and non-cyclical symptoms that were previously thought to have resolved) reappear temporarily during the course of improvement?	+1	0	0
8. Are there alternate causes (other than the medicine) that—with a high probability—could have caused the improvement? (Consider known course of disease, other forms of treatment, and other clinically relevant interventions)	–3	+1	0
9. Was the health improvement confirmed by any objective evidence? (e.g., laboratory test, clinical observation, etc.)	+2	0	0
10. Did repeat dosing, if conducted, create similar clinical improvement?	+1	0	0

Note: Maximum score = 13, minimum score = −6.

The analysis was done by CDL and RS. Thereafter, LR and RvH helped in reaching agreement/consensus among raters.

The study was conducted in three phases as reflected in the study flowchart ([Fig. 1]). Inter-rater agreement for each domain was assessed via calculation and analysis of the kappa value (κ) for nominal (i.e., categorical) variables. A measure is said to be reliable if it gives the same results under consistent conditions. Hence, the Modified Naranjo Criteria for Homeopathy algorithm was tested for reliability, and thus consistency, by assessing inter-rater agreement by evaluating the ratings for each domain given by four raters. The κ value enables an assessment of the level of agreement “over and above chance” between different raters.

Fig. 1 Study flowchart. Domain 10h3: CARE guidelines domain 10, HOM-CASE extension3.

Kappa values[13] and percentage for inter-rater agreement were calculated based on 3-by-3 contingency tables for the responses received from the raters under three categories: namely, “Yes”, “No”, and “Not sure or Not applicable (N/A)”. The analysis was done using Statistical Package for Social Sciences (SPSS).

In the present study, as there were more than two raters, percentage agreement for all possible combinations of raters was calculated, and subsequently the mean level of agreement across all four raters was calculated. The κ values[14] were interpreted as follows: κ < 0, “less than chance agreement”; κ 0.01 to 0.20, “slight agreement”; κ 0.21 to 0.40, “fair agreement”; κ 0.41 to 0.60, “moderate agreement”; κ 0.61 to 0.80, “substantial agreement”; κ 0.81 to 0.99, “almost perfect agreement”. A minimum of “fair agreement”, in the absence of concerns with regard to the face validity of the item, was taken as an indicator of validity of a given domain.

Results

Sixty of 80 case reports fulfilled the eligibility criteria ([Fig. 1]). Details of the 60 reports are listed in [Supplementary Table 1] (online only).

Domain-wise mean percentage agreement of four raters and κ calculation for the Modified Naranjo Criteria domains were evaluated in the case reports. Inter-rater agreement for each domain was as follows ([Table 2]): “perfect” for domains 1 (100%, κ = 1.00) and 2 (100%, κ = 1.00); “almost perfect” for domain 8 (97.5%, κ = 0.86); “substantial” for domains 3 (96.7%, κ = 0.80) and 5 (91.1%, κ = 0.70); “moderate” for domains 4 (83.3%, κ = 0.60), 7 (67.8%, κ = 0.46) and 9 (99.2%, κ = 0.50); and “fair” for domain 10 (56.1%, κ = 0.38). For domains 6A (46.7%, κ = 0.03) and 6B (50.3%, κ = 0.18), there was “slight agreement” only. Thus, the algorithm was found to be valid for each domain except 6A and 6B ([Table 2]).

Table 2 Domain-wise mean percentage agreement and kappa calculation between Modified Naranjo Criteria for Homeopathy domains and evaluated case reports

Domain	Agreement (%)	Kappa (κ)	Inference	Valid (κ > 0.40)
1	100	1.00	Perfect agreement	Yes
2	100	1.00	Perfect agreement	Yes
3	96.7	0.80	Substantial agreement	Yes
4	83.3	0.60	Moderate agreement	Yes
5	91.1	0.70	Substantial agreement	Yes
6A	46.7	0.03	Slight agreement	No
6B	50.3	0.18	Slight agreement	No
7	67.8	0.46	Moderate agreement	Yes
8	97.5	0.86	Almost perfect agreement	Yes
9	99.2	0.50	Moderate agreement	Yes
10	56.1	0.38	Fair agreement	Yes

Table 3 MONARCH Inventory (improved version of the Modified Naranjo Criteria for Homeopathy)

Domains
1	Was there an improvement in the main symptom or condition for which the homeopathic medicine was prescribed?
2	Did the clinical improvement occur within a plausible timeframe relative to the medicine intake?
3	Was there a homeopathic aggravation of symptoms?
4	Did the effect encompass more than the main symptom or condition (i.e., were other symptoms, not related to the main presenting complaint, improved or changed)?
5	Did overall well-being improve? (Suggest using a validated scale or mention about changes in physical, emotional, and behavioral elements)
6A	Direction of cure: did some symptoms improve in the opposite order of the development of symptoms of the disease?
6B	Direction of cure: did at least one of the following aspects apply to the order of improvement in symptoms: –from organs of more importance to those of less importance? –from deeper to more superficial aspects of the individual? –from the top downwards?
7	Did “old symptoms” (defined as non-seasonal and non-cyclical symptoms that were previously thought to have resolved) reappear temporarily during the course of improvement?
8	Are there alternative causes (i.e., other than the medicine) that—with a high probability—could have produced the improvement? (Consider known course of disease, other forms of treatment, and other clinically relevant interventions)
9	Was the health improvement confirmed by any objective evidence? (e.g., investigations, clinical examination, etc.)
10	Did repeat dosing, if conducted, create similar clinical improvement?

Notes: Updated wording shown in bold (see domains 3, 4, 5, 6B and 9). Improvements have also been made to the wording of domains 2 and 8. The scores per domain are the same as for [Table 1].

Discussion

A clinical case report is considered a weak level of evidence for establishing causal relationship. But keeping in view the highly individualistic approach of homeopathy, conceptual difficulties with RCTs in the context of homeopathy and the resources involved, we need to strengthen the reporting of case records. Tools to attribute the likelihood of causal relationship that are aligned with the principles of homeopathy are important in this regard.

The results of this study indicate that the Modified Naranjo algorithm is generally feasible as a tool for assessing causality of homeopathic treatment, as there is, after some instruction, good consensus about the qualifications of the domains. There is, however, an exception for items 6A and 6B.

Findings and Recommendations Regarding the Modified Naranjo Criteria for Homeopathy Domains

As successfully treated cases were selected, there was no ambiguity and a perfect agreement among raters for domains 1 and 2 of the Modified Naranjo Criteria: therefore, these items do not need modification or elaboration. In domain 3, initial aggravation should clearly be attributable to homeopathy, because an aggravation may also be due to a conventional drug or a disease. Also, it was reported in a few cases only, where the raters largely agreed. The authors of case reports are encouraged to provide information on the incidence of homeopathic aggravations ([Table 3]).

In assessing domain 4, some of the evaluators thought the description included minor symptoms of the main condition, whereas some considered different symptoms not related to the main condition. Therefore, the question should be re-phrased to “Did the effect encompass more than the main symptom or condition (i.e., were other symptoms, not related to the main presenting complaint, improved or changed)?” ([Table 3]). Any symptoms not related to the main presenting symptom or condition are to be considered for assessing this domain.

Domain 5 was interpreted in different ways by the raters as most of the cases did not use any validated scales for the assessment of general well-being. Improvement in general condition or in associated complaints or other symptoms was considered as overall improvement by the raters. Hence, multiple assessment options led to similar conclusions. It is therefore recommended, if possible, to use a validated Quality of Life scale that is either generic or specific for a given disease condition. A note about improvement in physical condition, emotions, and behavior should be included to help judge the overall well-being ([Table 3]).

Domains 6A and 6B, pertaining to direction of cure, cannot be considered as validated due to insufficient reporting in the assessed case reports. The raters assessed domains 6A and 6B in most of the cases as “Not sure” or “Not applicable”. Therefore, only a few cases remained, with considerable disagreement. Direction of cure is an important aspect but either it was not observed, or it was inadequately reported in the analyzed case reports. Also, there was no consensus among raters whether it was reported or not and the raters principally agreed on the “Not sure” assessment. Domain 6A, which specifically assesses improvement in the opposite order to the development of symptoms of the disease, created the greatest difficulty in interpretation. In one case report, it was actually stated that the symptoms improved in the opposite order, and here there was consensus among evaluators. Otherwise, there was no consensus. The chronology of development and improvement in the symptoms should be mentioned explicitly by case-report authors so that the reader can better assess their sequence of onset.

Pertaining to domain 6B, in three case reports, observance of Hering's Law (which determines the order of symptom improvement) was mentioned by the original authors, on which basis the evaluators rated as “Yes” in this domain for its observance, but on discussion it was found that all three cases had mentioned only one of the three aspects, whereas at least two are required. Therefore, evaluators agreed that they were not sure. This finding, together with the greater difficulty of basing a positive score on varying combinations of two out of three items, leads us to recommend attributing a positive score to this item if at least one of the three aspects is applicable ([Table 3]). Lowering the threshold from “at least two” to “at least one” of these aspects gives these relatively uncommon but important items a greater “weight”.

A more general conclusion regarding domain 6 is that referring to the direction of cure is generally neglected in homeopathy case reports. This is a surprising finding given that the direction of cure is deemed to be important in homeopathy. It would be useful to look further into the reasons for our findings, and to communicate to the homeopathy community that this aspect should receive further attention when writing up and assessing clinical cases. Also, one of the reasons for non-reporting could be a lack of clear definitions: the terms “from organs of more importance to those of less importance”, “deeper to more superficial” and “from the top downwards” could usefully be further defined within domain 6B.

Regarding domain 7, there was no difference of opinion among raters in cases where an author had clearly mentioned that the old symptoms reappeared, but in other cases there was difference of opinion, subject to interpretation. Also, the raters agreed that old symptoms were not reported in most cases. Therefore, the chronological sequence of old symptoms in which these reappear should be summarized in the case reports to enable uniformity in assessment.

In domain 8, there should be consideration of known course of disease, other forms of treatment, other clinically relevant interventions, lifestyle changes, etc. Any concurrent treatment should be mentioned in the case reports. This is to help further substantiate the causal relationship between the homeopathic intervention and outcome. In the cases evaluated, the high percentage agreement in the Naranjo algorithm was for the response “No” (and an associated score of +1), which is consistent with its validity in determining that alternative causes were explicitly considered and excluded.

In domain 9, clinical observation is mentioned as an example for any objective evidence, which may be replaced by investigations or relevant clinical examination or photographs (for dermatological conditions) or a validated questionnaire (especially for subjective conditions) for better assessment ([Table 3]). It was adequately covered in the cases studied.

Repeat dosing, as referred to in domain 10, was infrequently observed in evaluated cases and should be further defined. What is understood from this question is that it should not be assessed based on the repetition of medicine during routine follow-up. Instead, this item is only applicable when the disease has been absent or under remission for quite a long time and the disease, with similar symptoms, reappears and is improved with the previously selected medicine. Only then was the repeat dosing considered to establish reproducibility. Diseases with relapsing remitting course, for example multiple sclerosis and rheumatoid arthritis, can be assessed during a new episode of relapse if the symptoms corroborate with the previous episode by repeat dosing, rather than by assessing repetition of the medicine at subsequent follow-up visits during the same disease episode.

There was thus good overall inter-rater agreement in assessing clinical case reports using the Modified Naranjo Criteria as a tool for attributing the likelihood of a causal relationship between homeopathic intervention and clinical outcome in quality case reports. Therefore, except for the questions about direction of cure (domain 6), the reliability and thus the content validity of this Causal Attribution Inventory were largely established. As found in this study, information to inform domains 3, 7, and 10 is not observed frequently in case reports but, when reported, these domains are of value in establishing causal attribution. We may not find them in all successfully treated cases but even their absence should be specified by the original author to ensure unambiguous case reporting.

Although a thorough review of the international literature took place to identify the case reports, the sample is likely to be biased toward a sub-set of case reports predominantly from India. This, in turn, might be attributed to greater public acceptability and case reporting in that country. The case reports were assessed as per contemporary HOM-CASE CARE guidelines[10] for inclusion in the study. As per evaluation, most domains of these guidelines were covered to a variable degree across the cases; however, none of the 60 cases presented information related to intervention adherence, tolerability, or adverse and unanticipated events. Therefore, it is important to follow these guidelines for complete reporting and for making each case count in subsequent analysis.

It is a limitation of our study that, being an assessment of case reports that were published before 2018, most of the cases were written before publication of the CARE and HOM-CASE guidelines, reducing the likelihood of the reports' compliance with them. Therefore, the domains assessed as per HOM-CASE CARE were not well represented in these cases. However, it is expected that in future, when more cases are published as per the HOM-CASE guidelines, firmer conclusions regarding the likelihood of a causal relationship between intervention and outcome can be drawn.

Further work on domains 6A and 6B of the Modified Naranjo Criteria is needed with a view to improving their validity. Describing the pattern of symptom unfolding or improvement is imprecise, especially for long-standing illness. It relies on the vagaries of memory, which may be subject to all sorts of contamination. Also, domain 5 brings up the important point that much of the material here is based not on measurable evidence or validated scales (such as Quality of Life) but personal recollection and medical history taking. The concept of aggravation (pertaining to domain 3) is also somewhat uncertain. What homeopaths attribute to the actions of the remedy is often explained by their allopathic colleagues as either early and transient side effects of treatment or a peculiar susceptibility of the patient to certain symptoms/adverse events. As these domains are related to core areas of homeopathy, further work on defining these basic principles is imperative to include them optimally for assessing causal relationship.

Conclusion

The Modified Naranjo Criteria for Homeopathy—Causal Attribution Inventory has been identified as a useful tool for assessing the likelihood of a causal relationship between homeopathic intervention and clinical outcome. Except for items relating to “direction of cure” (domains 6A and 6B), the reliability and validity of all other domains was largely established.

Some improvements to the wording of several domains of the Modified Naranjo Criteria for Homeopathy are proposed: these are presented under the newly introduced acronym, “MONARCH” ([Table 3]). Further elaboration of the MONARCH domains 6A and 6B is needed to validate them as these are related to some basic principles of homeopathy. Also, further validation of the MONARCH Inventory via a formal assessment of content and construct validity based on a broad set of clinical case reports is recommended.

The overarching CARE/HOM-CASE guidelines should be followed for standardized and therefore more thorough case reporting.

Conflict of Interest

None declared.

Acknowledgments

The authors are grateful to Dr. Joyce Frye, Dr. Todd Hoover, Dr. David Riley and Dr. Peter Smith of the HPCUS Clinical Data Working Group for their contribution to the development of the Modified Naranjo Criteria that were used in this project, as well as to Dr. Raj K. Manchanda, Former Director General and Dr. Anil Khurana, Director General, CCRH for their support. Authors are also thankful to Dr. Gian Singh, Scientist F (Retd.), NISCAIR, CSIR, New Delhi, for guidance in manuscript preparation and its critical review. Dr. Radhika Agarwal, Senior Research Fellow, CCRH, New Delhi, is also acknowledged for her valuable suggestions on the manuscript. Gratitude is also expressed to Ms. Meenakshi Bhatia, Junior Librarian, CCRH, for library support.

Authors' Contributions

CDL, VKG, RvH, and LR conceptualized the study; CDL collected the data; CDL, VKG, NM and AMM evaluated the cases; RvH and LR compiled and blinded the data; CDL and RS analyzed and interpreted the data and drafted the manuscript. All authors reviewed and provided final approval of the version to be published.

Note

This study was PhD work of the corresponding author, performed under the experienced supervision of Padma Shri Dr. V K. Gupta, MD (Hom.). The experts (raters) were also MD (Hom.) with 10 years teaching/research experience.

Dr. Robbert van Haselen, Director, World Integrated Medicine Forum, R&D Consultancy, International Institute for Integrated Medicine (INTMEDI), United Kingdom and Dr. Lex Rutten, independent researcher, Netherlands, were the advisers in this study. Both coordinated the blinding of raters and helped in reaching agreement/consensus among them.

• References

• 1 Gopikrishna V. A report on case reports. J Conserv Dent 2010; 13: 265-271
• 2 Nissen T, Wynn R. The history of the case report: a selective review. JRSM Open 2014; 5: 2054270414523410
• 3 Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach. BMC Med Res Methodol 2011; 11: 100
• 4 Gagnier JJ, Kienle G, Altman DG, Moher D, Sox H, Riley D. CARE Group. The CARE guidelines: consensus-based clinical case reporting guideline development. BMJ Case Rep 2013; 2013: 38-43
• 5 van Haselen RA. Towards improving the reporting quality of clinical case reports in complementary medicine: assessing and illustrating the need for guideline development. Complement Ther Med 2015; 23: 141-148
• 6 Arimone Y, Miremont-Salamé G, Haramburu F. et al. Inter-expert agreement of seven criteria in causality assessment of adverse drug reactions. Br J Clin Pharmacol 2007; 64: 482-488
• 7 Srinivasan R, Ramya G. Adverse drug reaction — causality assessment. Int J Res Pharm Chem 2011; 1: 606-612
• 8 Rutten LA. The importance of case histories for accepting and improving homeopathy. Complement Ther Med 2013; 21: 565-570
• 9 Rutten L. Data collection: treat every variable as a treasure. Homeopathy 2015; 104: 190-196
• 10 van Haselen RA. Homeopathic clinical case reports: development of a supplement (HOM-CASE) to the CARE clinical case reporting guideline. Complement Ther Med 2016; 25: 78-85
• 11 Dean ME, Coulter MK, Fisher P, Jobst KA, Walach H. Reporting data on homeopathic treatments (RedHot): a supplement to CONSORT. J Altern Complement Med 2007; 13: 19-23
• 12 Robinson RS. Purposive sampling. In: Michalos AC. , ed. Encyclopedia of Quality of Life and Well-Being Research. Dordrecht: Springer; 2014
• 13 McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb) 2012; 22: 276-282
• 14 Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med 2005; 37: 360-363

Publication History

Received: 31 May 2019

Accepted: 14 November 2019

Article published online:
25 March 2020