Kappa statistic considerations in evaluating inter-rater reliability between two raters: which, when and context matters

Li, Ming; Gao, Qian; Yu, Tianfei

doi:10.1186/s12885-023-11325-z

Cited by 10 publications

(8 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In some cases, the level of agreement might appear low using the κ statistic due to the high prevalence of a particular value for a variable. This is known as the κ paradox [73][74][75][76]. Therefore, we will also consider the level of agreement acceptable if κ is ≤0.6, but crude agreement is ≥85%.…”

Section: Discussionmentioning

confidence: 99%

Social Media Promotion of Health Tests With Potential for Overdiagnosis or Overuse: Protocol for a Content Analysis

Nickel,

Heiss,

Shih

et al. 2024

JMIR Res Protoc

View full text Add to dashboard Cite

Background In recent years, social media have emerged as important spaces for commercial marketing of health tests, which can be used for the screening and diagnosis of otherwise generally healthy people. However, little is known about how health tests are promoted on social media, whether the information provided is accurate and balanced, and if there is transparency around conflicts of interest. Objective This study aims to understand and quantify how social media is being used to discuss or promote health tests with the potential for overdiagnosis or overuse to generally healthy people. Methods Content analysis of social media posts on the anti-Mullerian hormone test, whole-body magnetic resonance imaging scan, multicancer early detection, testosterone test, and gut microbe test from influential international social media accounts on Instagram and TikTok. The 5 tests have been identified as having the following criteria: (1) there are evidence-based concerns about overdiagnosis or overuse, (2) there is evidence or concerns that the results of tests do not lead to improved health outcomes for generally healthy people and may cause harm or waste, and (3) the tests are being promoted on social media to generally healthy people. English language text-only posts, images, infographics, articles, recorded videos including reels, and audio-only posts are included. Posts from accounts with <1000 followers as well as stories, live videos, and non-English posts are excluded. Using keywords related to the test, the top posts were searched and screened until there were 100 eligible posts from each platform for each test (total of 1000 posts). Data from the caption, video, and on-screen text are being summarized and extracted into a Microsoft Excel (Microsoft Corporation) spreadsheet and included in the analysis. The analysis will take a combined inductive approach when generating key themes and a deductive approach using a prespecified framework. Quantitative data will be analyzed in Stata SE (version 18.0; Stata Corp). Results Data on Instagram and TikTok have been searched and screened. Analysis has now commenced. The findings will be disseminated via publications in peer-reviewed international medical journals and will also be presented at national and international conferences in late 2024 and 2025. Conclusions This study will contribute to the limited evidence base on the nature of the relationship between social media and the problems of overdiagnosis and overuse of health care services. This understanding is essential to develop strategies to mitigate potential harm and plan solutions, with the aim of helping to protect members of the public from being marketed low-value tests, becoming patients unnecessarily, and taking resources away from genuine needs within the health system. International Registered Report Identifier (IRRID) DERR1-10.2196/56899

show abstract

Section: Discussionmentioning

confidence: 99%

Social Media Promotion of Health Tests With Potential for Overdiagnosis or Overuse: Protocol for a Content Analysis

Nickel,

Heiss,

Shih

et al. 2024

JMIR Res Protoc

View full text Add to dashboard Cite

show abstract

“…For interrater agreement at the dimension level, the quadratic weighted kappa was used [ 32 ]. Kappa values can range from − 1 to + 1, with positive (negative) values reflecting the strength of agreement (disagreement).…”

Section: Methodsmentioning

confidence: 99%

Examining interrater agreement between self-report and proxy-report responses for the quality of life-aged care consumers (QOL-ACC) instrument

Hutchinson,

Khadka,

Crocker

et al. 2024

J Patient Rep Outcomes

View full text Add to dashboard Cite

Background Quality of life is an important quality indicator for health and aged care sectors. However, self-reporting of quality of life is not always possible given the relatively high prevalence of cognitive impairment amongst older people, hence proxy reporting is often utilised as the default option. Internationally, there is little evidence on the impact of proxy perspective on interrater agreement between self and proxy report. Objectives To assess the impacts of (i) cognition level and (ii) proxy perspective on interrater agreement using a utility instrument, the Quality of Life-Aged Care Consumers (QOL-ACC). Methods A cross-sectional study was undertaken with aged care residents and family member proxies. Residents completed the self-report QOL-ACC, while proxies completed two proxy versions: proxy-proxy perspective (their own opinion), and proxy-person perspective (how they believe the resident would respond). Interrater agreement was assessed using quadratic weighted kappas for dimension-level data and concordance correlation coefficients and Bland-Altman plots for utility scores. Results Sixty-three residents (22, no cognitive impairment; 41, mild-to-moderate cognitive impairment) and proxies participated. In the full sample and in the mild-to-moderate impairment group, the mean self-reported QOL-ACC utility score was significantly higher than the means reported by proxies, regardless of perspective (p < 0.01). Agreement with self-reported QOL-ACC utility scores was higher when proxies adopted a proxy-person perspective. Conclusion Regardless of cognition level and proxy perspective, proxies tend to rate quality of life lower than residents. Further research is needed to explore the impact of such divergences for quality assessment and economic evaluation in aged care.

show abstract

“…The value ranged from 0.80 to 0.90, which was classified as a strong level of agreement. 16 Data collection of variables was carried out using Cognito, an electronic application, in Puteran village in Pager Ageung district, Tasikmalaya Regent, West Java, Indonesia, in November 2022. The severity of caries data was analysed using the PUFA formulas and the PUFA index, and it is presented in the table of frequency distribution.…”

Section: Methodsmentioning

confidence: 99%

The severity of untreated caries in the deciduous and permanent teeth among children aged 6-12 years using PUFA and pufa index: a cross-sectional study

Suwargiani,

Aripin,

Susilawati

et al. 2024

Padjadjaran J Dent

View full text Add to dashboard Cite

Introduction: The PUFA Index and pufa are used to measure the severity of dental and oral conditionwhich are not commonly used due to its novelty, and this index can be used to measure untreated dental caries7 and caries activity. The aims of study are to assess the severity of untreated caries in children’s deciduous and permanent teeth by using the PUFA and pufa index. Methods: An observational study using a cross-sectional study design was conducted among children who lived in Puteran village in Pager Ageung district, Tasikmalaya Regent, West Java, Indonesia. Purposive sampling was used, and the children who met the inclusion and exclusion criteria were invited to participate in the study. Severity of caries was assessed using the pufa and PUFA index for deciduous and permanent teeth, respectively. Severity of caries data were analysed using formulas for pufa and PUFA index. Results: A total of 206 children participated in the study. The PUFA assessment showed the severity of caries was high in the deciduous dentition with pufa index of 3.5 with SD 1,9-3,6 However, the PUFA assessment showed a relatively lower severity of caries in permanent teeth, with index PUFA of 0.5 with SD 1,9-3,6. The severity of caries was higher in the maxillary than the mandibular teeth in deciduous teeth. On the other hand, the severity of caries was higher in the mandibular than maxillary teeth in the permanent teeth. Conclusions: Each child aged 6 and 12 years in Puteran village had severe untreated caries in one permanent tooth and severe untreated caries in 3 to 4 of deciduous teeth. The findings of the study will be used to develop caries prevention and oral health promotion programs.KEYWORDS Caries, Severity, Deciduous, Permanent, teeth, PUFA, PUFA index, children

show abstract

Kappa statistic considerations in evaluating inter-rater reliability between two raters: which, when and context matters

Cited by 10 publications

References 15 publications

Social Media Promotion of Health Tests With Potential for Overdiagnosis or Overuse: Protocol for a Content Analysis

Social Media Promotion of Health Tests With Potential for Overdiagnosis or Overuse: Protocol for a Content Analysis

Examining interrater agreement between self-report and proxy-report responses for the quality of life-aged care consumers (QOL-ACC) instrument

The severity of untreated caries in the deciduous and permanent teeth among children aged 6-12 years using PUFA and pufa index: a cross-sectional study

Contact Info

Product

Resources

About