On Assessing Interrater Agreement for Multiple Attribute Responses

Kupper, Lawrence L.; Hafner, Kerry B.

doi:10.2307/2531695

Cited by 37 publications

(23 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Inter-rater reliability (IRR) was computed using the KupperHafner statistic [32]. For the Choice question, we computed IRR for each pair of raters.…”

Section: Discussionmentioning

confidence: 99%

An Experience Sampling Study of User Reactions to Browser Warnings in the Field

Reeder

Felt

Consolvo

et al. 2018

Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Web browser warnings should help protect people from malware, phishing, and network attacks. Adhering to warnings keeps people safer online. Recent improvements in warning design have raised adherence rates, but they could still be higher. And prior work suggests many people still do not understand them. Thus, two challenges remain: increasing both comprehension and adherence rates. To dig deeper into user decision making and comprehension of warnings, we performed an experience sampling study of web browser security warnings, which involved surveying over 6,000 Chrome and Firefox users in situ to gather reasons for adhering or not to real warnings. We find these reasons are many and vary with context. Contrary to older prior work, we do not find a single dominant failure in modern warning design-like habituation-that prevents effective decisions. We conclude that further improvements to warnings will require solving a range of smaller contextual misunderstandings.

show abstract

“…Inter-rater reliability (IRR) was computed using the KupperHafner statistic [32]. For the Choice question, we computed IRR for each pair of raters.…”

Section: Discussionmentioning

confidence: 99%

An Experience Sampling Study of User Reactions to Browser Warnings in the Field

Reeder

Felt

Consolvo

et al. 2018

Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

show abstract

“…The inter-rater agreement was calculated over 3500 codes (of two independent raters) for 1512 segments from networks A and B, periods 1 and 2. The inter-rater correlation was calculated using the Kupper measure (Kupper & Hafner, 1989). The correlation was very high being 0.96 for all codes together.…”

Section: Discussionmentioning

confidence: 99%

Teachers’ development of professional knowledge through action research and the facilitation of this by teacher educators

Ponte

Beijaard

et al. 2004

Teaching and Teacher Education

125

View full text Add to dashboard Cite

“…Thus, it involved non-mutual exclusivity of codes; it also lacked a finite set of instances for potential agreement between raters. These characteristics ruled out assessment of inter-rater agreement using tools that rely on these discrete counts, such as the kappa measure of agreement [25][26][27].…”

Section: Content Analysismentioning

confidence: 99%

Content validity in the PROMIS social-health domain: a qualitative analysis of focus-group data

et al. 2008

View full text Add to dashboard Cite

Purpose-Our aim was to assess the content validity of the Patient-Reported Outcomes Measurement Information System (PROMIS) social health item banks by comparing a prespecified conceptual model with concepts that focus-group participants identified as important social-healthrelated outcomes. These data will inform the process of improving health-related quality-of-life measures.Methods-Twenty-five patients with a range of social limitations due to chronic health conditions were recruited at two sites; four focus groups were conducted. Raters independently classified participants' statements using a hierarchical, nested schema that included health-related outcomes, role performance, role satisfaction, family/friends, work, and leisure.Results-Key themes that emerged were fulfilling both family and work responsibilities and the distinction between activities done out of responsibility versus enjoyment. Although focus-group participants identified volunteerism and pet ownership as important social-health-related concepts, these were not in our original conceptual model. The concept of satisfaction was often found to overlap with the concept of performance.Conclusion-Our conceptual model appeared mostly comprehensive but is being further refined to more appropriately (a) distinguish between responsibilities versus discretionary activities, and (b) situate the outcome of satisfaction as it relates to impairment in social and other domains of health.

show abstract

On Assessing Interrater Agreement for Multiple Attribute Responses

Cited by 37 publications

References 23 publications

An Experience Sampling Study of User Reactions to Browser Warnings in the Field

An Experience Sampling Study of User Reactions to Browser Warnings in the Field

Teachers’ development of professional knowledge through action research and the facilitation of this by teacher educators

Content validity in the PROMIS social-health domain: a qualitative analysis of focus-group data

Contact Info

Product

Resources

About