A Comparison of Gwet’s AC1 and Kappa When Calculating Inter-Rater Reliability Coefficients in a Teacher Evaluation Context

Jiménez, Albert M.; Zepeda, Sally J.

doi:10.3138/jehr-2019-0001

Cited by 7 publications

(11 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Gwet's AC tends to lessen the kappa limitations; [22,23] hence we consider Gwet's AC providing more stable interrater reliability coe cients in our study following few recent studies that have also preferred Gwet's AC over Cohen/Conger's Kappa for being a more stable inter-rater reliability coe cient. [24,25,26]…”

Section: Discussionmentioning

confidence: 99%

Comparison of Electronic Medical Records with Self-report in Measuring Cancer Screening

Bhattacharyya

Rawl

Dickinson

et al. 2022

Preprint

View full text Add to dashboard Cite

Background: Efficient measurement of the receipt of cancer screening has been attempted with electronic health records (EHRs), but EHRs are commonly implemented within a single health care setting. However, health information exchange (HIE) includes EHR data from multiple health care systems and settings, thereby providing a more population-based measurement approach. In this study, we set out to understand the value of statewide HIE data in comparison to survey self-report (SR) to measure population-based cancer screening. Methods: A statewide survey was conducted among residents in Indiana who had been seen at an ambulatory or inpatient clinical setting in the past year. Measured cancer screening tests included Colonoscopy, Fecal Immunochemical Test (FIT) for colorectal cancer, Human Papilloma Virus (HPV) and Pap tests for cervical cancer, and Mammogram for breast cancer. For each screening test, the self-reported response for receipt of the screening (yes/no) and ‘time since last screening’ were compared with the corresponding information from patient HIE to evaluate the association between the two measures. Results: Percent agreement of HIE and self-report of screening receipt ranged from 59% to 81%, with the Gwet’s AC ranging from 0.24-0.76, indicating a fair to substantial concordance. For the time since receipt of last screening test, Percent agreement ranged from 71% to 93% along with the Gwet’s AC ranging from 0.47-0.91. In comparison with SR data, HIE data provided relatively more additional information about laboratory-based tests: FIT (15% HIE alone vs. 4% SR alone) and HPV tests (24% HIE alone vs. 12% SR alone) and less additional information about procedures: Colonoscopy (7% HIE alone vs. 34% SR alone), Pap test (10% HIE alone vs. 27% SR alone), or Mammography (8% HIE alone vs. 20% SR alone). Conclusion: Studies that use a single data source should consider the type of cancer screening test to choose the optimal data collection method. HIE and self-report both provided unique information in measuring cancer screening, and the most robust measurement approach involves collecting screening information from both HIE and patient self-report.

show abstract

Section: Discussionmentioning

confidence: 99%

Comparison of Electronic Medical Records with Self-report in Measuring Cancer Screening

Bhattacharyya

Rawl

Dickinson

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The second stage involved testing the reliability of the assessment method. The reliability of the proposed method was analyzed by Gwet's AC1 [15,18,32]. In evaluating reliability among experts, Gwet's AC1 is better than Kappa Statistics.…”

Section: Methodsmentioning

confidence: 99%

A Novel Approach to Automated Behavioral Diagram Assessment using Label Similarity and Subgraph Edit Distance

Fauzan

Siahaan

Rochimah

et al. 2021

csci

View full text Add to dashboard Cite

Unified Modelling Language (UML) is one of the standard languages used in modelling software. Therefore, UML is widely taught in many universities. Generally, teachers assign students to build UML diagram designs based on a predetermined project. However, the assessment of such assignments can be challenging and teachers may be inconsistent in assessing students’ answers. Thus, automated UML diagram assessment becomes essential to maintaining assessment consistency. This study uses a behavioral diagram as the object of research since it is a commonly taught UML diagram. The behavioral diagram can show a dynamic view of the software. This study proposes a new approach to automatically assessing the similarity of behaviour diagrams as reliably as experts. We divide the assessment into two portions: semantic assessment and structural assessment. Label similarity is used to calculate semantic assessment, while subgraph edit distance is used to calculate structural assessment. The results suggest that the proposed approach is as reliable as an expert in assessing the similarity between two behaviour diagrams. The observed agreement value suggests strong agreement between the use of experts and the proposed approach.

show abstract

“…This can involve using the kappa statistic [37,38,39] or Gwet's AC1 [40,41,42] to measure a method's results and the Alpha Cronbach [43,44] to ensure the data's reliability. But, Gwet's AC1 could be better than the kappa statistic for assessment case [45,46,47,48]. Software reuse papers tested their research using precision and recall, with none utilizing similarity measurements to test their research.…”

Section: What Are the Parameters (Measuring Instruments) Used To Measmentioning

confidence: 99%

Software similarity measurements using UML diagrams: A systematic literature review

Triandini

Fauzan

Siahaan

et al. 2021

regist. j. ilm. teknol. sist. inf.

View full text Add to dashboard Cite

Every piece of software uses a model to derive its operational, auxiliary, and functional procedures. Unified Modeling Language (UML) is a standard displaying language for determining, recording, and building a software product. Several algorithms have been used by researchers to measure similarities between UML artifacts. However, there no literature studies have considered measurements of UML diagram similarities. This paper presents the results of a systematic literature review concerning similarity measurements between the UML diagrams of different software products. The study reviews and identifies similarity measurements of UML artifacts, with class diagram, sequence diagram, statechart diagram, and use case diagram being UML diagrams that are widely used as research objects for measuring similarity. Measuring similarity enables resolution of the problem domains of software reuse, similarity measurement, and clone detection. The instruments used to measure similarity are semantic and structural similarity. The findings indicate opportunities for future research regarding calculating other UML diagrams, compiling calculation information for each diagram, adapting semantic and structural similarity calculation methods, determining the best weight for each item in the diagram, testing novel proposed methods, and building or finding good datasets for use as testing material.

show abstract

A Comparison of Gwet’s AC1 and Kappa When Calculating Inter-Rater Reliability Coefficients in a Teacher Evaluation Context

Cited by 7 publications

References 8 publications

Comparison of Electronic Medical Records with Self-report in Measuring Cancer Screening

Comparison of Electronic Medical Records with Self-report in Measuring Cancer Screening

A Novel Approach to Automated Behavioral Diagram Assessment using Label Similarity and Subgraph Edit Distance

Software similarity measurements using UML diagrams: A systematic literature review

Contact Info

Product

Resources

About