Computing inter‐rater reliability and its variance in the presence of high agreement

Gwet, Kilem L.

doi:10.1348/000711006x126600

Cited by 1,407 publications

(1,248 citation statements)

References 15 publications

(24 reference statements)

Supporting

Mentioning

1,139

Contrasting

Unclassified

Order By: Relevance

“…A majority rules approach was taken by the vEEG reviewers to identify GTCSs, which were required to involve a tonic phase followed immediately by a clonic phase 14. Gwet's AC215 was used to calculate inter‐rater agreement for device placement and categorization of events as GTCSs.…”

Section: Methodsmentioning

confidence: 99%

Detection of generalized tonic–clonic seizures using surface electromyographic monitoring

et al. 2017

View full text Add to dashboard Cite

SummaryObjectiveA prospective multicenter phase III trial was undertaken to evaluate the performance and tolerability in the epilepsy monitoring unit (EMU) of an investigational wearable surface electromyographic (sEMG) monitoring system for the detection of generalized tonic–clonic seizures (GTCSs).MethodsOne hundred ninety‐nine patients with a history of GTCSs who were admitted to the EMU in 11 level IV epilepsy centers for clinically indicated video‐electroencephalographic monitoring also received sEMG monitoring with a wearable device that was worn on the arm over the biceps muscle. All recorded sEMG data were processed at a central site using a previously developed detection algorithm. Detected GTCSs were compared to events verified by a majority of three expert reviewers.ResultsFor all subjects, the detection algorithm detected 35 of 46 (76%, 95% confidence interval [CI] = 0.61–0.87) of the GTCSs, with a positive predictive value (PPV) of 0.03 and a mean false alarm rate (FAR) of 2.52 per 24 h. For data recorded while the device was placed over the midline of the biceps muscle, the system detected 29 of 29 GTCSs (100%, 95% CI = 0.88–1.00), with a detection delay averaging 7.70 s, a PPV of 6.2%, and a mean FAR of 1.44 per 24 h. Mild to moderate adverse events were reported in 28% (55 of 199) of subjects and led to study withdrawal in 9% (17 of 199). These adverse events consisted mostly of skin irritation caused by the electrode patch that resolved without treatment. No serious adverse events were reported.SignificanceDetection of GTCSs using an sEMG monitoring device on the biceps is feasible. Proper positioning of this device is important for accuracy, and for some patients, minimizing the number of false positives may be challenging.

show abstract

Section: Methodsmentioning

confidence: 99%

Detection of generalized tonic–clonic seizures using surface electromyographic monitoring

et al. 2017

View full text Add to dashboard Cite

show abstract

“…If |AVGC-0.5|<sVGC then the difference was considered not significant (n.s.). Inter-observer reliability was assessed for each setting, criterion and type of examination using Gwet's AC1 coefficient (20) . The AC1 coefficient is an alternative to the Kappa coefficient, which is similar in formulation but does not suffer from some of the statistical problems of the Kappa coefficient (21) .…”

Section: Discussionmentioning

confidence: 99%

Clinical Audit of Image Quality in Radiology Using Visual Grading Characteristics Analysis

Tesselaar

Dahlström

Sandborg

2015

Radiat Prot Dosimetry

View full text Add to dashboard Cite

The aim of this work was to assess whether an audit of clinical image quality could be efficiently implementedwithin in a limited time frame using visual grading characteristics (VGC) analysis.Lumbar spine radiography, bedside chest radiography and abdominal CT were selected. For each examination, images were acquired or reconstructed in two ways. Twenty images per examination were assessed by 40 radiology residents using visual grading of image criteria. The results were analysed using VGC. Inter-observer reliability was assessed.The results of the visual grading analysis were consistent with expected outcomes. The interobserver reliability was moderate to good and correlated with perceived image quality (r 2 =0.47). The median observation time per image or image series was within 2 minutes.These results suggest that the use of visual grading of image criteria to assess the quality of radiographs provides a rapid method for performing an image quality audit in a clinical environment.O12-5 Image quality audit using VGC analysis 3

show abstract

“…Traditionally, Cohen's Kappa or Scott's Pi has been used to evaluate reliability of human judgment. According to Gwet [13], those ratios can be a problem when the extent of agreement between evaluators is too high. To avoid obtaining biased reliability, we adopted the AC 1 reliability measure [12,13].…”

Section: Effect Of the Constant Comparison Methodsmentioning

confidence: 99%

“…According to Gwet [13], those ratios can be a problem when the extent of agreement between evaluators is too high. To avoid obtaining biased reliability, we adopted the AC 1 reliability measure [12,13]. Overall, the average of AC1 shows positive signs.…”

Section: Effect Of the Constant Comparison Methodsmentioning

confidence: 99%

Study on the Use of the Constant Comparison Method : Lessons from Training Novice Modelers

Kim¹,

Park

Rho³

2013

The Journal of Society for e-Business Studies

View full text Add to dashboard Cite

Conceptual modeling is a critical activity for developing successful business information systems. The objective of this study is to evaluate the possibility of applying the constant comparison method from the grounded theory to conceptual modeling. To achieve the objective, we trained novice modelers and split them into two groups for evaluation. The experimental results show that applying the constant comparison method could increase acceptability from more experienced conceptual modelers. Moreover, while the control group was experienced difficulties when domain knowledge is unfamiliar, the experimental group could handle difficulties more effectively. In addition, applying the constant comparison method also decreased the time to complete analysis for conceptual modeling.

show abstract

Computing inter‐rater reliability and its variance in the presence of high agreement

Cited by 1,407 publications

References 15 publications

Detection of generalized tonic–clonic seizures using surface electromyographic monitoring

Detection of generalized tonic–clonic seizures using surface electromyographic monitoring

Clinical Audit of Image Quality in Radiology Using Visual Grading Characteristics Analysis

Study on the Use of the Constant Comparison Method : Lessons from Training Novice Modelers

Contact Info

Product

Resources

About