A Statistical Approach to Assessment of Agreement Involving Multiple Raters

Hedayat, A.; Lou, Congrong; Sinha, Bikas K.

doi:10.1080/03610920902947220

Cited by 6 publications

(2 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Lin et al 21 gave a review and comparison of various measures, including CCC, by comparing powers of the following three tests: (i) equality of two means, that is,

μ_{x} = μ_{y}

, (ii) equality of two SDs, that is,

σ_{x} = σ_{y}

, and (iii) equality of Pearson correlation to a standard value, that is,

ρ_{x y} = ρ_{0}

, where

ρ_{0}

is a given value. This work was further extended by Hedayat et al 22 involving multiple raters. In another direction, Yimprayoon et al 19 extended the work of Lin et al by combining the problems of testing for

μ_{x} = μ_{y}

σ_{x} = σ_{y}

, and

ρ_{x y} \geq ρ_{0}

into one overall testing problem under a bivariate normal set‐up, and then they presented results based on a simulation study.…”

Section: Introductionmentioning

confidence: 81%

Generalized confidence interval for an agreement between raters

Bhaumik

Shi

Reda

et al. 2021

Statistics in Medicine

Self Cite

View full text Add to dashboard Cite

Estimation and inference are two key components toward the solution of any statistical problem; however, the inferential issues of statistical assessment of agreement among two or more raters have not been well developed as compared to the development of estimation procedures in this area. The fundamental reason for this gap is the complex expression of the concordance correlation coefficient (CCC) that is frequently used in assessing agreement among raters.Large sample-based statistical tests for CCC often fail to produce desired results for small samples. Hence, inferential procedures for small samples are urgently needed to evaluate agreement between raters. We argue that hypothesis testing of CCC has little value in practice due to the absence of a gold standard of agreement. In this article, we construct the generalized confidence interval (GCI) for CCC utilizing a bivariate normal distribution of measurements, and also develop a large sample-based confidence interval (LSCI). We establish satisfactory performance of GCI by providing the desired coverage probability (CP) via simulation. Results of GCI and LSCI are illustrated and compared with a data set of a recent study performed at U.S. Department of Veterans Affairs, Hines.

show abstract

“…Lin et al 21 gave a review and comparison of various measures, including CCC, by comparing powers of the following three tests: (i) equality of two means, that is,

μ_{x} = μ_{y}

, (ii) equality of two SDs, that is,

σ_{x} = σ_{y}

, and (iii) equality of Pearson correlation to a standard value, that is,

ρ_{x y} = ρ_{0}

, where

ρ_{0}

μ_{x} = μ_{y}

σ_{x} = σ_{y}

, and

ρ_{x y} \geq ρ_{0}

into one overall testing problem under a bivariate normal set‐up, and then they presented results based on a simulation study.…”

Section: Introductionmentioning

confidence: 81%

Generalized confidence interval for an agreement between raters

Bhaumik

Shi

Reda

et al. 2021

Statistics in Medicine

Self Cite

View full text Add to dashboard Cite

show abstract

“…Modeling the method comparison data is the frst step. For this purpose, the mixed-efects model [2] is commonly used and assumes an independent normal distribution for both random efects and errors when the variability of the measurement remains constant over the entire measurement range [3][4][5][6][7][8][9]. Te second step is the agreement evaluation between the methods.…”

Section: Introductionmentioning

confidence: 99%

An Improved Measurement Error Model for Analyzing Unreplicated Method Comparison Data under Asymmetric Heavy-Tailed Distributions

Duwarahan

Nawarathna

2022

Journal of Probability and Statistics

View full text Add to dashboard Cite

Method comparison studies mainly focus on determining if the two methods of measuring a continuous variable are agreeable enough to be used interchangeably. Typically, a standard mixed-effects model uses to model the method comparison data that assume normality for both random effects and errors. However, these assumptions are frequently violated in practice due to the skewness and heavy tails. In particular, the biases of the methods may vary with the extent of measurement. Thus, we propose a methodology for method comparison data to deal with these issues in the context of the measurement error model (MEM) that assumes a skew- t (ST) distribution for the true covariates and centered Student’s t (cT) distribution for the errors with known error variances, named STcT-MEM. An expectation conditional maximization (ECM) algorithm is used to compute the maximum likelihood (ML) estimates. The simulation study is performed to validate the proposed methodology. This methodology is illustrated by analyzing gold particle data and then compared with the standard measurement error model (SMEM). The likelihood ratio (LR) test is used to identify the most appropriate model among the above models. In addition, the total deviation index (TDI) and concordance correlation coefficient (CCC) were used to check the agreement between the methods. The findings suggest that our proposed framework for analyzing unreplicated method comparison data with asymmetry and heavy tails works effectively for modest and large samples.

show abstract

References

2017

Measuring Agreement

View full text Add to dashboard Cite

A Statistical Approach to Assessment of Agreement Involving Multiple Raters

Cited by 6 publications

References 17 publications

Generalized confidence interval for an agreement between raters

Generalized confidence interval for an agreement between raters

An Improved Measurement Error Model for Analyzing Unreplicated Method Comparison Data under Asymmetric Heavy-Tailed Distributions

References

Contact Info

Product

Resources

About