Comparing Hyperprior Distributions to Estimate Variance Components for Interrater Reliability Coefficients

Hove, Debby ten; Jorgensen, Terrence D.; Ark, L. Andries van der

doi:10.1007/978-3-030-43469-4_7

Cited by 3 publications

(4 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The code includes functions to estimate ICCs by means of maximum likelihood estimation (MLE) using the R package lme4 (Bates et al, 2015) or Markov chain Monte Carlo (MCMC) estimation using the R package brms (Bürkner, 2017), and is accompanied by example data-mimicking the design of Example 3-and example analyses of these data using both estimation methods. For more information about the MCMC method, we refer to Ten Hove et al (2020, 2021, and for more information about the MLE estimation method we refer to work by Jiang (2018).…”

Section: Discussionmentioning

confidence: 99%

“…The MCMC and MLE estimation methods we provide on the OSF were shown to yield similar point estimates (Ten Hove et al, 2021). The MCMC estimation readily provides credible intervals for the ICCs (see, e.g., Ten Hove et al, 2020, 2021. The MLE-estimation method is supplemented with Monte-Carlo based confidence intervals that are specifically useful for coefficients such as ICCs (i.e., functions of parameters whose sampling distributions cannot be expected to be normal; cf.…”

Section: Guidelines On Selecting An Icc For Irr 11mentioning

confidence: 92%

“…A pilot study should then be conducted to inspect the IRR of the measures, for example, using multiple ratings of a subset of the subjects (see Examples 1 and 3). • A minimum of two raters per subject is required to estimate an ICC, though at least three raters are required to yield accurately estimated ICCs (see, e.g., Briesch et al, 2014;Ten Hove et al, 2020).…”

Section: Further Considerationsmentioning

confidence: 99%

See 2 more Smart Citations

Updated guidelines on selecting an intraclass correlation coefficient for interrater reliability, with applications to incomplete observational designs.

Hove¹,

Jorgensen²,

Ark³

2022

Psychological Methods

View full text Add to dashboard Cite

Several intraclass correlation coefficients (ICCs) are available to assess the interrater reliability (IRR) of observational measurements. Selecting an ICC is complicated, and existing guidelines have three major limitations. First, they do not discuss incomplete designs, in which raters partially vary across subjects. Second, they provide no coherent perspective on the error variance in an ICC, clouding the choice between the available coefficients. Third, the distinction between fixed or random raters is often misunderstood. Based on generalizability theory (GT), we provide updated guidelines on selecting an ICC for IRR, which are applicable to both complete and incomplete observational designs. We challenge conventional wisdom about ICCs for IRR by claiming that raters should seldom (if ever) be considered fixed. Also, we clarify how to interpret ICCs in the case of unbalanced and incomplete designs. We explain four choices a researcher needs to make when selecting an ICC for IRR, and guide researchers through these choices by means of a flowchart, which we apply to three empirical examples from clinical and developmental domains. In the Discussion, we provide guidance in reporting, interpreting, and estimating ICCs, and propose future directions for research into the ICCs for IRR.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Guidelines On Selecting An Icc For Irr 11mentioning

confidence: 92%

Section: Further Considerationsmentioning

confidence: 99%

See 1 more Smart Citation

Updated guidelines on selecting an intraclass correlation coefficient for interrater reliability, with applications to incomplete observational designs.

Hove¹,

Jorgensen²,

Ark³

2022

Psychological Methods

View full text Add to dashboard Cite

show abstract

“…The kurtosis ( 6 d f −4 ) is undefined due to division by 0 (thus approaches infinity in the limit), making it less restrictive than a truncated normal distribution about values being much larger than the mean. Simulation research has shown that this prior works well in variance-decomposition models [77], and it is the default prior for scale parameters in the R package brms [78]. Stan provides location (µ) and scale (σ) parameters, to center the t distribution at a different mean or to have greater variance.…”

Section: Stage 1: Estimate Multivariate Srm Parametersmentioning

confidence: 99%

Two-Stage Limited-Information Estimation for Structural Equation Models of Round-Robin Variables

Jorgensen,

Bhangale,

Rosseel

2024

Stats

View full text Add to dashboard Cite

We propose and demonstrate a new two-stage maximum likelihood estimator for parameters of a social relations structural equation model (SR-SEM) using estimated summary statistics (Σ^) as data, as well as uncertainty about Σ^ to obtain robust inferential statistics. The SR-SEM is a generalization of a traditional SEM for round-robin data, which have a dyadic network structure (i.e., each group member responds to or interacts with each other member). Our two-stage estimator is developed using similar logic as previous two-stage estimators for SEM, developed for application to multilevel data and multiple imputations of missing data. We demonstrate out estimator on a publicly available data set from a 2018 publication about social mimicry. We employ Markov chain Monte Carlo estimation of Σ^ in Stage 1, implemented using the R package rstan. In Stage 2, the posterior mean estimates of Σ^ are used as input data to estimate SEM parameters with the R package lavaan. The posterior covariance matrix of estimated Σ^ is also calculated so that lavaan can use it to calculate robust standard errors and test statistics. Results are compared to full-information maximum likelihood (FIML) estimation of SR-SEM parameters using the R package srm. We discuss how differences between estimators highlight the need for future research to establish best practices under realistic conditions (e.g., how to specify empirical Bayes priors in Stage 1), as well as extensions that would make 2-stage estimation particularly advantageous over single-stage FIML.

show abstract

Comparing Maximum Likelihood to Markov Chain Monte Carlo Estimation of the Multivariate Social Relations Model

Bhangale,

Jorgensen

2024

Springer Proceedings in Mathematics &Amp; Statistics

View full text Add to dashboard Cite

Comparing Hyperprior Distributions to Estimate Variance Components for Interrater Reliability Coefficients

Abstract: UvA-DARE (Digital Academic Repository) Comparing hyperprior distributions to estimate variance components for interrater reliability coefficients ten Hove, D.; Jorgensen, T.D.; van der Ark, L.A.

Cited by 3 publications

References 14 publications

Updated guidelines on selecting an intraclass correlation coefficient for interrater reliability, with applications to incomplete observational designs.

Updated guidelines on selecting an intraclass correlation coefficient for interrater reliability, with applications to incomplete observational designs.

Two-Stage Limited-Information Estimation for Structural Equation Models of Round-Robin Variables

Comparing Maximum Likelihood to Markov Chain Monte Carlo Estimation of the Multivariate Social Relations Model

Contact Info

Product

Resources

About