Assessing inter-rater agreement between multiple medical instruments with heteroscedastic measurements

Abstract

Background: In clinical medicine, agreement evaluation plays a major role in determining the compatibility and the accuracy of newly introduced methods with pre-existing methods. These methods may be assays, clinical observers, medical devices etc. It is vital to assess the compatibility and the accuracy of these newly introduced techniques because they deal with the measurements of the human body, such as blood pressure, cholesterol level, heart rate etc. In practice, agreement evaluation is carried out among two methods of measurements and deals with the data that are homoscedastic. The main objective of this study is to extend the standard mixed model to allow the error variances to depend on magnitude of measurement and evaluate agreement between multiple methods assuming the new model, taking the heteroscedasticity into account. Methods: In order to assess the agreement, there are two typical steps in method comparison studies. The first step is to model the data using the Heteroscedastic mixed effects model. The model fitting is carried out by using two main approaches, namely the mean method and the best linear unbiased predictor method. After fitting the model for the second step, the agreement evaluation is carried out using Concordance correlation coefficient and Total deviation index. Results: The illustrative example contained five methods of measurements and was with heteroscedastic measurements. First, the model fitting was carried out according to the two approaches and the resulting parameters were almost identical. After the model fitting, the agreement evaluation was performed. According to the values resulted from the agreement measurements, it is clear that all five methods agree sufficiently well with the reference method. Conclusions: The proposed model can be used to model the method comparison data with heteroscedastic measurements with multiple methods of measurements as well as balanced and unbalanced data designs. Under the proposed model, the agreement evaluation methodology for comparing multiple methods is also developed taking heteroscedasticity into account.

Non-surgical Management of Large Periapical Lesions with the Agreement Evaluation of the Methods

“…Then for the situations where the key assumptions such as constant error variance (homoscedastic error variance) are violated, a multiple heteroscedastic mixed effects model was used to model the data. The fitted model was validated using the 10-fold cross validation technique [19,20]. In order to assess the agreement between the three treatments, Concordance Correlation Coefficient (CCC) and Total Deviation Index (TDI) were used [21].…”

Critical issues concerning the assessment sets in agreement assessment

“…The last issue is the assessment between multiple methods. Due to rapid technological innovation, a large number of measurement methods have been introduced, making it imperative to assess agreement between multiple methods 9 . Nevertheless, the existing approaches for assessing agreement between multiple methods conduct separate assessments between each pair of the methods, 10 which does not lead to the desired exact proportion, as we will show in Section 3.…”

“…Due to rapid technological innovation, a large number of measurement methods have been introduced, making it imperative to assess agreement between multiple methods. 9 Nevertheless, the existing approaches for assessing agreement between multiple methods conduct separate assessments between each pair of the methods, 10 which does not lead to the desired exact 95% proportion, as we will show in Section 3. Thus, we propose two approaches that lead to an exact 95% proportion in the higher dimensions, one of which is optimal with minimum volume under certain type of distributions, and the other provides an easy-to-compute alternative when the optimal assessment set cannot be computed easily.…”

Appraisal of agreement between treatment procedures on deep caries lesions in the posterior permanent teeth of adults

“…In either case, the interest is in evaluating the agreement in order to check whether the raters can be used interchangeably. [1][2][3][4][5] Dental field is a major biomedical field that requires conducting agreement evaluation for pulp therapy treatments. The goal of pulp therapy is to treat, restore, and save the affected pulp of a tooth.…”