Assessing Agreement When Agreement Is Hard to Assess—The Agatston Score for Coronary Calcification

Abstract

Method comparison studies comprised simple scatterplots of paired measurements, a 45-degree line as benchmark, and correlation coefficients up to the advent of Bland–Altman analysis in the 1980s. The Agatston score for coronary calcification is based on computed tomography of the heart, and it originated in 1990. A peculiarity of the Agatston score is the often-observed skewed distribution in screening populations. As the Agatston score has manifested itself in preventive cardiology, it is of interest to investigate how reproducibility of the Agatston score has been established. This review is based on literature findings indexed in MEDLINE/PubMed before 20 November 2021. Out of 503 identified articles, 49 papers were included in this review. Sample sizes were highly variable (10–9761), the main focus comprised intra- and interrater as well as intra- and interscanner variability assessments. Simple analysis tools such as scatterplots and correlation coefficients were successively supplemented by first difference, later Bland–Altman plots; however, only very few publications were capable of deriving Limits of Agreement that fit the observed data visually in a convincing way. Moreover, several attempts have been made in the recent past to improve the analysis and reporting of method comparison studies. These warrant increased attention in the future.

Modeling Bland–Altman Limits of Agreement with Fractional Polynomials—An Example with the Agatston Score for Coronary Calcification

“…Since its original proposal back in 1990 [30], the CAC score has repeatedly been subject to agreement assessments in order to investigate the score's reliability. A recent review [31] found sample sizes to be highly variable in studies of agreement on the CAC score , and research groups focused on intra-and inter-rater as well as intra-and inter-scanner variability assessments. Andersen and Gerke [31] concluded that only very few research articles were capable of deriving limits of agreement that fit the observed data visually in a convincing way.…”

“…A recent review [31] found sample sizes to be highly variable in studies of agreement on the CAC score , and research groups focused on intra-and inter-rater as well as intra-and inter-scanner variability assessments. Andersen and Gerke [31] concluded that only very few research articles were capable of deriving limits of agreement that fit the observed data visually in a convincing way. This is why alternative methods like fractional polynomial modeling may prove useful in assessing the agreement of the CAC score.…”

“…Andersen and Gerke [31] concluded that only very few research articles were capable of deriving limits of agreement that fit the observed data visually in a convincing way. This is why alternative methods like fractional polynomial modeling may prove useful in assessing the agreement of the CAC score.…”