How to assess intra- and inter-observer agreement with quantitative PET using variance component analysis: a proposal for standardisation

Abstract

BackgroundQuantitative measurement procedures need to be accurate and precise to justify their clinical use. Precision reflects deviation of groups of measurement from another, often expressed as proportions of agreement, standard errors of measurement, coefficients of variation, or the Bland-Altman plot. We suggest variance component analysis (VCA) to estimate the influence of errors due to single elements of a PET scan (scanner, time point, observer, etc.) to express the composite uncertainty of repeated measurements and obtain relevant repeatability coefficients (RCs) which have a unique relation to Bland-Altman plots. Here, we present this approach for assessment of intra- and inter-observer variation with PET/CT exemplified with data from two clinical studies.MethodsIn study 1, 30 patients were scanned pre-operatively for the assessment of ovarian cancer, and their scans were assessed twice by the same observer to study intra-observer agreement. In study 2, 14 patients with glioma were scanned up to five times. Resulting 49 scans were assessed by three observers to examine inter-observer agreement. Outcome variables were SUVmax in study 1 and cerebral total hemispheric glycolysis (THG) in study 2.ResultsIn study 1, we found a RC of 2.46 equalling half the width of the Bland-Altman limits of agreement. In study 2, the RC for identical conditions (same scanner, patient, time point, and observer) was 2392; allowing for different scanners increased the RC to 2543. Inter-observer differences were negligible compared to differences owing to other factors; between observer 1 and 2: −10 (95 % CI: −352 to 332) and between observer 1 vs 3: 28 (95 % CI: −313 to 370).ConclusionsVCA is an appealing approach for weighing different sources of variation against each other, summarised as RCs. The involved linear mixed effects models require carefully considered sample sizes to account for the challenge of sufficiently accurately estimating variance components.Electronic supplementary materialThe online version of this article (doi:10.1186/s12880-016-0159-3) contains supplementary material, which is available to authorized users.

Intra- and inter-observer agreement and reliability of bone mineral density measurements around acetabular cup: a porcine ex-vivo study using single- and dual-energy computed tomography

“…Repeatability coefficients (RC) were calculated for the repeated measurements according to Bartlett and Frost ( 30 ). RC is an estimate below which the absolute difference would be expected to lie with 95% certainty ( 30 – 32 ). To compare RC and the minimal clinically relevant difference, the RCs were converted to percentages by dividing RC with the mean BMD of measurement 1.…”

Group-sequential analysis may allow for early trial termination: illustration by an intra-observer repeatability study

“…We used data from an ongoing clinical study in patients with suspicion of ovarian cancer which was described elsewhere [ 11 ]. In brief, this study’s primary hypothesis was that dual time FDG-PET/CT performed at 60 and 180 min after injection of tracer would increase the diagnostic accuracy of FDG-PET/CT imaging (routinely performed at 60 min only) for preoperative assessment of resectability, provided optimal debulking is achievable.…”

Morphometric Study of The Ventricular Indexes in Healthy Ovine Brain Using MRI

“…Firstly, descriptive statistics like mean and standard deviations have been calculated. Then, differences in measurements between the two observers (inter-observer variability), and the differences from the same observer in different time points (intra-observer variability) were compared via Bland-Altman plots (7,8) ( Fig. 6).…”