Reliability, repeatability and reproducibility: analysis of measurement errors in continuous variables

Bartlett, Jonathan; Frost, Chris

doi:10.1002/uog.5256

Cited by 776 publications

(797 citation statements)

References 18 publications

(26 reference statements)

Supporting

Mentioning

756

Contrasting

Unclassified

Order By: Relevance

“…In a measurement error study, we often consider the observers as a random sample from a larger population of potential observers who may be used in future studies or clinical practice 41. In the present study, we treated each segmentation method as a different observer of the tumor volume.…”

Section: Methodsmentioning

confidence: 99%

“…We considered any feature below the low level as nonreproducible (NR). We based this approach on methods reported in several clinical studies 41, 53, 54, 55. Also, Galavis et al36 used a similar scale to categorize the features based on their variation, and Tixier et al37 indicated that such limits were referenced to previously defined reproducibility limits for standard uptake values.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Reproducibility of F18‐FDG PET radiomic features for different cervical tumor segmentation methods, gray‐level discretization, and reconstruction algorithms

Altazi

Zhang

Fernandez

et al. 2017

J Applied Clin Med Phys

View full text Add to dashboard Cite

Site‐specific investigations of the role of radiomics in cancer diagnosis and therapy are emerging. We evaluated the reproducibility of radiomic features extracted from 18Flourine–fluorodeoxyglucose (18F‐FDG) PET images for three parameters: manual versus computer‐aided segmentation methods, gray‐level discretization, and PET image reconstruction algorithms. Our cohort consisted of pretreatment PET/CT scans from 88 cervical cancer patients. Two board‐certified radiation oncologists manually segmented the metabolic tumor volume (MTV1 and MTV2) for each patient. For comparison, we used a graphical‐based method to generate semiautomated segmented volumes (GBSV). To address any perturbations in radiomic feature values, we down‐sampled the tumor volumes into three gray‐levels: 32, 64, and 128 from the original gray‐level of 256. Finally, we analyzed the effect on radiomic features on PET images of eight patients due to four PET 3D‐reconstruction algorithms: maximum likelihood‐ordered subset expectation maximization (OSEM) iterative reconstruction (IR) method, fourier rebinning‐ML‐OSEM (FOREIR), FORE‐filtered back projection (FOREFBP), and 3D‐Reprojection (3DRP) analytical method. We extracted 79 features from all segmentation method, gray‐levels of down‐sampled volumes, and PET reconstruction algorithms. The features were extracted using gray‐level co‐occurrence matrices (GLCM), gray‐level size zone matrices (GLSZM), gray‐level run‐length matrices (GLRLM), neighborhood gray‐tone difference matrices (NGTDM), shape‐based features (SF), and intensity histogram features (IHF). We computed the Dice coefficient between each MTV and GBSV to measure segmentation accuracy. Coefficient values close to one indicate high agreement, and values close to zero indicate low agreement. We evaluated the effect on radiomic features by calculating the mean percentage differences (d¯) between feature values measured from each pair of parameter elements (i.e. segmentation methods: MTV1‐MTV2, MTV1‐GBSV, MTV2‐GBSV; gray‐levels: 64‐32, 64‐128, and 64‐256; reconstruction algorithms: OSEM‐FORE‐OSEM, OSEM‐FOREFBP, and OSEM‐3DRP). We used false|normald¯false| as a measure of radiomic feature reproducibility level, where any feature scored false|normald¯false| ±SD ≤ |25|% ± 35% was considered reproducible. We used Bland–Altman analysis to evaluate the mean, standard deviation (SD), and upper/lower reproducibility limits (U/LRL) for radiomic features in response to variation in each testing parameter. Furthermore, we proposed U/LRL as a method to classify the level of reproducibility: High— ±1% ≤ U/LRL ≤ ±30%; Intermediate— ±30% < U/LRL ≤ ±45%; Low— ±45 < U/LRL ≤ ±50%. We considered any feature below the low level as nonreproducible (NR). Finally, we calculated the interclass correlation coefficient (ICC) to evaluate the reliability of radiomic feature measurements for each parameter. The segmented volumes of 65 patients (81.3%) scored Dice coefficient >0.75 for all three volumes. The result outcomes revealed a tendency of higher radiomic fe...

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Reproducibility of F18‐FDG PET radiomic features for different cervical tumor segmentation methods, gray‐level discretization, and reconstruction algorithms

Altazi

Zhang

Fernandez

et al. 2017

J Applied Clin Med Phys

View full text Add to dashboard Cite

show abstract

“…Repeatability describes the variation in repeat measurements under the same conditions (testretest) [10], whilst reliability describes the measurement error relative to the inherent variability in the population. Reliability therefore indicates the ability of a diagnostic test to distinguish between members of a population [10].…”

Section: Introductionmentioning

confidence: 99%

“…Reliability therefore indicates the ability of a diagnostic test to distinguish between members of a population [10].…”

Section: Introductionmentioning

confidence: 99%

Use of the oral sugar test in ponies when performed with or without prior fasting

Knowles

Harris²,

Elliott

et al. 2016

Equine Veterinary Journal

View full text Add to dashboard Cite

Summary Background It is recommended that the oral sugar test (OST) for insulin dysregulation (ID) be performed after an overnight fast, but fasting is impractical in ponies kept solely at pasture. There are few data on OST repeatability and reliability in ponies. Objectives To report 1) whether OST results obtained in the morning after an overnight fast or without fasting in the afternoon (FASTING/FED) can be used interchangeably, 2) time of highest insulin concentration Tmax[insulin], repeatability and reliability of insulin response to the OST when FASTING or FED and 3) dichotomous agreement (ID/normal) within a small sample when FASTING or FED. Study design Method comparison study. Methods Oral sugar tests were performed on four occasions in 10 adult native British ponies, twice FASTING and twice FED. Insulin concentrations were measured by radioimmunoassay at 0–120 min (T0,30,60,75,90,120). Differences between FASTING and FED results were assessed using mixed effects models. Indices of repeatability and reliability were calculated; dichotomous agreement was reported using kappa statistics. Results Serum [insulin] was significantly (P≤0.05) higher at T60–T90 with prior fasting (estimated differences [95% confidence intervals]): T60: 23.5 μiu/ml (8.7–38.4 μiu/ml), T75: 27.1 μiu/ml (12.3–41.8 μiu/ml), T90: 15.1 (0.36–29.9 μiu/ml). Most frequently, Tmax[ins] occurred at T30. At any single time point, within‐subject coefficients of variation were: FASTING: 40% and FED: 31%. The 95% limits for repeatability were FASTING: 29–340%, FED: 41–240%. Test reliabilities were FASTING: 0.70 and FED: 0.67. For dichotomous interpretation similar results (kappa = 0.7) were obtained using cut‐offs of [Insulin] >60 μiu/ml at T60 or T90 for FASTING and [Insulin] >51 μiu/ml at T30 or T60 for FED samples. Main limitations Oral sugar tests were performed on a small number of animals on one pasture during one season (spring). Conclusions Clinicians should beware of interpreting changes in absolute OST results owing to poor repeatability. When stabling is unavailable, OSTs of ponies at pasture may yield similar dichotomous results without prior fasting.

show abstract

“…However, this computer-based method has not been fully validated in terms of reliability; it includes subjective manual aspects such as ROI setting and image superimposing. Reliability relates the magnitude of the measurement error in observed measurements to the inherent variability in the 'error-free', 'true', or underlying level of the quantity between subjects [12].…”

Section: Introductionmentioning

confidence: 99%

A reliability study using computer-based analysis of finger joint space narrowing in rheumatoid arthritis patients

et al. 2016

View full text Add to dashboard Cite

Author identifying informationConflict of Interest: All the authors declare no conflict of interest.Funding: This study has no relevant funding. AbstractObjectives: The joint space difference index (JSDI) is a newly developed radiographic index which can quantitatively assess joint space narrowing progression of rheumatoid arthritis (RA) patients by using an image subtraction method on a computer. The aim of this study was to investigate the reliability of this method by non-experts utilizing RA image evaluation.Method: Four non-experts assessed JSDI for radiographic images of 260 metacarpophalangeal (MP) joints from 26 RA patients twice with an interval of more than 2 weeks. Two rheumatologists and 1 radiologist as well as the 4 non-experts examined the joints by using the Sharp-van der Heijde Scoring (SHS) method. The radiologist and 4 non-experts repeated the scoring with an interval of more than 2 weeks. We calculated intra-/inter-observer reliability using the intra-class correlation coefficients (ICC) for JSDI and SHS scoring, respectively.Results: The intra-/inter-observer reliabilities for the computer-based method were almost perfect ). Contrary to this, intra-/inter-observer reliability for SHS by experts was moderate to almost perfect (inter observer ICC, 0.576-0.862; intra observer ICC, 0.654-0.846). Conclusion:The results suggest that our computer-based method has high reliability to detect finger joint space narrowing progression in RA patients.

show abstract

Reliability, repeatability and reproducibility: analysis of measurement errors in continuous variables

Cited by 776 publications

References 18 publications

Reproducibility of F18‐FDG PET radiomic features for different cervical tumor segmentation methods, gray‐level discretization, and reconstruction algorithms

Reproducibility of F18‐FDG PET radiomic features for different cervical tumor segmentation methods, gray‐level discretization, and reconstruction algorithms

Use of the oral sugar test in ponies when performed with or without prior fasting

A reliability study using computer-based analysis of finger joint space narrowing in rheumatoid arthritis patients

Contact Info

Product

Resources

About