The Gold Standard Paradox in Digital Image Analysis: Manual Versus Automated Scoring as Ground Truth

Aeffner, Famke; Wilson, Kathryn; Martin, Nathan; Black, Joshua C.; Hendriks, Cris L. Luengo; Bolon, Brad; Rudmann, Daniel G.; Gianani, Roberto; Koegler, Sally R.; Krueger, Joseph S.; Young, G. Dave

doi:10.5858/arpa.2016-0386-ra

Cited by 148 publications

(148 citation statements)

References 79 publications

Supporting

Mentioning

147

Contrasting

Unclassified

Order By: Relevance

“…IHC/IF excels at providing fiber-by-fiber data, exclusive assessment of muscle fibers (allowing exclusion of areas of necrosis, fibrofatty infiltrate), assessment of revertant fibers, measurement of expression heterogeneity across the tissue and within individual muscle fibers (completeness of membrane expression), and measurement of variation in staining intensity. Manual methods can provide robust data, but they must be carefully controlled or the quantification of dystrophin is likely to be affected by scoring biases (Aeffner et al , 2017, Arechavala-Gomeza et al , 2010b, Beekman et al , 2014). Despite these challenges, there is strong correlation of the number of positive fibers counted in IF-stained sections and protein quantified by western blot, suggesting that IF/IHC and western blot are comparable methods (Nicholson et al , 1993).…”

Section: Evaluating Efficacy Of Therapeutic Candidatesmentioning

confidence: 99%

Duchenne and Becker Muscular Dystrophies: A Review of Animal Models, Clinical End Points, and Biomarker Quantification

Wilson¹,

Faelan²,

Patterson-Kane³

et al. 2017

Toxicol Pathol

Self Cite

View full text Add to dashboard Cite

Duchenne and Becker muscular dystrophies (DMD/BMD) are neuromuscular disorders that primarily affect boys due to an X-linked mutation in the DMD gene, resulting in reduced to near absence of dystrophin or expression of truncated forms of dystrophin. Some newer therapeutic interventions aim to increase sarcolemmal dystrophin expression, and accurate dystrophin quantification is critical for demonstrating pharmacodynamic relationships in preclinical studies and clinical trials. Current challenges with measuring dystrophin include the variation in protein expression within individual muscle fibers and across whole muscle samples, the presence of pre-existing dystrophin-positive revertant fibers, and trace amounts of residual dystrophin. Immunofluorescence quantification of dystrophin can overcome many of these challenges, but manual quantification of protein expression may be complicated by variations in the collection of images, reproducible scoring of fluorescent intensity, and bias introduced by manual scoring of typically only a few high-power fields. This review highlights the pathology of DMD/BMD, discusses animal models of DMD/BMD, and describes dystrophin biomarker quantitation in DMD/BMD, with several image analysis approaches, including a new automated method, that evaluates protein expression of individual muscle fibers.

show abstract

Section: Evaluating Efficacy Of Therapeutic Candidatesmentioning

confidence: 99%

Duchenne and Becker Muscular Dystrophies: A Review of Animal Models, Clinical End Points, and Biomarker Quantification

Wilson¹,

Faelan²,

Patterson-Kane³

et al. 2017

Toxicol Pathol

Self Cite

View full text Add to dashboard Cite

show abstract

“…To demonstrate issues that can have an effect on the evaluation of fluorescently-labeled muscle sections and manual scoring, Dr. Aeffner briefly reviewed cognitive and visual traps that can influence the estimation of numbers (e.g., of cells or events present on a slide), the perception of staining intensity in general, and specifically the perception of staining intensity and contrast in the fluorescent setting (Aeffner et al , 2017). …”

Section: Introductionmentioning

confidence: 99%

“…Another publication studying pathologists ability to specifically evaluate non-small cell lung cancer biopsies documented that visual assessment of estimated neoplastic cell concentration regularly over-estimates the tumor burden by 10 to 20% (Warth et al , 2012). Other cognitive traps and biases that can have an effect on accuracy and reproducibility of manual pathology scoring results include avoidance of extremes and terminal digit bias (reviewed in Aeffner et al , 2017). …”

Section: Introductionmentioning

confidence: 99%

“…This effect is likely at play when it has been reported that the same membranous staining intensity is perceived as being different, depending upon if it is evaluated next to cytoplasm with staining or devoid of staining (Conway et al , 2008). Readers further interested in this matter are also encouraged to investigate Adelson's Checker-Shadow-Illusion (Albert, 2007, Adelson, 2001, Aeffner et al , 2017). …”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Proceedings of the 2017 National Toxicology Program Satellite Symposium

Elmore

Aeffner

Bangari³

et al. 2017

Toxicol Pathol

Self Cite

View full text Add to dashboard Cite

The 2017 annual National Toxicology Program Satellite Symposium, entitled “Pathology Potpourri,” was held in Montreal, Quebec, Canada, at the Society of Toxicologic Pathology's (STP) 36th annual meeting. The goal of this symposium was to present and discuss challenging diagnostic pathology and/or nomenclature issues. This article presents summaries of the speakers' talks, along with select images that were used by the audience for voting and discussion. Various lesions and other topics covered during the symposium included renal papillary degeneration in perinatally-exposed animals; an atriocaval mesothelioma; an unusual presentation of an alveolar bronchiolar carcinoma; a paraganglioma of the Organ of Zuckerkandl, also called an extra adrenal pheochromocytoma; the use of human muscle samples to illustrate the challenges of manual scoring of fluorescent staining; intertubular spermatocytic seminomas; medical device pathology assessment and discussion of the approval process; collagen induced arthritis; incisor denticles; ameloblast degeneration and poorly mineralized enamel matrix; connective tissue paragangliomas; microcystin LR toxicity; perivascular mast cells in the forebrain thalamus unrelated to treatment; and two cases that provided a review of the International Harmonization of Nomenclature and Diagnostic Criteria (INHAND) bone nomenclature and recommended application of the terminology in routine nonclinical toxicity studies.

show abstract

“…Interestingly, digital computer-assisted methods may improve IHC quantification. However, the availability of these approaches is limited and still needs standardization (18). In cases where manual scoring is severely hampered, an alternative digital method may be considered in quantifying PD-L1 expression.…”

mentioning

confidence: 99%

Reproducibility of PD-L1 assessment in non-small cell lung cancer—know your limits but never stop trying to exceed them

Ilié

Hofman

2017

Transl. Lung Cancer Res.

View full text Add to dashboard Cite

The Gold Standard Paradox in Digital Image Analysis: Manual Versus Automated Scoring as Ground Truth

Abstract: - Awareness of the gold standard paradox is necessary when using traditional pathologist scores to analytically validate a tIA tool because image analysis is used specifically to overcome known sources of bias in visual assessment of tissue sections.

Cited by 148 publications

References 79 publications

Duchenne and Becker Muscular Dystrophies: A Review of Animal Models, Clinical End Points, and Biomarker Quantification

Duchenne and Becker Muscular Dystrophies: A Review of Animal Models, Clinical End Points, and Biomarker Quantification

Proceedings of the 2017 National Toxicology Program Satellite Symposium

Reproducibility of PD-L1 assessment in non-small cell lung cancer—know your limits but never stop trying to exceed them

Contact Info

Product

Resources

About