Pathologists should probably forget about kappa. Percent agreement, diagnostic specificity and related metrics provide more clinically applicable measures of interobserver variability

Marchevsky, Alberto M.; Walts, Ann E.; Lissenberg‐Witte, Birgit I.; Thunnissen, Erik

doi:10.1016/j.anndiagpath.2020.151561

Cited by 19 publications

(16 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is also little consensus on the ideal method for measuring observer agreement in pathology diagnosis. It has been suggested that both percentages of agreement and ĸ -statistics do not take into account the prevalence of a particular diagnosis in a set of cases, or completely rule out concordances due to chance [ 33 , 34 ]. Validity of the cut-offs that are used to interpret levels of agreement from ĸ -values has also been challenged [ 30 , 35 ].…”

Section: Discussionmentioning

confidence: 99%

Histological interpretation of differentiated vulvar intraepithelial neoplasia (dVIN) remains challenging—observations from a bi-national ring-study

et al. 2021

View full text Add to dashboard Cite

Differentiated vulvar intraepithelial neoplasia (dVIN) is a premalignant lesion that is known to progress rapidly to invasive carcinoma. Accurate histological diagnosis is therefore crucial to allow appropriate treatment. To identify reliable diagnostic features, we evaluated the inter-observer agreement in the histological assessment of dVIN, among a bi-national, multi-institutional group of pathologists. Two investigators from Erasmus MC selected 36 hematoxylin-eosin-stained glass slides of dVIN and no-dysplasia, and prepared a list of 15 histological features of dVIN. Nine participating pathologists (i) diagnosed each slide as dVIN or no-dysplasia, (ii) indicated which features they used for the diagnosis, and (iii) rated these features in terms of their diagnostic usefulness. Diagnoses rendered by > 50% participants were taken as the consensus (gold standard). p53-immunohistochemistry (IHC) was performed for all cases, and the expression patterns were correlated with the consensus diagnoses. Kappa (ĸ)-statistics were computed to measure inter-observer agreements, and concordance of the p53-IHC patterns with the consensus diagnoses. For the diagnosis of dVIN, overall agreement was moderate (ĸ = 0.42), and pair-wise agreements ranged from slight (ĸ = 0.10) to substantial (ĸ = 0.73). Based on the levels of agreement and ratings of usefulness, the most helpful diagnostic features were parakeratosis, cobblestone appearance, chromatin abnormality, angulated nuclei, atypia discernable under × 100, and altered cellular alignment. p53-IHC patterns showed substantial concordance (ĸ = 0.67) with the consensus diagnoses. Histological interpretation of dVIN remains challenging with suboptimal inter-observer agreement. We identified the histological features that may facilitate the diagnosis of dVIN. For cases with a histological suspicion of dVIN, consensus-based pathological evaluation may improve the reliability of the diagnosis.

show abstract

Section: Discussionmentioning

confidence: 99%

Histological interpretation of differentiated vulvar intraepithelial neoplasia (dVIN) remains challenging—observations from a bi-national ring-study

et al. 2021

View full text Add to dashboard Cite

show abstract

“…The Big Five personality domains, which are scored using the NEO-PI-r Big Five personality test. The diagnostic accuracy of the pathologists was expressed in sensitivity, specificity and percent agreement, according to a method proposed by Marchevsky et al [18] To calculate the diagnostic accuracy, a majority score for each tumor was determined, which was defined as the category (<1%, 1-49% or ≥ 50%) in which the majority of the pathologists score a tumor.…”

Section: Tablementioning

confidence: 99%

“…[36] To the best of our knowledge, this current study is the first to evaluate PD-L1 immunoscoring using sensitivity and specificity as proposed by Marchevsky and colleagues. [18].…”

Section: Tablementioning

confidence: 99%

The impact of a pathologist’s personality on the interobserver variability and diagnostic accuracy of predictive PD-L1 immunohistochemistry in lung cancer

Butter

Hondelink

Elswijk³

et al. 2022

Lung Cancer

Self Cite

View full text Add to dashboard Cite

“…We also note that, possibly as a result, one current trend in ILD diagnostics is to discard specific histologic categories in favor of a more general progressive fibrotic phenotype [ 27 ]. It may also be that there are no fixed borders among fibrotic ILD and that all such efforts at distinction may fail due to lack of underlying discrete categories [ 28 ].…”

Section: Discussionmentioning

confidence: 99%

Reliability of histopathologic diagnosis of fibrotic interstitial lung disease: an international collaborative standardization project

et al. 2021

View full text Add to dashboard Cite

Background Current interstitial lung disease (ILD) diagnostic guidelines assess criteria across clinical, radiologic and pathologic domains. Significant interobserver variation in histopathologic evaluation has previously been shown but the specific source of these discrepancies is poorly documented. We sought to document specific areas of difficulty and develop improved criteria that would reduce overall interobserver variation. Methods Using an internet-based approach, we reviewed selected images of specific diagnostic features of ILD histopathology and whole slide images of fibrotic ILD. After an initial round of review, we confirmed the presence of interobserver variation among our group. We then developed refined criteria and reviewed a second set of cases. Results The initial round reproduced the existing literature on interobserver variation in diagnosis of ILD. Cases which were pre-selected as inconsistent with usual interstitial pneumonia/idiopathic pulmonary fibrosis (UIP/IPF) were confirmed as such by multi-observer review. Cases which were thought to be in the spectrum of chronic fibrotic ILD for which UIP/IPF were in the differential showed marked variation in nearly all aspects of ILD evaluation including extent of inflammation and extent and pattern of fibrosis. A proposed set of more explicit criteria had only modest effects on this outcome. While we were only modestly successful in reducing interobserver variation, we did identify specific reasons that current histopathologic criteria of fibrotic ILD are not well defined in practice. Conclusions Any additional classification scheme must address interobserver variation in histopathologic diagnosis of fibrotic ILD order to remain clinically relevant. Improvements to tissue-based diagnostics may require substantial resources such as larger datasets or novel technologies to improve reproducibility. Benchmarks should be established for expected outcomes among clinically defined subgroups as a quality metric.

show abstract

Pathologists should probably forget about kappa. Percent agreement, diagnostic specificity and related metrics provide more clinically applicable measures of interobserver variability

Cited by 19 publications

References 44 publications

Histological interpretation of differentiated vulvar intraepithelial neoplasia (dVIN) remains challenging—observations from a bi-national ring-study

Histological interpretation of differentiated vulvar intraepithelial neoplasia (dVIN) remains challenging—observations from a bi-national ring-study

The impact of a pathologist’s personality on the interobserver variability and diagnostic accuracy of predictive PD-L1 immunohistochemistry in lung cancer

Reliability of histopathologic diagnosis of fibrotic interstitial lung disease: an international collaborative standardization project

Contact Info

Product

Resources

About