Development of a diagnostic test set to assess agreement in breast pathology: practical application of the Guidelines for Reporting Reliability and Agreement Studies (GRRAS)

Abstract

BackgroundDiagnostic test sets are a valuable research tool that contributes importantly to the validity and reliability of studies that assess agreement in breast pathology. In order to fully understand the strengths and weaknesses of any agreement and reliability study, however, the methods should be fully reported. In this paper we provide a step-by-step description of the methods used to create four complex test sets for a study of diagnostic agreement among pathologists interpreting breast biopsy specimens. We use the newly developed Guidelines for Reporting Reliability and Agreement Studies (GRRAS) as a basis to report these methods.MethodsBreast tissue biopsies were selected from the National Cancer Institute-funded Breast Cancer Surveillance Consortium sites. We used a random sampling stratified according to woman’s age (40–49 vs. ≥50), parenchymal breast density (low vs. high) and interpretation of the original pathologist. A 3-member panel of expert breast pathologists first independently interpreted each case using five primary diagnostic categories (non-proliferative changes, proliferative changes without atypia, atypical ductal hyperplasia, ductal carcinoma in situ, and invasive carcinoma). When the experts did not unanimously agree on a case diagnosis a modified Delphi method was used to determine the reference standard consensus diagnosis. The final test cases were stratified and randomly assigned into one of four unique test sets.ConclusionsWe found GRRAS recommendations to be very useful in reporting diagnostic test set development and recommend inclusion of two additional criteria: 1) characterizing the study population and 2) describing the methods for reference diagnosis, when applicable.

Region of interest identification and diagnostic agreement in breast pathology

“…The methods for test case identification and the development and recruitment of pathologists have been previously described. 3,7 Briefly, single, representative diagnostic slides from excisional or core breast biopsies of 180 women were included in this pilot study. Each slide was digitally scanned (iScan Coreo, Ventana Medical Systems, Tucson, AZ, USA), and a whole slide image (WSI) was created, allowing the digital virtual slide to be viewed, magnified, and annotated on a computer using a web-based viewer.…”

“…Three experienced breast pathologists, who were involved in the original B-Path study, 3,7 independently interpreted the 180 cases in the digital WSI format using a standardized diagnosis reporting form and a web-based viewer. Each expert recorded an independent diagnosis and digitally marked an ROI that best exemplified the critical features on the slide, supporting the most severe diagnosis.…”

“…The electronic ROI tool allowed the experts to select a rectangular region that could be as small or as large as the pathologist desired. Webinars and in-person meetings of these same three experts were then held, using a modified Delphi approach 7 to establish a final consensus reference diagnosis and one or more consensus ROIs that supported the diagnosis for each case. A consensus diagnosis was achieved for 176 cases using the categorical mapping scheme described in Supplementary Appendix 1.…”

Localization of Diagnostically Relevant Regions of Interest in Whole Slide Images: a Comparative Study

“…Cases included benign without atypia (30 %), atypical ductal hyperplasia (30 %), ductal carcinoma in situ (30 %), and invasive breast cancer (10 %). The methods of case development and data collection from pathologists have been previously described [41][42][43][44] and will be summarized here briefly.…”

“…The breast pathology (B-Path) and digital pathology (digiPATH) study [41][42][43][44] aims are to understand the diagnostic patterns of pathologists and evaluate the accuracy and efficiency of interpretations using glass slides and digital whole slide images. For this purpose, three expert pathologists were invited to interpret a series of 240 breast biopsies on glass or digital media.…”