Analysis of intra‐observer and inter‐observer variability of pathologists for non‐benign thyroid fine needle aspiration cytology according to Bethesda system categories

Abstract

AimTo examine the intra‐ and inter‐observer variability for non‐benign thyroid cytological subcategories according to the Bethesda classification system after the second review.MethodsBetween November 2018 and May 2019, thyroid fine needle aspiration biopsies of 381 nodules were retrospectively evaluated. Among them, 74 non‐benign (category III‐VI) thyroid biopsies, analyzed according to the Bethesda system (pathologist 1:40 vs pathologist 2:34) by two independent pathologists, were reassessed by the same pathologists and by a cytopathologist. In this observer‐blinded study, weighted Cohen's kappa was used to assess the intra‐observer agreement, and Krippendorff's alpha was used to assess the inter‐observer agreement.ResultsAt the first and second evaluations of pathologists 1 and 2, the percentage agreement was 62.5% for pathologist 1 and 58.8% for pathologist 2. The intra‐observer agreement was substantial (κ = 0.705) for pathologist 1, and moderate (κ = 0.447) for pathologist 2. In the second evaluation of pathologist 1 and 2, which was compared with the cytopathologist, the agreement percentage of pathologist 1 with the cytopathologist was 50.0%, and that of pathologist 2 was 56.8%. The inter‐observer agreement was below the lowest acceptable limit for an overall agreement (α = 0.634) among the three raters. The inter‐observer agreement was only acceptable between the cytopathologist and the second pathologist, while it was low between the other raters. In the evaluation of the non‐benign nodules, the mean category score of the cytopathologist was 3.22 and lower than both pathologists (3.73 and 3.58, respectively).ConclusionsThe intra‐observer agreement of pathologists was moderate‐to‐substantial in the evaluation of non‐benign thyroid biopsies according to the Bethesda reporting system. However, the inter‐observer agreement was below the lowest acceptable limit when the cytopathologist was taken as a reference.

Repeat thyroid FNAC: Inter-observer agreement among high- and low-volume centers in Naples metropolitan area and correlation with the EU-TIRADS

“…Firstly, we demonstrated that the overall FNAC results according to the Italian thyroid cytology system were significantly different between the two centers: indeed, as already noted in some studies according to other internationally recognized systems for reporting thyroid cytology ( 20 , 21 ), we observed that the cytopathologists of the HV center tended to score lower the benign nodules and higher those with final diagnosis of malignancy. Perhaps, this was due to the extensive expertise of our thyroid cytopathologists to more easily detect both the benign and malignant cases, compared with operators that work in LV services where thyroid FNAC occupies only a small place in the daily routine.…”

“…However, to our knowledge, limited literature exists on the evaluation of the inter-observer variability of the cytological diagnosis of thyroid nodules according to the three main classification systems (i.e., Bethesda, RCPath, ICCRTC) ( 20 – 29 ). Moreover, few studies were based on raters from institutions with different expertise and case load ( 21 , 29 ).…”

Combining molecular testing and the Bethesda category III:VI ratio for thyroid fine‐needle aspirates: A quality‐assurance metric for evaluating diagnostic performance in a cytopathology laboratory

“…Unfortunately, RoM suffers from wide interinstitutional and interobserver variability, particularly for specimens that fall into the morphologically indeterminate TBS categories: III (atypia of undetermined significance [AUS]), IV (follicular neoplasm or suspicious for follicular neoplasm [SFN), and V (suspicious for malignancy [SUS]). [3][4][5][6][7] In addition, because only a subset of indeterminate thyroid nodules undergo surgical resection, assessment of the RoM is challenging. Beyond the RoM, the only recommendation that the first (2010) edition of TBS made regarding secondary QA metrics was for most practice settings to use an interim goal for the frequency of AUS interpretations of approximately 7% of all thyroid fine-needle aspiration (FNA) diagnoses.…”

“…The RoM is particularly important because it serves both as an indicator of clinical actionability and as the principal quality‐assurance (QA) metric for individual pathologists and pathology laboratories that use TBS. Unfortunately, RoM suffers from wide interinstitutional and interobserver variability, particularly for specimens that fall into the morphologically indeterminate TBS categories: III ( atypia of undetermined significance [AUS]), IV ( follicular neoplasm or suspicious for follicular neoplasm [SFN), and V ( suspicious for malignancy [SUS]) 3‐7 . In addition, because only a subset of indeterminate thyroid nodules undergo surgical resection, assessment of the RoM is challenging.…”

Recent Application of Artificial Intelligence in Non-Gynecological Cancer Cytopathology: A Systematic Review

“…As cytological diagnosis is a more qualitative or subjective mode of interpretation than it is a quantitative analysis, there is a possibility of inter- and intra-observer variation among cytologists’ interpretations [ 50 , 56 ]. For example, even though the Paris system (TPS) was established to standardize the cytological interpretation of urine samples, there are still many conflicting opinions about the criteria for some categories.…”