Inter-observer and intra-observer agreement of Bosniak classification of cystic renal masses: Comparison between original version to version 2019 and effect of an online support calculator

Abstract

Osman et al Original vs. 2019 version of Bosniak classification of cystic renal masses 2 © 2021 Canadian Urological Association MethodsThis IRB approved cross-sectional study evaluated 103 consecutively identified cystic (≤25% enhancing (1)) masses: 70.9% (73/103) with pathological confirmation (80.8% [59/73] malignant, 19.2% [14/73] benign), and 29.1% (30/103) Bosniak 2/2F masses classified by an expert genitourinary radiologist each with minimum 5 years stability on follow-up (1); the latter group added to balance distribution of cystic masses among the Bosniak categories, Figure 1. Cystic masses were identified through a PACS and pathology database search for the term 'Bosniak' between 2010-2020. All masses were imaged with renal protocol CT or MRI. For patients with both CT and MRI, MRI was evaluated since MRI is known to result in a higher Bosniak class (6). A portion of the study population has been evaluated previously [BLINDED]; however, the purpose of specifically comparing agreement between the two systems over time with and without use of a support calculator has not been previously studied.Three blinded fellowship-trained abdominal radiologists (R2-R4, years of experience 5-8) independently assigned Bosniak.original Class first and then Bosniak.v2019 Class second during one session. After a ≥2-month washout period, R2-R4 re-assigned Bosniak.original and Bosniak.v2019 Class during a second session. Before the second session, radiologists were provided with a pictorial review of the Bosniak.v2019 system, decision support flow diagrams (5) and an on-line support calculator developed for Bosniak.v2019 (bosniakcalculator.herokuapp.com). The radiologists jointly trialed the calculator using 5 test cases in conjunction with R1. Inter-observer and intra-observer agreement were calculated using Cohen's Kappa and agreement between Bosniak.original and Bosniak.v2019 was also calculated for both sessions. Kappa values of 0-2.0 indicated slight agreement, 2.1-4.0 fair agreement, 4.1-6.0 moderate agreement, 6.1-8.0 substantial agreement and 8.1-1.0 almost perfect agreement. ResultsA summary of inter-observer and intra-observer agreement, using Bosniak.original and Bosniak.v2019 is provided in Table 1. In the first session, marginally higher moderate interobserver agreement was noted comparing Bosniak.v2019 (K=0.44) to Bosniak.original (K=0.39). There was no change in inter-observer agreement comparing Bosniak.v2019 classification with (K=0.42) or without the on-line calculator (K=0.44). Comparatively, interobserver agreement for Bosniak.original decreased in the second session (K=0.25). Within reader intra-observer agreement was similar for both Bosniak.orginal and Bosniak.v2019 and within reader level of agreement comparing class designation using Bosniak.original and Bosniak.v2019 was also similar for both sessions, Table 2. CUAJ -ResearchLetter Osman et al Original vs. 2019 version of Bosniak classification of cystic renal masses 3 © 2021 Canadian Urological Association DiscussionBosniak.v2019 presents a major r...

Validation of the updated Bosniak classification (2019) in pathologically confirmed CT-categorised cysts

“…Previous studies have either demonstrated higher interobserver agreement with v2019 compared to v2005 or no significant difference. 16,[18][19][20][21][22] nevertheless, it must be stated that the inter-observer agreement rate remains low and is a concern going forward. For example, Yan et al quote a kappa ranging between 0.26 and 0.47 and Shampain et al report a gwet coefficient of 0.50.…”

2023 UPDATE – Canadian Urological Association guideline: Management of cystic renal lesions

“…25 Although its clinical impact and benefit of the inter-rater variability remain to be well-studied, the panel members believe that the v2019 classification is currently the preferred Bosniak classification. [26][27][28][29][30] Importantly, if there is disagreement or doubt regarding the classification of a complex renal cyst, it is the panel's opinion that such cases should be presented at a multidisciplinary meeting if this has the potential to alter management.…”