Six of one, half a dozen of the other: A measure of multidisciplinary inter/intra-rater reliability of the society for fetal urology and urinary tract dilation grading systems for hydronephrosis

Rickard, Mandy; Easterbrook, Bethany; Kim, Soojin; Farrokhyar, Forough; Stein, Nina Rodrigues; Arora, Sourabh; Belostotsky, Vladimir; DeMaria, Jorge; Lorenzo, Armando J.; Braga, Luis H.

doi:10.1016/j.jpurol.2016.09.005

Cited by 40 publications

(19 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When comparing mild (0, I, II) and severe (III, IV) HN images, our model achieved an average accuracy of 78%, which is well above chance level. When the model discriminated between moderate grades (II and III), which is less reliable for physicians (12,13), performance only dropped to 71%. There is no direct comparison to be made against physician accuracy, however, considering the known difficulties in distinguishing between moderate HN grades (12,13), these results are encouraging.…”

Section: Binary Classification Performancementioning

confidence: 94%

“…Further investigation into the output of our model reveals that there are many borderline images where there is not an obvious choice for which class the image belongs to (e.g., Figures 3A,C). In cases such as these where two grades possible are, it must choose a single HN grade according to the SFU system, much like a physician (12,13).…”

Section: Five-way Classification Performancementioning

confidence: 99%

“…Discriminating between moderate HN grades is known to be challenging (12,13), and therefore we wanted to investigate our model's performance on this same task. When comparing mild (0, I, II) and severe (III, IV) HN images, our model achieved an average accuracy of 78%, which is well above chance level.…”

Section: Binary Classification Performancementioning

confidence: 99%

“…Given the need for accurate and unambiguous classification of HN, numerous HN grading systems have been developed (11). However, poor inter-rater reliability (12,13), particularly for intermediate HN grades, suggests that grading still relies on subjective interpretation of ultrasound images, as clear and objective criteria have not been fully established.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Using Deep Learning Algorithms to Grade Hydronephrosis Severity: Toward a Clinical Adjunct

et al. 2020

Self Cite

View full text Add to dashboard Cite

Grading hydronephrosis severity relies on subjective interpretation of renal ultrasound images. Deep learning is a data-driven algorithmic approach to classifying data, including images, presenting a promising option for grading hydronephrosis. The current study explored the potential of deep convolutional neural networks (CNN), a type of deep learning algorithm, to grade hydronephrosis ultrasound images according to the 5-point Society for Fetal Urology (SFU) classification system, and discusses its potential applications in developing decision and teaching aids for clinical practice. We developed a five-layer CNN to grade 2,420 sagittal hydronephrosis ultrasound images [191 SFU 0 (8%), 407 SFU I (17%), 666 SFU II (28%), 833 SFU III (34%), and 323 SFU IV (13%)], from 673 patients ranging from 0 to 116.29 months old (M age = 16.53, SD = 17.80). Five-way (all grades) and two-way classification problems [i.e., II vs. III, and low (0-II) vs. high (III-IV)] were explored. The CNN classified 94% (95% CI, 93-95%) of the images correctly or within one grade of the provided label in the five-way classification problem. Fifty-one percent of these images (95% CI, 49-53%) were correctly predicted, with an average weighted F1 score of 0.49 (95% CI, 0.47-0.51). The CNN achieved an average accuracy of 78% (95% CI, 75-82%) with an average weighted F1 of 0.78 (95% CI, 0.74-0.82) when classifying low vs. high grades, and an average accuracy of 71% (95% CI, 68-74%) with an average weighted F1 score of 0.71 (95% CI, 0.68-0.75) when discriminating between grades II vs. III. Our model performs well above chance level, and classifies almost all images either correctly or within one grade of the provided label. We have demonstrated the applicability of a CNN approach to hydronephrosis ultrasound image classification. Further investigation into a deep learning-based clinical adjunct for hydronephrosis is warranted.

show abstract

Section: Binary Classification Performancementioning

confidence: 94%

Section: Five-way Classification Performancementioning

confidence: 99%

Section: Binary Classification Performancementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Using Deep Learning Algorithms to Grade Hydronephrosis Severity: Toward a Clinical Adjunct

et al. 2020

Self Cite

View full text Add to dashboard Cite

show abstract

“…The SFU grading system has problems with inter-and intra-rater variability [16,29]. We tried to minimize the problems by adopting a second single reviewer for all ultrasonography ndings; however, the reliability problem for grades 2 and 3 remains [16,30]. Further studies are needed to overcome these limitations and con rm the ndings of the present study.…”

Section: Discussionmentioning

confidence: 79%

Appropriate timing for performing abdominal ultrasonography and termination of observation follow-up for antenatal grades 1or 2 hydronephrosis

Nakane

Mizuno²,

Kato

et al. 2020

Preprint

View full text Add to dashboard Cite

Background: Most cases of antenatal Society of Fetal Urology (SFU) grades 1 or 2 hydronephrosis (HN) improve or resolve spontaneously with conservative treatment. However, there is no consensus on the duration of follow-up for grades 1 or 2 HN. The aim of this study was to determine the need for a continuous follow-up period and to propose a new follow-up protocol for children with antenatal grades 1 or 2 HN.Methods: Subjects underwent ultrasonography examination according to the SFU classification. We retrospectively evaluated data from 112 patients with postnatal grade 1 HN and 69 with grade 2 HN diagnosed between January 2010 and December 2017. We examined the change in HN grades using serial ultrasonography. The Kaplan-Meier method was used to show the effect of HN grade on the rate of HN changes.Results: The mean follow-up duration was 44.9 ± 36.4 months (range, 12–274 months). Initial SFU grade 1 HN disappeared in 47.0% of cases at 12 months, 66.4% at 24 months and 73.2% at 48 months. Initial SFU grade 2 HN improved in 74.7% of cases at 12 months, 88.3% at 24 months and 89.5% at 48 months. However, 14.6% of SFU grade 1 and 2.8% of SFU grade 2 cases increased in grade, and 99.3% of these cases worsened within the first 6 months. No cases with increased grade required pyeloplasty. Initial disappearance and later reappearance of HN occurred in 40.5% of SFU grade 1 and 2 cases. The mean duration of later reappearance was 39.1 ± 36.2 months (range, 12–137 months). No reappearance was detected after more than 1 year.Conclusions: Ultrasonography within the first 6 months was necessary for management of children with antenatal grades 1 or 2 HN because some patients showed worsening. Thereafter, it was considered safe to increase the follow-up intervals for stable cases. Most cases resolved spontaneously; however, a few cases reappeared within 1 year. Therefore, ultrasonography after 1 year was necessary for children with HN that had spontaneously disappeared. The appropriate time to terminate follow-up was considered to be 1 year or more after HN disappearance was confirmed.

show abstract

Interobserver Reliability of the Antenatal Consensus Classification System for Urinary Tract Dilatation

Nelson

Heller

Benson

et al. 2019

J of Ultrasound Medicine

View full text Add to dashboard Cite

Objectives The consensus classification system for urinary tract dilatation (UTD) was designed to be more objective and reproducible than previously used systems. We sought to evaluate interobserver reliability of UTD components and overall scores in a prenatal population undergoing third‐trimester ultrasound examinations. Methods We retrospectively identified patients who underwent antenatal ultrasound examinations for UTD between 28 and 40 weeks’ gestation. All images from individual studies of 300 fetuses were reviewed independently by 5 experienced sonologists (1 maternal‐fetal medicine specialist and 4 radiologists). Urinary tract dilatation scores (normal, A1, or A2/3) and Society for Fetal Urology (SFU) scores were assigned. Interobserver agreement between raters was evaluated with the Fleiss κ statistic. Results Overall interobserver agreement for the antenatal UTD risk score showed substantial agreement among all 5 readers (κ = 0.657 [95% confidence interval, 0.632, 0.683]; P < .001). All 5 readers applied the same UTD risk score in 53.7% of cases. Some variability in the antenatal UTD score and individual elements was observed. At least 2 UTD risk scores were assigned to a specific individual patient in 46.3% of cases (139 of 300), and all 3 UTD risk scores were assigned to a specific individual patient in 1.7% of cases (5 of 300). In 18.0% of cases (54 of 300), at least 2 readers assigned a UTD score different from that assigned by the other readers. Agreement was lowest for parenchymal appearance (κ = 0.225). Agreement for the SFU system was fair (κ = 0.368; P < .001). Conclusions Interobserver agreement for the antenatal UTD grading system was substantial. Compared to the SFU system, the antenatal UTD system showed better agreement among readers.

show abstract

Six of one, half a dozen of the other: A measure of multidisciplinary inter/intra-rater reliability of the society for fetal urology and urinary tract dilation grading systems for hydronephrosis

Cited by 40 publications

References 17 publications

Using Deep Learning Algorithms to Grade Hydronephrosis Severity: Toward a Clinical Adjunct

Using Deep Learning Algorithms to Grade Hydronephrosis Severity: Toward a Clinical Adjunct

Appropriate timing for performing abdominal ultrasonography and termination of observation follow-up for antenatal grades 1or 2 hydronephrosis

Interobserver Reliability of the Antenatal Consensus Classification System for Urinary Tract Dilatation

Contact Info

Product

Resources

About