Reducing Inter-Rater Variability in the Assessment of Nuchal Translucency Image Quality

Abstract

Introduction: Standardization of first-trimester nuchal translucency (NT) image acquisition is crucial to the success of screening for Down syndrome. Rigorous audit of operator performance and constructive feedback from assessors maintain standards. This process relies on good inter-rater agreement on image assessment. We describe the Australian approach to NT image assessment and evaluate the impact of a targeted intervention on inter-rater agreement. Methods: Between 2002 and 2008 a group of experienced practitioners met nine times to compare their assessment of a series of NT images. Each assessor had previously scored the images according to a system described in 2002. Inter-rater agreement was evaluated before and after an intervention where the assessors were required to refer to a detailed resource manual designed to reduce the subjectivity inherent in image assessment. Results: There was a statistical improvement in inter-rater agreement for all elements of image assessment (original scores and individual component scores) after the intervention, apart from horizontal fetal position. However, even after the intervention, inter-rater agreement levels generally remained moderate (kappa range: 0.14–0.58). Conclusions: This study has shown that provision of detailed resource documentation to experienced assessors can significantly improve inter-rater agreement in all facets of NT image assessment. It also highlights areas of image assessment that require critical review. It is recommended that all audit bodies regularly review their inter-rater agreement to ensure consistent feedback to operators who submit images for expert peer review.

Quality management of nuchal translucency ultrasound measurement in Australia

“…The audit cycle involves defining a standard that can be used as a benchmark, assessment of a practitioner's work against that standard, determining areas of work that need improvement, acting on these findings, and then re‐auditing to assess if improvement has occurred . NT measurement lends itself well to audit as measurements follow a clearly described distribution and images are obtained using internationally standardised guidelines allowing for the development of scoring systems …”

Updated Guidelines for the Medical Assessment and Care of Children Who May Have Been Sexually Abused

“…The cost of misdiagnosis can be both financial, in the case of expensive medical procedures, and societal, if child abuse is inaccurately diagnosed based on an examiner's misinterpretation of physical findings. Imaged-based specialties such as radiology and pathology have studied inter-rater reliability issues and have proposed methodology for improvement [41][42][43]. While the child's history remains the most important piece of evidence in child sexual abuse evaluations, physical findings resulting from sexual abuse, when present, are important in the investigative and legal arena.…”