Development and validation of a checklist for use with automatically generated radiotherapy plans

Abstract

To develop a checklist that improves the rate of error detection during the plan review of automatically generated radiotherapy plans. Methods: A custom checklist was developed using guidance from American Association of Physicists in Medicine task groups 275 and 315 and the results of a failure modes and effects analysis of the Radiation Planning Assistant (RPA), an automated contouring and treatment planning tool. The preliminary checklist contained 90 review items for each automatically generated plan. In the first study, eight physicists were recruited from our institution who were familiar with the RPA. Each physicist reviewed 10 artificial intelligencegenerated resident treatment plans from the RPA for safety and plan quality, five of which contained errors. Physicists performed plan checks, recorded errors, and rated each plan's clinical acceptability. Following a 2-week break, physicists reviewed 10 additional plans with a similar distribution of errors using our customized checklist. Participants then provided feedback on the usability of the checklist and it was modified accordingly. In a second study, this process was repeated with 14 senior medical physics residents who were randomly assigned to checklist or no checklist for their reviews. Each reviewed 10 plans, five of which contained errors, and completed the corresponding survey.Results: In the first study, the checklist significantly improved the rate of error detection from 3.4 ± 1.1 to 4.4 ± 0.74 errors per participant without and with the checklist, respectively (p = 0.02). Error detection increased by 20% when the custom checklist was utilized. In the second study, 2.9 ± 0.84 and 3.5 ± 0.84 errors per participant were detected without and with the revised checklist, respectively (p = 0.08). Despite the lack of statistical significance for this cohort, error detection increased by 18% when the checklist was utilized. Conclusion:Our results indicate that the use of a customized checklist when reviewing automated treatment plans will result in improved patient safety.

Radiation Planning Assistant - A Web-based Tool to Support High-quality Radiotherapy in Clinics with Limited Resources

“…Although the user interface has been designed to minimize the risk of inappropriate data entry, such as CT images that do not have sufficient field-of-view or errors in manual data entry, there is still potential for error. Specifically, human error, automation bias (over-reliance on the results), and software error are of concern 9 In general, these reviews should follow the same process as is followed for the review of clinical plans by physicists and radiation oncologists, although this may be supported by the use of checklists developed specifically to supplement manual review of automatically created treatment plans 24 .…”

Addressing the Global Expertise Gap in Radiation Oncology: The Radiation Planning Assistant

“… 24 We also encourage the use of checklists, which can improve the rate of error detection by around 20%. 26 Automated quality assurance. Automated verification steps may substantially reduce the risk of automated planning.…”

“… 24 We also encourage the use of checklists, which can improve the rate of error detection by around 20%. 26 …”

Enhancing patient safety in radiotherapy: Implementation of a customized electronic checklist for radiation therapists

“…Radiotherapy is a complex treatment modality that uses ionising radiation to treat, palliate and cure cancer patients [1] . The radiotherapy process involves the synergic work of multiple professionals and develops in several steps: planning CT acquisition, targets and organs at risk delineation, optimization of a treatment plan and treatment delivery [1] , [2] , [3] . To ensure safe and successful radiotherapy, the treatment accuracy must be guaranteed: it is crucial that the correct dose is delivered to the planned target volume of the right patient [3] .…”