Utilizing simulated errors in radiotherapy plans to quantify the effectiveness of the physics plan review

Abstract

Purpose The review of a radiation therapy plan by a physicist prior to treatment is a standard tool for ensuring the quality of treatments. However, little is known about how well this task is performed in practice. The goal of this study is to present a novel method to measure the effectiveness of physics plan review by introducing simulated errors into computerized “mock” treatment charts and measuring the performance of plan review by physicists. Methods We generated six simulated treatment charts containing multiple errors. To select errors, we compiled a list based on events from a departmental incident learning system and an international incident learning system (SAFRON). Seventeen errors with the highest scores for frequency and severity were included in the simulations included six mock treatment charts. Eight physicists reviewed the simulated charts as they would a normal pretreatment plan review, with each chart being reviewed by at least six physicists. There were 113 data points for evaluation. Observer bias was minimized using a simple error vs hidden error approach, using detectability scores for stratification. The confidence interval for the proportion of errors detected was computed using the Wilson score interval. Results Simulated errors were detected in 67% of reviews [58–75%] (95% confidence interval [CI] in brackets). Of the errors included in the simulated plans, the following error scenarios had the highest detection rates: an incorrect isocenter in DRR (93% [70–99%]), a planned dose different from the prescribed dose (92% [67–99%]) and invalid QA (85% [58–96%]). Errors with low detection rates included incorrect CT dataset (0%, [0–39%]) and incorrect isocenter localization in planning system (38% [18–64%]). Detection rates of errors from simulated charts were compared against observed detection rates of errors from a departmental incident learning system. Conclusions It has been notoriously difficult to quantify error and safety performance in oncology. This study uses a novel technique of simulated errors to quantify performance and suggests that the pretreatment physics plan review identifies some errors with high fidelity while other errors are more challenging to detect. These data will guide future work on standardization and automation. The example process studied here was physics plan review, but this approach of simulated errors may be applied in other contexts as well and may also be useful for training and education purposes.

A Blinded, Prospective Study of Error Detection During Physician Chart Rounds in Radiation Oncology

“…A median of 2 problematic plans were presented weekly (range, 1-4). In 75% of problematic plans, the severity score 19 was >7 corresponding to expected potentially serious toxicity or tumor underdose (range, 4-10), and in 75% of problematic plans the detectability score 19 was <7 corresponding to a !95% expected likelihood of detection (range, [3][4][5][6][7][8][9][10].…”

“…This finding is consistent with detection rates resulting from similar forms of peer review. 7 The problematic plans included were considered serious errors and were, by design, highly detectable in 17 of 20 instances. We conclude that the detection of clinically significant problematic plans during chart rounds could be significantly improved.…”

“…6 Additionally, even errors with a high likelihood of detection upstream of physician chart rounds in the QA workflow, such as planning on the wrong scan, can pass through QA to the point of treatment delivery. 7,8 Physician peer review is often one of the final quality assurance steps for assessing plan quality and plan safety 9 and can provide a final "stop the line" moment for some of these previously reviewed treatment planning details. Physician chart rounds should function as a broad and effective net for error detection in the radiation oncology workflow.…”

“…An institutional study of simulated error found that physics chart checks missed one-third of errors, despite the unblinded nature of that study. 7 When a retrospective review of reported incidents was analyzed, only 62% of errors that could have been identified by physics chart checks were actually identified 17 by this QA and patient safety measure.…”

Automatic Verification of Beam Apertures for Cervical Cancer Radiation Therapy

“…Reviewing the treatment plan is a timeconsuming task, and although it is considered one of the most effective forms of QA, it still does not reliably catch all errors that occur in treatment planning. [3][4][5] Automating the plan review process could improve the effectiveness of error detection by drawing reviewers' attention to aspects of the treatment plan that are incorrect or suboptimal and sparing them from reviewing every minute plan detail.…”