The effectiveness of pretreatment physics plan review for detecting errors in radiation therapy

Gopan, Olga; Zeng, Jing; Novak, Avrey; Nyflot, Matthew J.; Ford, Eric

doi:10.1118/1.4961010

Cited by 48 publications

(67 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, in considering the “best‐case scenario”, these studies do not measure actual performance of error checking and assume that all the errors that could have been detected are detected. The actual sensitivity of the pretreatment physics plan review in detecting near‐miss events, which were reported to the ILS, was as low as 38% in one study . Second, the present study analyzes the ability of the pretreatment physics plan review to detect specific types of errors.…”

Section: Discussionmentioning

confidence: 87%

“…The correlation coefficient, R , between the two approaches (error simulation vs actual ILS data) was computed to be 0.89, showing that the two measurement techniques were strongly related. Differences between the two datasets may reflect the overreporting of errors in the current approach due to observer bias (reviewers expecting errors in the simulated charts) and the well‐known underreporting of errors in the ILS system as discussed in a prior study …”

Section: Resultsmentioning

confidence: 98%

“…First, this study quantifies the actual sensitivity of a simulated pretreatment physics plan review while previous studies (with the exception of one study) only quantified its potential sensitivity, that is, the “best case scenario” for possible detection rates in an idealized scenario. These previous studies found the potential sensitivity of the pretreatment physics plan review to be 62%, 53%, and 51–80% . However, in considering the “best‐case scenario”, these studies do not measure actual performance of error checking and assume that all the errors that could have been detected are detected.…”

Section: Discussionmentioning

confidence: 99%

See 2 more Smart Citations

Utilizing simulated errors in radiotherapy plans to quantify the effectiveness of the physics plan review

et al. 2018

Self Cite

View full text Add to dashboard Cite

Purpose The review of a radiation therapy plan by a physicist prior to treatment is a standard tool for ensuring the quality of treatments. However, little is known about how well this task is performed in practice. The goal of this study is to present a novel method to measure the effectiveness of physics plan review by introducing simulated errors into computerized “mock” treatment charts and measuring the performance of plan review by physicists. Methods We generated six simulated treatment charts containing multiple errors. To select errors, we compiled a list based on events from a departmental incident learning system and an international incident learning system (SAFRON). Seventeen errors with the highest scores for frequency and severity were included in the simulations included six mock treatment charts. Eight physicists reviewed the simulated charts as they would a normal pretreatment plan review, with each chart being reviewed by at least six physicists. There were 113 data points for evaluation. Observer bias was minimized using a simple error vs hidden error approach, using detectability scores for stratification. The confidence interval for the proportion of errors detected was computed using the Wilson score interval. Results Simulated errors were detected in 67% of reviews [58–75%] (95% confidence interval [CI] in brackets). Of the errors included in the simulated plans, the following error scenarios had the highest detection rates: an incorrect isocenter in DRR (93% [70–99%]), a planned dose different from the prescribed dose (92% [67–99%]) and invalid QA (85% [58–96%]). Errors with low detection rates included incorrect CT dataset (0%, [0–39%]) and incorrect isocenter localization in planning system (38% [18–64%]). Detection rates of errors from simulated charts were compared against observed detection rates of errors from a departmental incident learning system. Conclusions It has been notoriously difficult to quantify error and safety performance in oncology. This study uses a novel technique of simulated errors to quantify performance and suggests that the pretreatment physics plan review identifies some errors with high fidelity while other errors are more challenging to detect. These data will guide future work on standardization and automation. The example process studied here was physics plan review, but this approach of simulated errors may be applied in other contexts as well and may also be useful for training and education purposes.

show abstract

Section: Discussionmentioning

confidence: 87%

Section: Resultsmentioning

confidence: 98%

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Utilizing simulated errors in radiotherapy plans to quantify the effectiveness of the physics plan review

et al. 2018

Self Cite

View full text Add to dashboard Cite

show abstract

“…A fraction (5.8%) of the test cases were altered by manually introducing a known set of potential errors designed for the most recent clinical practice, for example, wrong total dose, wrong fractional dose, and convolved errors such as nonoptimal energy for given site. The introduced errors were selected based on our departmental incident reporting system and the SAFRON system, an international voluntary incident reporting system implemented by the International Atomic Energy Agency, to ensure realistic types of errors and appropriate rates of error . This includes major potential error pathways such as prescription errors.…”

Section: Methodsmentioning

confidence: 99%

Characterization of a Bayesian network‐based radiotherapy plan verification model

et al. 2019

View full text Add to dashboard Cite

Purpose The current process for radiotherapy treatment plan quality assurance relies on human inspection of treatment plans, which is time‐consuming, error prone and oft reliant on inconsistently applied professional judgments. A previous proof‐of‐principle paper describes the use of a Bayesian network (BN) to aid in this process. This work studied how such a BN could be expanded and trained to better represent clinical practice. Methods We obtained 51 540 unique radiotherapy cases including diagnostic, prescription, plan/beam, and therapy setup factors from a de‐identified Elekta oncology information system from the years 2010–2017 from a single institution. Using a knowledge base derived from clinical experience, factors were coordinated into a 29‐node, 40‐edge BN representing dependencies among the variables. Conditional probabilities were machine learned using expectation maximization module using all data except a subset of 500 patient cases withheld for testing. Different classes of errors that were obtained from incident learning systems were introduced to the testing set of cases which were withheld from the dataset used for building the BN. Different sizes of datasets were used to train the network. In addition, the BN was trained using data from different length epochs as well as different eras. Its performance under these different conditions was evaluated by means of Areas Under the receiver operating characteristic Curve (AUC). Results Our performance analysis found AUCs of 0.82, 0.85, 0.89, and 0.88 in networks trained with 2‐yr, 3‐yr 4‐yr and 5‐yr windows. With a 4‐yr sliding window, we found AUC reduction of 3% per year when moving the window back in time in 1‐yr steps. Compared to the 4‐yr window moved back by 4 yrs (2010–2013 vs 2014–2017), the largest component of overall reduction in AUC over time was from the loss of detection performance in plan/beam error types. Conclusions The expanded BN method demonstrates the ability to detect classes of errors commonly encountered in radiotherapy planning. The results suggest that a 4‐yr training dataset optimizes the performance of the network in this institutional dataset, and that yearly updates are sufficient to capture the evolution of clinical practice and maintain fidelity.

show abstract

“…5 Some may argue that these plan review activities can be replaced by automation and artificial intelligence (AI), and that the role of clinical medical physicists therefore becomes less important for the quality and safety of patient care. Such catches can certainly improve the quality of the treatment plan with more accurate dose calculations.…”

mentioning

confidence: 99%

The role of clinical medical physicists in the future: Quality, safety, technology implementation, and enhanced direct patient care

Wang

White²

2019

J Applied Clin Med Phys

View full text Add to dashboard Cite

The effectiveness of pretreatment physics plan review for detecting errors in radiation therapy

Cited by 48 publications

References 24 publications

Utilizing simulated errors in radiotherapy plans to quantify the effectiveness of the physics plan review

Utilizing simulated errors in radiotherapy plans to quantify the effectiveness of the physics plan review

Characterization of a Bayesian network‐based radiotherapy plan verification model

The role of clinical medical physicists in the future: Quality, safety, technology implementation, and enhanced direct patient care

Contact Info

Product

Resources

About