Use of Receiver Operating Curve Analysis and Machine Learning With an Independent Dose Calculation System Reduces the Number of Physical Dose Measurements Required for Patient-Specific Quality Assurance

Hasse, Katelyn; Scholey, Jonathan M.; Ziemer, Benjamin; Natsuaki, Yutaka; Morin, Olivier; Solberg, T.D.; Hirata, Emily; Valdés, Gilmer; Witztum, A.

doi:10.1016/j.ijrobp.2020.10.035

Cited by 5 publications

(13 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The ROC curve was used in evaluating the performance across different criteria ( 19 , 33 , 34 ). This value is plotted as the true positive rate (sensitivity) varies with false positive rate (1–specificity).…”

Section: Methodsmentioning

confidence: 99%

“…For the MC calculation, variable thresholds were characterized with different criteria (3%/3 mm, 3%/ 2 mm, and 2%/2 mm). The classification accuracy was evaluated with sensitivity and specificity analyzes (19,(33)(34)(35). All plans can be divided into four categories labeled by ArcCHECK measurements and ArcherQA calculation: true positive (TP), false positive (FP), false negative (FN) and true negative (TN) as illustrated in Figure 2.…”

Section: Criterion For Selecting Treatment Plans For Measurementmentioning

confidence: 99%

See 1 more Smart Citation

Treatment plan prescreening for patient-specific quality assurance measurements using independent Monte Carlo dose calculations

Zhang

Liu

et al. 2022

Front. Oncol.

View full text Add to dashboard Cite

PurposeThis study proposes a method to identify plans that failed patient-specific quality assurance (QA) and attempts to establish a criterion to prescreen treatment plans for patient-specific QA measurements with independent Monte Carlo dose calculations.Materials and methodsPatient-specific QA results measured with an ArcCHECK diode array of 207 patients (head and neck: 25; thorax: 61; abdomen: 121) were retrospectively analyzed. All patients were treated with the volumetric modulated arc therapy (VMAT) technique and plans were optimized with a Pinnacle v16.2 treatment planning system using an analytical algorithm-based dose engine. Afterwards, phantom verification plans were designed and recalculated by an independent GPU-accelerated Monte Carlo (MC) dose engine, ArcherQA. Moreover, sensitivity and specificity analyzes of gamma passing rates between measurements and MC calculations were carried out to show the ability of MC to monitor failing plans (ArcCHECK 3%/3 mm,<90%), and attempt to determine the appropriate threshold and gamma passing rate criterion utilized by ArcherQA to prescreen treatment plans for ArcCHECK measurements. The receiver operator characteristic (ROC) curve was also utilized to characterize the performance of different gamma passing rate criterion used by ArcherQA.ResultsThe thresholds for 100% sensitivity to detect plans that failed patient-specific QA by independent calculation were 97.0%, 95.4%, and 91.0% for criterion 3%/3 mm, 3%/2 mm, and 2%/2 mm, respectively, which corresponded to specificities of 0.720, 0.528, and 0.585, respectively. It was shown that the 3%/3 mm criterion with 97% threshold for ArcherQA demonstrated perfect sensitivity and the highest specificity compared with other criteria, which may be suitable for prescreening treatment plans treated with the investigated machine to implement measurement-based patient-specific QA of patient plans. In addition, the area under the curve (AUC) calculated from ROC analysis for criterion 3%/3 mm, 3%/2 mm, and 2%/2 mm used by ArcherQA were 0.948, 0.924, and 0.929, respectively.ConclusionsIndependent dose calculation with the MC-based program ArcherQA has potential as a prescreen treatment for measurement-based patient-specific QA. AUC values (>0.9) showed excellent classification accuracy for monitoring failing plans with independent MC calculations.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Criterion For Selecting Treatment Plans For Measurementmentioning

confidence: 99%

Treatment plan prescreening for patient-specific quality assurance measurements using independent Monte Carlo dose calculations

Zhang

Liu

et al. 2022

Front. Oncol.

View full text Add to dashboard Cite

show abstract

“…These algorithms are unique in that they can predict data without explicit instructions from the modeler 1,5,8,9 . XGBoost and Random Forest, two well‐known machine learning algorithms, have been shown to be significantly more accurate than linear and logistic regression 4,10,11 . However, in the case of machine learning (ML) algorithms, decreased interpretability comes at the cost of increased predictive accuracy, and ML algorithms are frequently referred to as “black boxes” because they cannot be understood 1,5,7‐11 .…”

Section: Introductionmentioning

confidence: 99%

“…XGBoost and Random Forest, two well‐known machine learning algorithms, have been shown to be significantly more accurate than linear and logistic regression 4,10,11 . However, in the case of machine learning (ML) algorithms, decreased interpretability comes at the cost of increased predictive accuracy, and ML algorithms are frequently referred to as “black boxes” because they cannot be understood 1,5,7‐11 . When evaluating these methods, researchers heavily rely on model metrics like area under the receiver operator characteristic curve (AUROC), sensitivity, specificity, and accuracy 1,2,4 …”

Section: Introductionmentioning

confidence: 99%

Computation of the distribution of model accuracy statistics in machine learning: Comparison between analytically derived distributions and simulation‐based methods

Huang

2023

Health Science Reports

View full text Add to dashboard Cite

Background and Aims: All fields have seen an increase in machine-learning techniques. To accurately evaluate the efficacy of novel modeling methods, it is necessary to conduct a critical evaluation of the utilized model metrics, such as sensitivity, specificity, and area under the receiver operator characteristic curve (AUROC). For commonly used model metrics, we proposed the use of analytically derived distributions (ADDs) and compared it with simulation-based approaches.Methods: A retrospective cohort study was conducted using the England National Health Services Heart Disease Prediction Cohort. Four machine learning models (XGBoost, Random Forest, Artificial Neural Network, and Adaptive Boost) were used. The distribution of the model metrics and covariate gain statistics were empirically derived using boot-strap simulation (N = 10,000). The ADDs were created from analytic formulas from the covariates to describe the distribution of the model metrics and compared with those of bootstrap simulation.Results: XGBoost had the most optimal model having the highest AUROC and the highest aggregate score considering six other model metrics. Based on the Anderson-Darling test, the distribution of the model metrics created from bootstrap did not significantly deviate from a normal distribution. The variance created from the ADD led to smaller SDs than those derived from bootstrap simulation, whereas the rest of the distribution remained not statistically significantly different.Conclusions: ADD allows for cross study comparison of model metrics, which is usually done with bootstrapping that rely on simulations, which cannot be replicated by the reader.

show abstract

“…It performs a full recalculation of dose on the patient CT and allows for quality assurance of the treatment plan by offering a "delivered dose" calculation, generated using the M3D model and treatment machine's log files. 18,19 In a recent study, Hasse et al 20 investigated the possibility to use M3D dose calculation software to reduce the number of physical measurements and the required amount of on-site personnel, during corona virus disease, while maintaining patient safety.…”

Section: Introductionmentioning

confidence: 99%

Is it still necessary to perform measured based pre‐treatment patient‐specific QA for SRS HyperArc treatments?

Cavalli,

Bonanno,

Borzì

et al. 2023

J Applied Clin Med Phys

View full text Add to dashboard Cite

PurposeThe Mobius3D system was validated as a modern secondary check dosimetry system.In particular, our objective has been to assess the suitability of the M3D as pre‐treatment patient‐specific Quality Assurance (QA) tool for Stereotactic Radiosurgery (SRS) HyperArc (HA) treatments. We aimed to determine whether Mobius3D could safely replace the measurements‐based patient‐specific QA for this type of treatment.Methods30 SRS HA treatment plans for brain were selected. The dose distributions, calculated by Mobius and our routinely used algorithm (AcurosXB v.15.6), were compared using gamma analysis index and DVH parameters based on the patient's CT dataset.All 30 plans were then delivered across the ionization chamber in a homogeneous phantom and the measured dose was compared with both M3D and TPS calculated one.The plans were delivered and verified in terms of PSQA using the electronic portal imaging device (EPID) with Portal Dosimetry (PD) and myQA SRS (IBA Dosimetry) detector.Plans that achieved a global gamma passing rate (GPR) ≥ 97% based on 2%/2 mm criteria, with both Mobius3D and the conventional methods were evaluated acceptable. Finally, we assessed the capability of the M3D system to detect errors related to the position of the Multi‐Leaf Collimator (MLC) in comparison to the analyzed measurement‐based systems.ResultsNo relevant differences were observed in the comparison between the dose calculated on the CT‐dataset by M3D and the TPS. Observed discrepancies are imputable to different used algorithms, but no discrepancies related to goodness of plans have been found.Average differences between calculated (M3D and TPS) vs measured dose with ionization chamber were 2.5% (from 0.41% to 3.2%) and 1.81% (from 0.66% to 2.65%), for M3D and TPS, respectively.All plans passed with a gamma passing rate > 97% using conventional PSQA methods with a gamma criterion of 2% dose difference and 2 mm distance‐to‐agreement. The average gamma passing rate for the M3D system was determined to be 99.4% (from 97.3% to 100%). Results from this study also demonstrated Mobius has better error detectability than conventional measurement‐based systems.ConclusionOur study shows Mobius3D could be a suitable alternative to conventional measured based QA methods for SRS HyperArc treatments.

show abstract

Use of Receiver Operating Curve Analysis and Machine Learning With an Independent Dose Calculation System Reduces the Number of Physical Dose Measurements Required for Patient-Specific Quality Assurance

Cited by 5 publications

References 16 publications

Treatment plan prescreening for patient-specific quality assurance measurements using independent Monte Carlo dose calculations

Treatment plan prescreening for patient-specific quality assurance measurements using independent Monte Carlo dose calculations

Computation of the distribution of model accuracy statistics in machine learning: Comparison between analytically derived distributions and simulation‐based methods

Is it still necessary to perform measured based pre‐treatment patient‐specific QA for SRS HyperArc treatments?

Contact Info

Product

Resources

About