Patient-specific Quality Assurance Failure Prediction with Deep Tabular Models

Abstract

Background: Patient-specific quality assurance (PSQA) is part of the standard practice to ensure that a patient receives the dose from intensity-modulated radiotherapy (IMRT) beams as planned in the treatment planning system (TPS). PSQA failures can cause a delay in patient care and increase workload and stress of staff members. A large body of previous work for PSQA failure prediction focuses on non-learned plan complexity measures. Another prominent line of work uses machine learning methods, often in conjunction with feature engineering. Currently, there are no machine learning solutions which work directly with multi-leaf collimator (MLC) leaf positions, providing an opportunity to improve leaf sequencing algorithms using these techniques. Purpose: To improve patient safety and work efficiency, we develop a tabular transformer model based directly on the MLC leaf positions (without any feature engineering) to predict IMRT PSQA failure. This neural model provides an end-to-end differentiable map from MLC leaf positions to the probability of PSQA plan failure, which could be useful for regularizing gradient-based leaf sequencing optimization algorithms and generating a plan that is more likely to pass PSQA. Method: We retrospectively collected DICOM RT PLAN files of 968 patient plans treated with volumetric arc therapy. We construct a beam-level tabular dataset with 1873 beams as samples and MLC leaf positions as features. We train an attention-based neural network FT-Transformer to predict the ArcCheck-based PSQA gamma pass rates. In addition to the regression task, we evaluate the model in the binary classification context predicting the pass or fail of PSQA. The performance was compared to the results of the two leading tree ensemble methods (CatBoost and XGBoost) and a non-learned method based on mean MLC gap. Results: The FT-Transformer model achieves 1.44% Mean Absolute Error (MAE) in the regression task of the gamma pass rate prediction and performs on par with XGBoost (1.53 % MAE) and CatBoost (1.40 % MAE). In the binary classification task of PSQA failure prediction, FT-Transformer achieves 0.85 ROC AUC (with CatBoost and XGBoost achieving 0.87 ROC AUC and the mean-MLC-gap complexity metric achieving 0.72 ROC AUC). %While AUC summarizes the performance of binary classification models for all prediction thresholds, the performance at specific thresholds that maximize the true positive rate of PSQA failure identification while maintaining low false positive rate is of interest for clinical applications. Moreover, FT-Transformer, CatBoost, and XGBoost all achieve 80\% true positive rate while keeping the false positive rate under 20%. Conclusions: We demonstrate that reliable PSQA failure predictors can be successfully developed based solely on MLC leaf positions. Our FT-Transformer neural network can reduce the need for patient rescheduling due to PSQA failures by 80% while sending only 20% of plans that would not have failed the PSQA for replanning. FT-Transformer achieves comparable performance with the leading tree ensemble methods while having an additional benefit of providing an end-to-end differentiable map from MLC leaf positions to the probability of PSQA failure.

Amalur: Data Integration Meets Machine Learning

“…In recent data integration works [50], [53], [54], the embeddings are applied to capture the similarities between features or tuples in source tables. Taking one step further, representations of the entire table [55], [56] allow for many more applications, e.g., transfer learning and multimodal machine learning. Many research questions remain open, regarding more types of DI metadata, their representations, and their roles in improving ML tasks.…”

Clinical Advancement Forecasting

“…note that neural models are a significant omission from this framework. While deep learning methods for tabular data are competitive with gradient-boosted tree models [34,35,36], they are rarely superior without transfer learning [34,37,38]. Additionally, the data in this proof-of-concept is limited to Open Targets evidence scores that already reflect a large degree of feature engineering, thereby limiting the extent to which deep learning networks can identify more granular predictive signals.…”

Patient-specific quality assurance failure prediction with deep tabular models

“…Although numerous models have been proposed based on using differentiable ensembles [49][50][51][52][53], leveraging attention-based transformer neural networks [35,[54][55][56][57][58], as well as other approaches [59][60][61][62][63][64], recent work on systematic evaluation of deep tabular models [35,48] shows that there is no universally best model capable of consistently outperforming GBDT. Transformer-based models have been shown to be the strongest competitor of GBDT [35,54,58,65,66], especially when coupled with a powerful hyperparameter tuning toolkit [35,67].…”