Toward predicting the evolution of lung tumors during radiotherapy observed on a longitudinal MR imaging study via a deep learning algorithm

Abstract

Purpose To predict the spatial and temporal trajectories of lung tumor during radiotherapy monitored under a longitudinal magnetic resonance imaging (MRI) study via a deep learning algorithm for facilitating adaptive radiotherapy (ART). Methods We monitored 10 lung cancer patients by acquiring weekly MRI‐T2w scans over a course of radiotherapy. Under an ART workflow, we developed a predictive neural network (P‐net) to predict the spatial distributions of tumors in the coming weeks utilizing images acquired earlier in the course. The three‐step P‐net consisted of a convolutional neural network to extract relevant features of the tumor and its environment, followed by a recurrence neural network constructed with gated recurrent units to analyze trajectories of tumor evolution in response to radiotherapy, and finally an attention model to weight the importance of weekly observations and produce the predictions. The performance of P‐net was measured with Dice and root mean square surface distance (RMSSD) between the algorithm‐predicted and experts‐contoured tumors under a leave‐one‐out scheme. Results Tumor shrinkage was 60% ± 27% (mean ± standard deviation) by the end of radiotherapy across nine patients. Using images from the first three weeks, P‐net predicted tumors on future weeks (4, 5, 6) with a Dice and RMSSD of (0.78 ± 0.22, 0.69 ± 0.24, 0.69 ± 0.26), and (2.1 ± 1.1 mm, 2.3 ± 0.8 mm, 2.6 ± 1.4 mm), respectively. Conclusion The proposed deep learning algorithm can capture and predict spatial and temporal patterns of tumor regression in a longitudinal imaging study. It closely follows the clinical workflow, and could facilitate the decision‐making of ART. A prospective study including more patients is warranted.

A method for predictive modeling of tumor regression for lung adaptive radiotherapy

“…An alternative method, investigated by Wang et al, would be to treat each patient as a time series, with the goal of predicting the future state of the change tumor volume. 32 We elected to generate daily predictions to provide a simplified mode that did not rely on the possible uncertainty of the future state, allowing physicians to make real time decisions based the changes that have already occurred.…”

“…In cases in which the reviewer does not support the need for ART, the probability threshold can be increased mid‐treatment, only providing subsequent alerts in cases with continued corresponding increase in the magnitude of tumor regression. An alternative method, investigated by Wang et al, would be to treat each patient as a time series, with the goal of predicting the future state of the change tumor volume 32 . We elected to generate daily predictions to provide a simplified mode that did not rely on the possible uncertainty of the future state, allowing physicians to make real time decisions based the changes that have already occurred.…”

Applications of artificial intelligence and deep learning in molecular imaging and radiotherapy

“…Recurrent and/or recursive neural networks (RNNs) are among the supervised deep learning algorithms, which in contrast to CNN models, have the ability to access/send/process information over time steps. RNNs take sequences of input vectors to model them one at a time enabling both parallel and sequential processing appropriate for data analysis in time series, such as dynamic or longitudinal studies (Wang et al 2019a ).

Fig.

…”

Technical Note: 3D localization of lung tumors on cone beam CT projections via a convolutional recurrent neural network

“…Recurrent neural networks (RNN) are specialized in processing time series images . Such networks have been combined with CNN to track two‐dimensional motions of markers implanted in liver using ultrasound images, and recently to track lung tumor changes in response to radiotherapy at our institution . In this paper, we present a similar algorithm, a convolutional recurrence neural network (CRNN), for real‐time three‐dimensional (3D) localization of mobile lung tumors from CBCT projections.…”

“…23 Such networks have been combined with CNN to track two-dimensional motions of markers implanted in liver using ultrasound images, 24 and recently to track lung tumor changes in response to radiotherapy at our institution. 25 In this paper, we present a similar algorithm, a convolutional recurrence neural network (CRNN), for real-time three-dimensional (3D) localization of mobile lung tumors from CBCT projections. The CRNN is designed to process both static features embedded in a particular kV projection, as well as the temporal evolution of such features in a stream of projections.…”