Offline and online LSTM networks for respiratory motion prediction in MR-guided radiotherapy

Lombardo, Elia; Rabe, Moritz; Xiong, Yuqing; Nierer, Lukas; Cusumano, D.; Placidi, L.; Boldrini, Luca; Corradini, Stefanie; Niyazi, Maximilian; Belka, Claus; Riboldi, Marco; Kurz, Christopher; Landry, Guillaume

doi:10.1088/1361-6560/ac60b7

Cited by 22 publications

(30 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Three respiratory motion prediction models previously compared in‐silico 20 and a baseline no‐predictor were implemented in this study and applied to the eight unseen motion traces: 1. Offline LSTM: this model had been previously trained and validated using motion traces extracted from 4 Hz cine MRIs of 70 patients treated with a 0.35 T MRI‐linac (13.1 h of data) 20 . It was applied without any changes to hyper‐parameters or weights to the unseen experiment traces and predicted the future target centroid position in 250 and 500 ms.…”

Section: Methodsmentioning

confidence: 99%

“…In contrast to the LSTM, the LR does not require iterative optimization, as an analytical solution exists 30 . For this reason, the online LR was solely based on the last 20 s of data and was solved from scratch on the updated set of sequences every 250 ms, that is, every time a new target position was available, as in the previous in‐silico study 20 . Also for the LR, linear interpolation between the 250 and 500 ms predictions provided the target position which was used for MLC‐tracking. …”

Section: Methodsmentioning

confidence: 99%

“…Three respiratory motion prediction models previously compared in-silico 20 and a baseline no-predictor were implemented in this study and applied to the eight unseen motion traces:…”

Section: Prediction Modelsmentioning

confidence: 99%

“…Recently, a class of machine learning algorithms called long short-term memory (LSTM) networks, which is ideally suited to deal with sequential input data, has been shown to be very promising for motion prediction both in RT 18,19 and in MRIgRT. 20,21 Specifically, LSTMs were shown to outperform LR models for the prediction of superior-inferior (SI) target centroid positions based on patient data acquired on a 0.35 T MRI -linac. 20 While all aforementioned studies were in-silico, to the best of our knowledge there are four studies which experimentally investigated motion prediction for MLCtracking during MRIgRT.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Experimental comparison of linear regression and LSTM motion prediction models for MLC‐tracking on an MRI‐linac

Lombardo,

Liu,

Waddington

et al. 2023

Medical Physics

Self Cite

View full text Add to dashboard Cite

BackgroundMagnetic resonance imaging (MRI)‐guided radiotherapy with multileaf collimator (MLC)‐tracking is a promising technique for intra‐fractional motion management, achieving high dose conformality without prolonging treatment times. To improve beam‐target alignment, the geometric error due to system latency should be reduced by using temporal prediction.PurposeTo experimentally compare linear regression (LR) and long‐short‐term memory (LSTM) motion prediction models for MLC‐tracking on an MRI‐linac using multiple patient‐derived traces with different complexities.MethodsExperiments were performed on a prototype 1.0 T MRI‐linac capable of MLC‐tracking. A motion phantom was programmed to move a target in superior‐inferior (SI) direction according to eight lung cancer patient respiratory motion traces. Target centroid positions were localized from sagittal 2D cine MRIs acquired at 4 Hz using a template matching algorithm. The centroid positions were input to one of four motion prediction models. We used (1) a LSTM network which had been optimized in a previous study on patient data from another cohort (offline LSTM). We also used (2) the same LSTM model as a starting point for continuous re‐optimization of its weights during the experiment based on recent motion (offline+online LSTM). Furthermore, we implemented (3) a continuously updated LR model, which was solely based on recent motion (online LR). Finally, we used (4) the last available target centroid without any changes as a baseline (no‐predictor). The predictions of the models were used to shift the MLC aperture in real‐time. An electronic portal imaging device (EPID) was used to visualize the target and MLC aperture during the experiments. Based on the EPID frames, the root‐mean‐square error (RMSE) between the target and the MLC aperture positions was used to assess the performance of the different motion predictors. Each combination of motion trace and prediction model was repeated twice to test stability, for a total of 64 experiments.ResultsThe end‐to‐end latency of the system was measured to be (389 ± 15) ms and was successfully mitigated by both LR and LSTM models. The offline+online LSTM was found to outperform the other models for all investigated motion traces. It obtained a median RMSE over all traces of (2.8 ± 1.3) mm, compared to the (3.2 ± 1.9) mm of the offline LSTM, the (3.3 ± 1.4) mm of the online LR and the (4.4 ± 2.4) mm when using the no‐predictor. According to statistical tests, differences were significant (p‐value <0.05) among all models in a pair‐wise comparison, but for the offline LSTM and online LR pair. The offline+online LSTM was found to be more reproducible than the offline LSTM and the online LR with a maximum deviation in RMSE between two measurements of 10%.ConclusionsThis study represents the first experimental comparison of different prediction models for MRI‐guided MLC‐tracking using several patient‐derived respiratory motion traces. We have shown that among the investigated models, continuously re‐optimized LSTM networks are the most promising to account for the end‐to‐end system latency in MRI‐guided radiotherapy with MLC‐tracking.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

“…Three respiratory motion prediction models previously compared in-silico 20 and a baseline no-predictor were implemented in this study and applied to the eight unseen motion traces:…”

Section: Prediction Modelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Experimental comparison of linear regression and LSTM motion prediction models for MLC‐tracking on an MRI‐linac

Lombardo,

Liu,

Waddington

et al. 2023

Medical Physics

Self Cite

View full text Add to dashboard Cite

show abstract

“…Claustrophobia in MRI scanners are common in the general population. Some studies estimate that 10-15% of patients would require some level of sedation to be able to be able to complete an MRI scan (45). Hospitals and clinics have established safety policies and standards in place for MRI safety, and the clinical team must maintain vigilance to ensure that the patient does not have MRI-incompatible material in their body or on their person (46).…”

Section: Contraindications To Mr Imagingmentioning

confidence: 99%

MRI-LINAC: A transformative technology in radiation oncology

Gregucci²,

Pennell³

et al. 2023

Front. Oncol.

View full text Add to dashboard Cite

Advances in radiotherapy technologies have enabled more precise target guidance, improved treatment verification, and greater control and versatility in radiation delivery. Amongst the recent novel technologies, Magnetic Resonance Imaging (MRI) guided radiotherapy (MRgRT) may hold the greatest potential to improve the therapeutic gains of image-guided delivery of radiation dose. The ability of the MRI linear accelerator (LINAC) to image tumors and organs with on-table MRI, to manage organ motion and dose delivery in real-time, and to adapt the radiotherapy plan on the day of treatment while the patient is on the table are major advances relative to current conventional radiation treatments. These advanced techniques demand efficient coordination and communication between members of the treatment team. MRgRT could fundamentally transform the radiotherapy delivery process within radiation oncology centers through the reorganization of the patient and treatment team workflow process. However, the MRgRT technology currently is limited by accessibility due to the cost of capital investment and the time and personnel allocation needed for each fractional treatment and the unclear clinical benefit compared to conventional radiotherapy platforms. As the technology evolves and becomes more widely available, we present the case that MRgRT has the potential to become a widely utilized treatment platform and transform the radiation oncology treatment process just as earlier disruptive radiation therapy technologies have done.

show abstract

Intra‐frame motion deterioration effects and deep‐learning‐based compensation in MR‐guided radiotherapy

Sui,

Palaniappan,

Brenner

et al. 2023

Medical Physics

View full text Add to dashboard Cite

BackgroundCurrent commercially available hybrid magnetic resonance linear accelerators (MR‐Linac) use 2D+t cine MR imaging to provide intra‐fractional motion monitoring. However, given the limited temporal resolution of cine MR imaging, target intra‐frame motion deterioration effects, resulting in effective time latency and motion artifacts in the image domain, can be appreciable, especially in the case of fast breathing.PurposeThe aim of this work is to investigate intra‐frame motion deterioration effects in MR‐guided radiotherapy (MRgRT) by simulating the motion‐corrupted image acquisition, and to explore the feasibility of deep‐learning‐based compensation approaches, relying on the intra‐frame motion information which is spatially and temporally encoded in the raw data (k‐space).MethodsAn intra‐frame motion model was defined to simulate motion‐corrupted MR images, with 4D anthropomorphic digital phantoms being exploited to provide ground truth 2D+t cine MR sequences. A total number of 10 digital phantoms were generated for lung cancer patients, with randomly selected eight patients for training or validation and the remaining two for testing. The simulation code served as the data generator, and a dedicated motion pattern perturbation scheme was proposed to build the intra‐frame motion database, where three degrees of freedom were designed to guarantee the diversity of intra‐frame motion trajectories, enabling a thorough exploration in the domain of the potential anatomical structure positions. U‐Nets with three types of loss functions: L1 or L2 loss defined in image or Fourier domain, referred to as NNImgLoss‐L1, NNFloss‐L1 and NNL2‐Loss were trained to extract information from the motion‐corrupted image and used to estimate the ground truth final‐position image, corresponding to the end of the acquisition. Images before and after compensation were evaluated in terms of (i) image mean‐squared error (MSE) and mean absolute error (MAE), and (ii) accuracy of gross tumor volume (GTV) contouring, based on optical‐flow image registration.ResultsImage degradation caused by intra‐frame motion was observed: for a linearly and fully acquired Cartesian readout k‐space trajectory, intra‐frame motion resulted in an imaging latency of approximately 50% of the acquisition time; in comparison, the motion artifacts exhibited only a negligible contribution to the overall geometric errors. All three compensation models led to a decrease in image MSE/MAE and GTV position offset compared to the motion‐corrupted image. In the investigated testing dataset for GTV contouring, the average dice similarity coefficients (DSC) improved from 88% to 96%, and the 95th percentile Hausdorff distance (HD95) dropped from 4.8 mm to 2.1 mm. Different models showed slight performance variations across different intra‐frame motion amplitude categories: NNImgLoss‐L1 excelled for small/medium amplitudes, whereas NNFloss‐L1 demonstrated higher DSC median values at larger amplitudes. The saliency maps of the motion‐corrupted image highlighted the major contribution of the later acquired k‐space data, as well as the edges of the moving anatomical structures at their final positions, during the model inference stage.ConclusionsOur results demonstrate the deep‐learning‐based approaches have the potential to compensate for intra‐frame motion by utilizing the later acquired data to drive the convergence of the earlier acquired k‐space components.

show abstract

Offline and online LSTM networks for respiratory motion prediction in MR-guided radiotherapy

Cited by 22 publications

References 37 publications

Experimental comparison of linear regression and LSTM motion prediction models for MLC‐tracking on an MRI‐linac

Experimental comparison of linear regression and LSTM motion prediction models for MLC‐tracking on an MRI‐linac

MRI-LINAC: A transformative technology in radiation oncology

Intra‐frame motion deterioration effects and deep‐learning‐based compensation in MR‐guided radiotherapy

Contact Info

Product

Resources

About