Operating a treatment planning system using a deep‐reinforcement learning‐based virtual treatment planner for prostate cancer intensity‐modulated radiation therapy treatment planning

Abstract

PurposeIn the treatment planning process of intensity‐modulated radiation therapy (IMRT), a human planner operates the treatment planning system (TPS) to adjust treatment planning parameters, for example, dose volume histogram (DVH) constraints’ locations and weights, to achieve a satisfactory plan for each patient. This process is usually time‐consuming, and the plan quality depends on planer’s experience and available planning time. In this study, we proposed to model the behaviors of human planners in treatment planning by a deep reinforcement learning (DRL)‐based virtual treatment planner network (VTPN), such that it can operate the TPS in a human‐like manner for treatment planning.Methods and MaterialsUsing prostate cancer IMRT as an example, we established the VTPN using a deep neural network developed. We considered an in‐house optimization engine with a weighted quadratic objective function. Virtual treatment planner network was designed to observe an intermediate plan DVHs and decide the action to improve the plan by changing weights and threshold dose in the objective function. We trained the VTPN in an end‐to‐end DRL process in 10 patient cases. A plan score was used to measure plan quality. We demonstrated the feasibility and effectiveness of the trained VTPN in another 64 patient cases.ResultsVirtual treatment planner network was trained to spontaneously learn how to adjust treatment planning parameters to generate high‐quality treatment plans. In the 64 testing cases, with initialized parameters, quality score was 4.97 (±2.02), with 9.0 being the highest possible score. Using VTPN to perform treatment planning improved quality score to 8.44 (±0.48).ConclusionsTo our knowledge, this was the first time that intelligent treatment planning behaviors of human planner in external beam IMRT are autonomously encoded in an artificial intelligence system. The trained VTPN is capable of behaving in a human‐like way to produce high‐quality plans.

Improving efficiency of training a virtual treatment planner network via knowledge‐guided deep reinforcement learning for intelligent automatic treatment planning of radiotherapy

“…[14][15][16] More recently, deep learning-based methods 17 have shown their great promise in the context of automatic treatment planning. [18][19][20][21][22][23] In particular, deep reinforcement learning (DRL) has been employed to develop an intelligent automatic treatment planning framework. Within this framework, a virtual treatment planner network (VTPN) was built to model the intelligent behaviors of human planners in the treatment planning process.…”

“…The feasibility of this approach has been demonstrated in preliminary studies in exemplary problems of high-dose-rate (HDR) brachytherapy for cervical cancer 20 and IMRT for prostate cancer. 23 Despite the initial success, a major concern was low efficiency of training a VTPN. Training a VTPN requires a large number of training data in the form of state-action pairs, that is, the combinations of plan DVHs and corresponding actions of adjusting TPPs.…”

“…The standard DRL approaches employ an ε-greedy algorithm to navigate the state-action space and generate the training data. 20,[23][24][25][26][27][28] Specifically, at each training step, it selects the optimal action predicted by the current VTPN for the plan state with a probably of (1-ε), and a random action among all possible actions with a probably of ε. The parameter ε is usually set to be close to unity in the early stage of training and is gradually reduced over the training process.…”

“…The change in plan quality can only be computed by comparing two plans prior to and after the TPP adjustment, for which plan optimization using the adjusted TPPs has to be performed. Given the fact that DRL usually requires a huge number of steps to navigate and sample the state-action space, the training time of VTPN can be days or even weeks in previous proof-of-principle studies using in-house TPSs, 20,23 in which only a small number of possible actions existed. It is expected that the efficiency would become a much more severe concern, when extending the DRL-based VTPN methods to more sophisticated but clinically relevant scenarios, for example, having the VTPN to adjust TPPs of a clinically realistic TPS.…”

Artificial intelligence‐based radiotherapy machine parameter optimization using reinforcement learning

“…22,23 There is growing interest in applying RL to complex decision-making problems in radiation oncology. 24,25 RL frameworks have recently been designed to adjust input weights for conventional inverse HDR brachytherapy 26 and prostate IMRT optimizers, 27 in which cases the RL agent mimicked the action of human planners with the goal of automating treatment planning.…”