Reinforcement learning using Deep $$Q$$ networks and $$Q$$ learning accurately localizes brain tumors on MRI with very small training sets

Abstract

Background Supervised deep learning in radiology suffers from notorious inherent limitations: 1) It requires large, hand-annotated data sets; (2) It is non-generalizable; and (3) It lacks explainability and intuition. It has recently been proposed that reinforcement learning addresses all three of these limitations. Notable prior work applied deep reinforcement learning to localize brain tumors with radiologist eye tracking points, which limits the state-action space. Here, we generalize Deep Q Learning to a gridworld-based environment so that only the images and image masks are required. Methods We trained a Deep $$Q$$ Q network on 30 two-dimensional image slices from the BraTS brain tumor database. Each image contained one lesion. We then tested the trained Deep Q network on a separate set of 30 testing set images. For comparison, we also trained and tested a keypoint detection supervised deep learning network on the same set of training/testing images. Results Whereas the supervised approach quickly overfit the training data and predictably performed poorly on the testing set (11% accuracy), the Deep $$Q$$ Q learning approach showed progressive improved generalizability to the testing set over training time, reaching 70% accuracy. Conclusion We have successfully applied reinforcement learning to localize brain tumors on 2D contrast-enhanced MRI brain images. This represents a generalization of recent work to a gridworld setting naturally suitable for analyzing medical images. We have shown that reinforcement learning does not over-fit small training sets, and can generalize to a separate testing set.

Machine Learning Empowering Personalized Medicine: A Comprehensive Review of Medical Image Analysis Methods

“…In object detection and extraction tasks, reinforcement learning benefits in tasks like breast lesion detection [61,62], where learning agents gradually learn the policy to choose among actions to transit, scale the bounding box, and finally localize the breast lesion. It is also used to address the lack of labeled data in brain tumor detection tasks [63,64]. Therefore, reinforcement learning models could work as robust lesion detectors with limited training data [65], reducing time consumption and providing some interpretability [66].…”

Applications of Artificial Intelligence in Temporal Bone Imaging: Advances and Future Challenges

“…Reinforcement learning describes the trial-and-error training process of the AI agent to maximize a reward signal [ 16 ]. Positive or negative feedback engages it in constant interaction with the environment and urges it to opt for the best actions to achieve a cumulative rewarding result over time [ 17 ]. Popular reinforcement learning algorithms include Q-Q-learning, deep Q-Q-networks (DQN), and policy gradient methods.…”

Deep Reinforcement Learning Method for 3D-CT Nasopharyngeal Cancer Localization with Prior Knowledge

“…This study shows that supervised deep learning is seriously overfitting on the training set, with low accuracy, while reinforcement learning may make meaningful predictions based on very small datasets. Based on this work, Stember et al [16] trained a Deep Q network on 30 2D image slices from the BraTS brain tumor database, which showed good accuracy (70% accuracy) compared to the supervised deep learning network (11% accuracy) on the testing set. Navarro et al [17] proposed a DRL method for CT organ localization for exhaustive or regional suggestion search strategies that require large amounts of annotated data.…”