Combining GAN and LSTM Models for 3D Reconstruction of Lung Tumors from CT Scans

Abstract

Generating a three-dimensional (3D) reconstruction of tumors is an efficient technique for obtaining accurate and highly detailed visualization of the structures of tumors. To create a 3D tumor model, a collection of 2D imaging data is required, including images from CT imaging. Generative adversarial networks (GANs) offer a method to learn helpful representations without annotating the training dataset considerably. The article proposes a technique for creating a 3D model of lung tumors from CT scans using a combination of GAN and LSTM models, with support from ResNet as a feature extractor for the 2D images. The model presented in this article involves three steps, starting with the segmentation of the lung, then the segmentation of the tumor, and concluding with the creation of a 3D reconstruction of the lung tumor. The segmentation of the lung and tumor is conducted utilizing snake optimization and Gustafson-Kessel (GK) method. To prepare the 3D reconstruction component for training, the ResNet model that has been pre-trained is utilized to capture characteristics from 2D lung tumor images. Subsequently, the series of characteristics that have been extracted are fed into a LSTM network to generate compressed features as the final output. Ultimately, the condensed feature is utilized as input for the GAN framework, in which the generator is accountable for generating a sophisticated 3D lung tumor image. Simultaneously, the discriminator evaluates whether the 3D lung tumor image produced by the generator is authentic or synthetic. This model is the initial attempt that utilizes a GAN model as a means for reconstructing 3D lung tumors. The suggested model is evaluated against traditional approaches using the LUNA dataset and standard evaluation metrics. The empirical findings suggest that the suggested approach shows a sufficient level of performance in comparison to other methods that are vying for the same objective.