Identifying Pitfalls and Solutions in Parallelizing Long Short-Term Memory Network on Graphical Processing Unit by Comparing with Tensor Processing Unit Parallelism

Ravikumar, Aswathy; Sriraman, Harini; Lokesh, S.; Saketh, P. Maruthi Sai

doi:10.1007/978-981-19-7402-1_9

Cited by 6 publications

(2 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another issue is that it solves the problem of parallelism observed in LSTM networks. However, Transformers cannot represent more than a fixed length of relationships, and splits often do not follow sequence limitations and result in context segmentation, which results in ineffective optimization (Saoud, Al-Marzouqi, and Hussein, 2022;Ravikumar et al, 2023).…”

Section: Transformers In Time Seriesmentioning

confidence: 99%

See 1 more Smart Citation

Transformers as a classifier for solar flare time series: a comparative study

Ferreira,

Gradvohl,

Silva

et al. 2024

Preprint

View full text Add to dashboard Cite

Solar flares are violent and sudden eruptions that occur in the solar atmosphere and release energy in the form of radiation. They can affect technological systems on Earth and in its orbit, causing financial losses and damage to human life. Therefore, it is necessary to predict the occurrence of such flares to mitigate their effects. Specialized instruments gather data for solar activity monitoring. Hence, we can create prediction models using machine learning from this data. From an analysis of the literature, we noticed the prevalence of some algorithms, such as Multi-layer Perceptrons (MLP), Support Vector Machines (SVM), and Long Short-Term Memory (LSTM), which presented good results, mainly considering the True Skill Statistic (TSS) metric. In parallel, in 2017, a new deep learning-based neural network architecture called Transformers emerged. Researchers initially created it for natural language processing. However, Transformers were successfully employed in other domains, such as time series forecasting. Solar activity data is considered a time series due to its continuous capture over time. Consequently, we can employ Transformers to develop a solar flare forecast model. Considering a significant lack of work using Transformers for solar flare forecasting, we ran experiments to test the Transformers' viability and performance in solar flare forecast models. We created models using other algorithms (MLP, SVM, LSTM, Transformers) to investigate the Transformers' performance and compared them using accuracy, TSS, and Area Under the ROC Curve (AUC) metrics. We observed that the Transformers had superior performance compared to the other models. For instance, the Transformers' TSS metric average was 0.9, contrasting the other models' TSS=0.4. The difference was slightly smaller in AUC, where Transformers reached 0.9, and the others reached no more than 0.7. Therefore, we can use the Transformers to classify solar flare data and obtain superior results compared to other models. We also conducted experiments using different forms of data balancing, including unbalanced data, balanced with undersampling, oversampling, and SMOTE techniques. The MLP, SVM, and LSTM models showed significant improvements in balance, where the average TSS increased from 0.1 to 0.4. On the other hand, Transformers were not sensitive to data balancing, presenting the most stable TSS in all cases.

show abstract

Section: Transformers In Time Seriesmentioning

confidence: 99%

“…The RNN performance is due to its intrinsically sequential nature, which can limit the full exploitation of the potential of parallelism in dedicated hardware environments. Transformers, in turn, allow parallel execution in GPUs and TPUs (Ravikumar et al, 2023).…”

Section: Transformers In Time Seriesmentioning

confidence: 99%

Transformers as a classifier for solar flare time series: a comparative study

Ferreira,

Gradvohl,

Silva

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

Selecting the optimal transfer learning model for precise breast cancer diagnosis utilizing pre-trained deep learning models and histopathology images

Ravikumar,

Sriraman,

Saleena

et al. 2023

Health Technol.

View full text Add to dashboard Cite

Computationally Efficient Neural Rendering for Generator Adversarial Networks Using a Multi-GPU Cluster in a Cloud Environment

Ravikumar

Sriraman

2023

IEEE Access

Self Cite

View full text Add to dashboard Cite

Due to its fantastic performance in the quality of the images created, Generator Adversarial Networks have recently become a viable option for image reconstruction. The main problem with employing GAN is how expensive the computations are. Researchers have developed techniques for distributing GANs across multiple nodes. However, these techniques typically do not scale because they frequently separate the components (Discriminator and Generator), leading to high communication overhead or encountering distribution-related problems unique to GAN training. In this study, the training procedure for the GAN is parallelized and carried out over many Graphical Processing Units (GPUs). TensorFlow's built-in logic and a custom loop were tweaked for more control over the resources allotted to each GPU worker. In this study, GPU image processing improvements and multi-GPU learning are used. The GAN model is accelerated using Distributed TensorFlow with synchronous data-parallel training on a single system and several GPUs. Acceleration was accomplished using the Genesis Cloud Platform and the NVIDIA®GeForceTM GTX 108 GPU accelerator. The speed-up of 1.322 for two GPUs, 1.688 for three GPUs, and 1.7792 for four GPUs using multi-GPU acceleration. The parameter server model's data initialization and image production bottlenecks are removed, but the results' speed-up is not linear. Increasing the number of GPUs and removing the connectivity constraint will accelerate things even more. The bottlenecks are detected using new network lines and resources, and solutions are suggested. Recomputation and quantization are the two techniques to reduce the amount of GPU acceleration in memory. Deployment and versioning are essential for successfully operating multi-node GAN models in MLflow. Properly deploying and versioning these models can improve scalability, reproducibility, and collaboration across teams working on the same model. MLflow provides built-in tools for versioning and tracking model performance, making it easier to manage multiple versions of the model and reproduce it in different environments.

show abstract

Identifying Pitfalls and Solutions in Parallelizing Long Short-Term Memory Network on Graphical Processing Unit by Comparing with Tensor Processing Unit Parallelism

Cited by 6 publications

References 11 publications

Transformers as a classifier for solar flare time series: a comparative study

Transformers as a classifier for solar flare time series: a comparative study

Selecting the optimal transfer learning model for precise breast cancer diagnosis utilizing pre-trained deep learning models and histopathology images

Computationally Efficient Neural Rendering for Generator Adversarial Networks Using a Multi-GPU Cluster in a Cloud Environment

Contact Info

Product

Resources

About