Enabling deeper learning on big data for materials informatics applications

Abstract

The application of machine learning (ML) techniques in materials science has attracted significant attention in recent years, due to their impressive ability to efficiently extract data-driven linkages from various input materials representations to their output properties. While the application of traditional ML techniques has become quite ubiquitous, there have been limited applications of more advanced deep learning (DL) techniques, primarily because big materials datasets are relatively rare. Given the demonstrated potential and advantages of DL and the increasing availability of big materials datasets, it is attractive to go for deeper neural networks in a bid to boost model performance, but in reality, it leads to performance degradation due to the vanishing gradient problem. In this paper, we address the question of how to enable deeper learning for cases where big materials data is available. Here, we present a general deep learning framework based on Individual Residual learning (IRNet) composed of very deep neural networks that can work with any vector-based materials representation as input to build accurate property prediction models. We find that the proposed IRNet models can not only successfully alleviate the vanishing gradient problem and enable deeper learning, but also lead to significantly (up to 47%) better model accuracy as compared to plain deep neural networks and traditional ML techniques for a given input materials representation in the presence of big data.

Prediction of Hemorrhagic Transformation after Ischemic Stroke Using Machine Learning

“…Third, the higher the number of hidden layers stacked in the ANN algorithm, the more chance that vanishing gradient problems could develop. However, ANN algorithms with few hidden layers, utilizing structured numerical data, can overcome these problems [32]. There is no exact information about how many layers can be stacked to overcome the disadvantage of ANN algorithms falling into the local minima.…”

Recent Advances and Applications of Deep Learning Methods in Materials Science

“…To address this issue, Jha et al [162] developed IRNet, which uses individual residual learning to allow a smoother flow of gradients and enable deeper learning for cases where big data is available. IRNet models were tested on a variety of big and small materials datasets, such as OQMD, AFLOW, Materials Project, JARVIS, using different vector-based materials representations (element fractions, MagPie, structural) and were found to not only successfully alleviate the vanishing gradient problem and enable deeper learning, but also lead to significantly better model accuracy as compared to plain deep neural networks and traditional ML techniques for a given input materials representation in the presence of big data [163]. Further, graph based methods such as Roost [164] have also been developed which can outperform many similar techniques.…”

“…Such methods have been used for diverse DFT datasets mentioned above in Table 1 as well as experimental datasets such as SuperCon [165,166] for quick pre-screening applications. In terms of applications, they have have been applied for predicting properties such as formation energy [151], band gap and magnetization [162], superconducting temperatures [166], bulk and shear modulus [163]. They have also been used for transfer learning across datasets for enhanced predictive accuracy on small data [34], even for different source and target properties [167].…”

Scalable deeper graph neural networks for high-performance materials property prediction

“…Recently, Jha et al. 33 applied the residual skip-connection idea to vector input-based materials property prediction. Their experiments showed that, when the dataset size is more than 15,000, their individual residual networks architecture outperforms both the plain multi-layer perceptron networks and the stacked residual network.…”