Limitations of Deep Learning Attention Mechanisms in Clinical Research: Empirical Case Study Based on the Korean Diabetic Disease Setting

Abstract

Background Despite excellent prediction performance, noninterpretability has undermined the value of applying deep-learning algorithms in clinical practice. To overcome this limitation, attention mechanism has been introduced to clinical research as an explanatory modeling method. However, potential limitations of using this attractive method have not been clarified to clinical researchers. Furthermore, there has been a lack of introductory information explaining attention mechanisms to clinical researchers. Objective The aim of this study was to introduce the basic concepts and design approaches of attention mechanisms. In addition, we aimed to empirically assess the potential limitations of current attention mechanisms in terms of prediction and interpretability performance. Methods First, the basic concepts and several key considerations regarding attention mechanisms were identified. Second, four approaches to attention mechanisms were suggested according to a two-dimensional framework based on the degrees of freedom and uncertainty awareness. Third, the prediction performance, probability reliability, concentration of variable importance, consistency of attention results, and generalizability of attention results to conventional statistics were assessed in the diabetic classification modeling setting. Fourth, the potential limitations of attention mechanisms were considered. Results Prediction performance was very high for all models. Probability reliability was high in models with uncertainty awareness. Variable importance was concentrated in several variables when uncertainty awareness was not considered. The consistency of attention results was high when uncertainty awareness was considered. The generalizability of attention results to conventional statistics was poor regardless of the modeling approach. Conclusions The attention mechanism is an attractive technique with potential to be very promising in the future. However, it may not yet be desirable to rely on this method to assess variable importance in clinical settings. Therefore, along with theoretical studies enhancing attention mechanisms, more empirical studies investigating potential limitations should be encouraged.

Multi-modal image classification of COVID-19 cases using computed tomography and X-rays scans

“…The LSTM based text model performs poorly as compared to the BoW approach text model with reduction in accuracy. This can be attributed to the fact that usually, attention mechanism don’t work very well with clinical data, as also shown by the study conducted by researchers in Korea Kim et al. (2020) .…”

A comparison of machine learning models and Cox proportional hazards models regarding their ability to predict the risk of gastrointestinal cancer based on metabolic syndrome and its components

“…Notably, with regard to variable importance, CPH model exhibits a straightforward interpretation as an HR, whereas a large important variable has more influence on the transition to predict an outcome, and ML models do not provide the sign of the prediction (negative or positive effect) (45). To date, the direct comparison of CPH with ML models regarding interpretation is limited due to the lack of a common metric.…”

Artificial intelligence in cardiology: Hope for the future and power for the present

“…Whilst attention mechanisms are potentially able to boost the performance of a neural network, they are not without limitations. They require vast amount of data for training, are not robust when generalised in other tasks other than the one they were trained for, cannot control spurious correlations in the data and no research has been undertaken to compare different attention models’ performances ( 29 , 30 ).…”