Limitations of Deep Learning Attention Mechanisms in Clinical Research: Empirical Case Study Based on the Korean Diabetic Disease Setting (Preprint)

Abstract

BACKGROUND Despite excellent prediction performance, non-interpretability has undermined the value of applying deep learning algorithms in clinical practice. To overcome this limitation, an explanatory modeling method called attention mechanism has been introduced to clinical research. However, gentle guidance and precautions for using this attractive method have not been well provided to clinical and informatics researchers. Furthermore, there has been a lack of discussion on the predictive and interpretive performance of this method when applied to health data. OBJECTIVE The purpose of this study is to provide clinical researchers with the basic concepts and design approaches of attention mechanisms. In addition, the study aims to evaluate current design approaches of attention mechanisms in terms of prediction and interpretability performance. METHODS First, the basic concepts and several key considerations regarding attention mechanisms are provided. Second, the four approaches to attention mechanisms are introduced according to a two-dimensional framework based on degree of freedom and uncertainty awareness. Third, 1) prediction performance, 2) probability reliability, 3) concentration of variable importance, 4) consistency of attention results, and 5) generalizability of attention results to conventional statistics, are assessed in the diabetic classification modeling setting. Fourth, the performances of the four attention design approaches are discussed. RESULTS Prediction performance was very high for all models. Probability reliability was high in models with a high degree of freedom. Variable importance was concentrated in several variables when uncertainty awareness was not considered. Consistency of attention results was high when uncertainty awareness was considered. The generalizability of attention results to conventional statistics was poor regardless of the modeling approach. CONCLUSIONS The attention mechanism is obviously an attractive technique, which could be very promising in the future. However, naive attention implementations may lead to poor results when determining variable importance. Therefore, more robust theoretical studies of attention mechanisms should be encouraged.