Humans have rich experience applying linear models and logical thinking, but only experts understand the behaviour of non-linear systems. However, the deep neural network (DNN) implementation of non-linear systems outperforms optimal linear models. Therefore, the forward DNN (the pattern recognition system in this paper) attracts attention to the necessity of interpreting the results obtained by DNN. To preserve the high performance of DNN, we focus on a post hoc explanation; this approach means building an explainable model for the decision obtained by the black box. To avoid the interpretation of a set of millions of non-linear functions, we divide DNN into two parts: the feature extractor and the classifier. Following that, we argue for a specific interpretation of each of them. While for classifiers, we have several suitable explainable models (and we decided on the fuzzy logical function), we believe that feature interpretation is a creative scientific activity corresponding to the usual research. The paper presents a tool to help researchers and users understand extracted features not necessarily known in the specific application domain. Explanation of the new features is the way of learning from computers.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.