Dropout Feature Ranking for Deep Learning Models

Abstract

Deep neural networks (DNNs) achieve state-of-the-art results in a variety of domains. Unfortunately, DNNs are notorious for their non-interpretability, and thus limit their applicability in hypothesis-driven domains such as biology and healthcare. Moreover, in the resource-constraint setting, it is critical to design tests relying on fewer more informative features leading to high accuracy performance within reasonable budget. We aim to close this gap by proposing a new general feature ranking method for deep learning. We show that our simple yet effective method performs on par or compares favorably to eight strawman, classical and deep-learning feature ranking methods in two simulations and five very different datasets on tasks ranging from classification to regression, in both static and time series scenarios. We also illustrate the use of our method on a drug response dataset and show that it identifies genes relevant to the drug-response.

Towards a Better Tradeoff between Effectiveness and Efficiency in Pre-Ranking: A Learnable Feature Selection based Approach

“…Inspired by the Dropout FR method based on the variational dropout layer [1] where the efficiency of the model is ignored, FSCD is proposed that considers both effectiveness and efficiency in a learnable process as illustrated in Figure 2. In the proposed FSCD, the effectiveness is optimized by the cross entropy based loss function, while the efficiency is optimized by the feature-wise regularization term in Eq.…”

“…The feature set for the pre-ranking model is a subset of that for the ranking model, which is selected by the proposed FSCD. All feature sets, consisting of 246 feature fields in their entirety, [1], the DFS method [6], and the proposed FSCD. include different types, e.g., simple features that directly look up embeddings and complex features that require complicated computations for online embeddings.…”

“…After the proposed FSCD process, the feature fields are ranked using the output parameter 𝛿 𝛿 𝛿. Table 2 lists the minimum, median, and maximum ranking indices for different types of features using Dropout FR [1] and deep feature selection (DFS) method [6]. For the proposed FSCD in this paper, the ranking indices for the simple features are much smaller than those for the complex feature types (see median and maximum).…”

“…The area under the curve (AUC), latency, and CPU consumption for online inference of the different models are illustrated in Table 3. The results of the conventional feature selection methods [1,6] are also provided for comparison. Specifically, when 𝐾 = 246, the feature set of pre-ranking model becomes exactly the same as that of the ranking model.…”

Explainability and Adversarial Robustness for RNNs

“…Another method that uses dropout for feature importance is [23], applying a technique called Variational Dropout to learn the optimal dropout rate for each feature. It tries to leave out as many features as possible and at the same time keep accuracy high.…”