LPFS: Learnable Polarizing Feature Selection for Click-Through Rate Prediction

Abstract

In industry, feature selection is a standard but necessary step to search for an optimal set of informative feature fields for efficient and effective training of deep Click-Through Rate (CTR) models. Most previous works measure the importance of feature fields by using their corresponding continuous weights from the model, then remove the feature fields with small weight values. However, removing many features that correspond to small but not exact zero weights will inevitably hurt model performance and not be friendly to hot-start model training. There is also no theoretical guarantee that the magnitude of weights can represent the importance, thus possibly leading to sub-optimal results if using these methods.To tackle this problem, we propose a novel Learnable Polarizing Feature Selection (LPFS) method using a smoothed-ℓ 0 function in literature. Furthermore, we extend LPFS to LPFS++ by our newly designed smoothed-ℓ 0 -liked function to select a more informative subset of features. LPFS and LPFS++ can be used as gates inserted at the input of the deep network to control the active and inactive state of each feature. When training is finished, some gates are exact zero, while others are around one, which is particularly favored by the practical hot-start training in the industry, due to no damage

Optimizing Feature Set for Click-Through Rate Prediction

“…When g 𝑘 𝑖 = 1, feature x 𝑘 𝑖 is in the optimal feature set X g and vice versa. Notice that previous work [8,30,32] ]. The final prediction can be formulated as follows:…”

“…3.1.3 Baseline Methods and Backbone Models. We compare the proposed method OptFS with the following feature selection methods: (i) AutoField [32]: This baseline utilizes neural architecture search techniques [15] to select the informative features on a field 1 https://www.kaggle.com/c/criteo-display-ad-challenge 2 http://www.kaggle.com/c/avazu-ctr-prediction 3 http://www.kddcup2012.org/c/kddcup2012-track2/data level; (ii) LPFS [8]: This baseline designs a customized, smoothedl 0 -liked function to select informative fields on a field level; (iii) AdaFS [13]: This baseline that selects the most relevant features for each sample via a novel controller network. We apply the above baselines over the following mainstream backbone models: FM [26], DeepFM [7], DCN [31] and IPNN [24].…”

“…Firstly, Several methods produce the feature set based on feature selection. Because of the large-scale CTR dataset, some methods [8,30,32] focus on the field level, which results in hundreds of fields instead of millions of features. However, the field level is too coarse to find an optimal feature set.…”

“…The id of certain cold users/items might be excluded from the feature set due to the sparsity problem [29], which is difficult to handle at the field level. Besides, these methods [8,13] fail to leverage the influence of feature interaction, which is commonly considered an enhancement for the model performance [20,38]. Secondly, there is also some weakness concerning feature interaction methods, which implicitly produce the feature set.…”

“…The second challenge is the number of features in large-scale datasets. Notice that the possible number of features considered in our research could be 10 6 , which is incredibly larger than 100 feature fields in previous works [8,32]. To navigate in the large search space, we introduce a learnable gate for each feature and adopt the learning-by-continuation [16,28,36] training scheme.…”