Accurate and Interpretable Factorization Machines

Abstract

Factorization Machines (FMs), a general predictor that can efficiently model high-order feature interactions, have been widely used for regression, classification and ranking problems. However, despite many successful applications of FMs, there are two main limitations of FMs: (1) FMs consider feature interactions among input features by using only polynomial expansion which fail to capture complex nonlinear patterns in data. (2) Existing FMs do not provide interpretable prediction to users. In this paper, we present a novel method named Subspace Encoding Factorization Machines (SEFM) to overcome these two limitations by using non-parametric subspace feature mapping. Due to the high sparsity of new feature representation, our proposed method achieves the same time complexity as the standard FMs but can capture more complex nonlinear patterns. Moreover, since the prediction score of our proposed model for a sample is a sum of contribution scores of the bins and grid cells that this sample lies in low-dimensional subspaces, it works similar like a scoring system which only involves data binning and score addition. Therefore, our proposed method naturally provides interpretable prediction. Our experimental results demonstrate that our proposed method efficiently provides accurate and interpretable prediction.

Memory-Efficient Factorization Machines via Binarizing both Data and Model Coefficients

“…-NFM: Neural Factorization Machines (He and Chua, 2017) which combines the FM with the neural networks. -SEFM: It improves expressive limitation of FM by one-hot encoding to each input features (Lan and Geng, 2019). -Binarized FM: our proposed method.…”

“…As shown in (1), the relationship of individual features and feature interactions to the class label in FM is just linear and therefore it fails to capture highly nonlinear pattern in the data. Recently, SEFM Lan and Geng (2019) was proposed to address the expressiveness limitation of FM. The basic idea of SEFM is to apply one-hot encoding to each individual feature.…”

“…We summarize our proposed method in Algorithm 1. The step 1 obtains the new feature representation Z by applying one-hot encoding on each feature in input data X which can be done in O(nd) time (Lan and Geng, 2019) for each nonzero z ij do 13:…”

“…of features. While FM has been successfully applied in different predictive tasks, it is also be noted that FM fails to capture highly nonlinear patterns in the data because it only uses polynomial feature expansion for nonlinear relationship modeling (He and Chua, 2017;Liu et al, 2017;Lan and Geng, 2019;Chen et al, 2019). To address the expressiveness limitation of the FM model, different methods have been proposed: (1) Embedding neural networks into FM models (He and Chua, 2017;Xiao et al, 2017;Guo et al, 2017): they use neural-network based approach to model feature interactions in a highly nonlinear way; (2) Locally Linear FM (LLFM) (Liu et al, 2017) and its extension (Chen et al, 2019): they use a local encoding technique to capture nonlinear concept based on the idea that nonlinear manifold behaves linearly in its local neighbors; (3) Subspace Encoding FM (SEFM) (Lan and Geng, 2019): it captures the nonlinear relationships of individual features and feature interactions to the class label by applying explicit nonlinear feature mapping to each input feature.…”

Task-distribution-aware Meta-learning for Cold-start CTR Prediction

“…We now come to the optimization for the min-max objective function obj (•, •) in Eq. (15). Consider that if the parameter p is given, the problem is then casted as minimization for a non-convex function obj (•, p).…”