Quaternion Factorization Machines: A Lightweight Solution to Intricate Feature Interaction Modelling

Abstract

Due to the sparsity of available features in web-scale predictive analytics, combinatorial features become a crucial means for deriving accurate predictions. As a well-established approach, factorization machine (FM) is capable of automatically learning high-order interactions among features to make predictions without the need for manual feature engineering. With the prominent development of deep neural networks (DNNs), there is a recent and ongoing trend of enhancing the expressiveness of FM-based models with DNNs. However, though better results are obtained with DNN-based FM variants, such performance gain is paid off by an enormous amount (usually millions) of excessive model parameters on top of the plain FM. Consequently, the heavy parameterization impedes the real-life practicality of those deep models, especially efficient deployment on resourceconstrained IoT and edge devices.In this paper, we move beyond the traditional real space where most deep FM-based models are defined, and seek solutions from quaternion representations within the hypercomplex space. Specifically, we propose the quaternion factorization machine (QFM) and quaternion neural factorization machine (QNFM), which are two novel lightweight and memory-efficient quaternion-valued models for sparse predictive analytics. By introducing a brand new take on FM-based models with the notion of quaternion algebra, our models not only enable expressive inter-component feature interactions, but also significantly reduce the parameter size due to lower degrees of freedom in the hypercomplex Hamilton product compared with real-valued matrix multiplication. Extensive experimental results on three large-scale datasets demonstrate that QFM achieves 4.36% performance improvement over the plain FM without introducing any extra parameters, while QNFM outperforms all baselines with up to two magnitudes' parameter size reduction in comparison to stateof-the-art peer methods.

CDRec: Cayley-Dickson Recommender

“…Besides, QNFM outperforms QFM in most cases. Such results are consistent with prior studies [6,14], indicating the power of deep learning techniques in recommender systems.…”

“…sedenions S ∈ H 16 , and trigintaduonions T ∈ H 32 . To verify the performance of CDRec, we compare it with the following methods, covering the classical models (MF [17], DMF [39], and NeuCF [14]), the GCN-based models (GCMC [3], NGCF [38], LightGCN [11], and BiGI [4]), and the hypercomplex-valued models (CCF [42], QCF [42], QFM [6], QNFM [6], and QGNN [23]). The characteristics of the comparison methods are included in supplement C.2.…”

“…The linear aggregator models a summary of the neighborhood, whose design is inspired by the GCNs [16]. In addition, since the feature interaction is important for recommendation [6,42], we carefully design an interaction aggregator to capture neighbor feature interactions.…”

“…Recently, there is a surge in studying the recommendation models in hypercomplex spaces (non-real spaces) [6,22,32,42]. In mathematics, hypercomplex algebra is defined as unital, distributive algebra, not necessarily associative or commutative [20].…”

“…Thereby, hypercomplex algebras have a richer representation capacity for modeling users and items. Based on these advantages, researchers propose diverse hypercomplex recommendation models which boost the performance over some real-valued recommenders [6,42].…”