An efficient algorithm for weak hierarchical lasso

Abstract

Linear regression is a widely used tool in data mining and machine learning. In many applications, fitting a regression model with only linear effects may not be sufficient for predictive or explanatory purposes. One strategy which has recently received increasing attention in statistics is to include feature interactions to capture the nonlinearity in the regression model. Such model has been applied successfully in many biomedical applications. One major challenge in the use of such model is that the data dimensionality is significantly higher than the original data, resulting in the small sample size large dimension problem. Recently, weak hierarchical Lasso, a sparse interaction regression model, is proposed that produces sparse and hierarchical structured estimator by exploiting the Lasso penalty and a set of hierarchical constraints. However, the hierarchical constraints make it a non-convex problem and the existing method finds the solution of its convex relaxation, which needs additional conditions to guarantee the hierarchical structure. In this paper, we propose to directly solve the non-convex weak hierarchical Lasso by making use of the GIST (General Iterative Shrinkage and Thresholding) optimization framework which has been shown to be efficient for solving non-convex sparse formulations. The key step in GIST is to compute a sequence of proximal operators. One of our key technical contributions is to show that the proximal operator associated with the non-convex weak hierarchical Lasso admits a closed form solution. However, a naive approach for solving each subproblem of the proximal operator leads to a quadratic time complexity, which is not desirable for largesize problems. To this end, we further develop an efficient algorithm for computing the subproblems with a linearithmic time complexity. We have conducted extensive experiments on both synthetic and real data sets. Results show that our proposed algorithm is much more efficient and effective than its convex relaxation.

Generalized Hierarchical Sparse Model for Arbitrary-Order Interactive Antigenic Sites Identification in Flu Virus Data

“…(5). Its R package ‘hierNet’ is available in ‘CRAN’; (4) eWHL [12], which is an efficient implementation for the w-hierNet method proposed in [1]; (5) strong hierNet (s-hierNet) [1], which is the HSM method using up to the 2nd-order interaction effects with strong heredity, i.e. solving the problem in Eq.…”

“…To measure the performance of different methods, we use the sensitivity (Sen.) and the specificity (Spe.) [12], where non-zero entries in the corresponding coefficient vector are treated as positive and zero entries are negative, for each order of interactions to test the recovery performance on the model coefficients and use the root mean square error (RMSE) on a test set having 200 samples for each setting. For each setting, we repeat each configuration for 10 times to report the average results.…”

“…The response vector is y ∈ ℝ n . The second-order interaction models [15, 3, 1, 11, 12, 16, 9, 22] commonly consider the following regression model: $$\mathrm{normaly}=true\sum _{i=1}^d{\beta }_i{\mathrm{x}}_i+true\sum _{i\ne j}^d{\varphi }_{i,j}false({\mathrm{x}}_i\odot {\mathrm{x}}_{j}false)+\epsilon ,$$where ⊙ denotes the element-wise product between vectors, β ∈ ℝ d with β i as its i th element is the coefficient vector for the original covariates, Φ ∈ ℝ d×d with ϕ i,j as its ( i, j )th element is the coefficient matrix for the pairwise interaction effects, and $\varepsilon ~\mathrm{scriptN}false(\mathrm{bold0},{\sigma }^2\mathrm{boldI}false)$ is a Gaussian noise vector. In the existing works, two types of heredity structure are considered for the second-order interactions, i.e.…”

“…the strong heredity and the weak heredity, whose definitions are as follow: $$\begin{array}{c}left\text{strong heredity}:{\varphi }_{i,j}\ne 0\Rightarrow {\beta }_i\ne 0\text{and }{\beta }_j\ne 0,\\ \text{or equivalently,}\\ {\beta }_i=0\text{or }{\beta }_j=0\Rightarrow {\varphi }_{i,j}=0;\end{array}$$ $$\begin{array}{c}left\text{weak heredity}:{\varphi }_{i,j}\ne 0\Rightarrow {\beta }_i\ne 0\text{or }{\beta }_j\ne 0,\\ \text{or equivalently},\\ {\beta }_i=0\text{and }{\beta }_j=0\Rightarrow {\varphi }_{i,j}=0.\end{array}$$Based on Eqs. (2) and (3), the HSM methods introduced in [1, 12], named as the strong and weak hierNet, explicitly enforce the heredity structure by adding inequality or symmetry constraints to the Lasso method as $$\begin{array}{c}left\text{strong hierNet}:\underset{\beta ,\mathbf{\Phi }}{min}l(\beta ,\mathbf{\Phi }|\mathbf{X},\mathbf{y})+\mathrm{\lambda }{\Vert \beta \Vert }_1+\frac{\mathrm{normal\lambda }}{2}{\Vert \mathbf{\Phi }\Vert }_1\\ \mathrm{s}.\mathrm{t}.\mathbf{\Phi }={\mathrm{bold\Phi }}^T,|{\beta }_i|\ge {\Vert {\mathrm{bold\Phi }}_{i,}.\Vert }_1\forall i|\in {\mathbb{N}}_d;\end{array}$$ $$\begin{array}{c}left\text{weak hierNet}:\underset{\beta ,\mathbf{\Phi }}{min}l(\beta ,\mathbf{\Phi }|\mathbf{X},\mathbf{y})+\mathrm{\lambda }{\Vert \beta \Vert }_{}\end{array}$$…”

“…In addition to the strong heredity, there is another type of hierarchical relation, the weak heredity , which introduces a constraint that a pairwise interaction is considered if either of its corresponding covariates was included. Both the strong and weak heredity are investigated in [1] and an efficient algorithm for dealing with the weak heredity is introduced in [12]. …”