Expectation Propagation for Bayesian Multi-task Feature Selection

Abstract

Abstract. In this paper we propose a Bayesian model for multi-task feature selection. This model is based on a generalized spike and slab sparse prior distribution that enforces the selection of a common subset of features across several tasks. Since exact Bayesian inference in this model is intractable, approximate inference is performed through expectation propagation (EP). EP approximates the posterior distribution of the model using a parametric probability distribution. This posterior approximation is particularly useful to identify relevant features for prediction. We focus on problems for which the number of features d is significantly larger than the number of instances for each task. We propose an efficient parametrization of the EP algorithm that offers a computational complexity linear in d. Experiments on several multi-task datasets show that the proposed model outperforms baseline approaches for single-task learning or data pooling across all tasks, as well as two state-of-the-art multi-task learning approaches. Additional experiments confirm the stability of the proposed feature selection with respect to various sub-samplings of the training data.

Hierarchical sparse modeling using Spike and Slab priors

“…Also we propose two hierarchical sparse models using Spike and Slab priors and relate them to HiLasso and C-HiLasso. Prior works using Spike and Slab priors for multi-task learning problems [15,16] only consider the block sparsity among different tasks while our work consider both the block sparsity across tasks and the group sparsity inside each task.…”

“…Different techniques can be used such as sampling methods or approximation methods. We choose expectation propagation (EP) because of its efficiency and demonstrated success for multi-task learning problems [16].…”

“…Although Spike and Slab priors have been used in multitask sparse modeling [15,16], only block sparsity (correlation between different tasks) has been considered. In this section, we will provide an extension of collaborative hierarchical sparse modeling using Spike and Slab priors (CHi-BCS).…”

The Variational Garrote

“…Since γ and β are given, the VG algorithm reduces to iterate eqs. (8) and (9) starting from a random m. Similarly, the PMF reduces to perform an E-step given the fixed hyperparameter values.…”

Variable Selection and Task Grouping for Multi-Task Learning

“…Traditional studies are based on a strict assumption that selected variables are shared among all tasks [22,27]. Recent studies have suggested a more flexible approach that involves selecting variables by decomposing a coefficient into a shared part and an individual part [12,15] or factorizing a coefficient using a variable specific part and a task-variable part [32]. Although the variable selection approach provides better interpretability than the other approaches, it has limited ability to share common information among related tasks.…”