Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernels

Abstract

The performance of multi-task learning in Convolutional Neural Networks (CNNs) hinges on the design of feature sharing between tasks within the architecture. The number of possible sharing patterns are combinatorial in the depth of the network and the number of tasks, and thus hand-crafting an architecture, purely based on the human intuitions of task relationships can be time-consuming and suboptimal. In this paper, we present a probabilistic approach to learning task-specific and shared representations in CNNs for multi-task learning. Specifically, we propose "stochastic filter groups" (SFG), a mechanism to assign convolution kernels in each layer to "specialist" or "generalist" groups, which are specific to or shared across different tasks, respectively. The SFG modules determine the connectivity between layers and the structures of task-specific and shared representations in the network. We employ variational inference to learn the posterior distribution over the possible grouping of kernels and network parameters. Experiments demonstrate that the proposed method generalises across multiple tasks and shows improved performance over baseline methods.

Multi-Task Learning for Dense Prediction Tasks: A Survey

“…To alleviate this issue, several recent works [9], [10], [11], [12] proposed efficient design procedures that automatically decide where to share or branch within the network. Similarly, stochastic filter groups [56] re-purposed the convolution kernels in each layer to support shared or task-specific behaviour. Soft Parameter Sharing.…”

“…Huang et al [67] introduced a method rooted in Neural Architecture Search (NAS) for the automated construction of a tree-based multi-attribute learning network. Stochastic filter groups [56] re-purposed the convolution kernels in each layer of the network to support shared or taskspecific behaviour. In a similar vein, feature partitioning [68] presented partitioning strategies to assign the convolution kernels in each layer of the network into different tasks.…”

Reparameterizing Convolutions for Incremental Multi-Task Learning Without Task Interference

“…Encoder-focused approaches primarily emphasize on architectures that can encode multi-purpose feature representations through supervision from multiple tasks. Such encoding is typically achieved, for example, via feature fusion [41], branching [25,43,36,61], selfsupervision [10], attention [33], or filter grouping [1]. Decoder-focused approaches start from the feature representations learned at the encoding stage, and further refine them at the decoding stage by distilling information across tasks in a one-off [63], sequential [65], recursive [66], or even multi-scale [62] manner.…”

“…When it comes to learning multiple tasks under a single model, multi-task learning (MTL) techniques [2,54] have been employed in the literature. On the one hand, encoder-focused approaches [41,25,36,10,43,33,1,61] emphasize learning feature representations from multi-task supervisory signals by employing architectures that encode shared and task-specific information. On the other hand, decoder-focused approaches [63,65,66,62] utilize the multi-task feature representations learned at the encoding stage to distill cross-task information at the decoding stage, thus refining the original feature representations.…”

A Multi-task Learning Model for Daily Activity Forecast in Smart Home

“…Attention-based LSTM was then used for feature embedding, but the outliers were not handled effectively, which could affect the model performance [43]. According to the needs of each task, CNNs stochastic filter groups grouped the convolution kernel of each convolution layer [44]. There are some other networks such as branched multi-task networks [45], sluice networks [46] and learning sparse sharing [47] to address multiple task sharing issues, but it was difficult to train them due to the high complexity of the model.…”