Evolutionary architecture search for deep multitask networks

Abstract

Multitask learning, i.e. learning several tasks at once with the same neural network, can improve performance in each of the tasks. Designing deep neural network architectures for multitask learning is a challenge: There are many ways to tie the tasks together, and the design choices matter. The size and complexity of this problem exceeds human design ability, making it a compelling domain for evolutionary optimization. Using the existing state of the art soft ordering architecture as the starting point, methods for evolving the modules of this architecture and for evolving the overall topology or routing between modules are evaluated in this paper. A synergetic approach of evolving custom routings with evolved, shared modules for each task is found to be very powerful, significantly improving the state of the art in the Omniglot multitask, multialphabet character recognition domain. This result demonstrates how evolution can be instrumental in advancing deep neural network and complex system design in general.MTL [3] exploits relationships across problems to increase overall performance. The underlying idea is that if multiple tasks are related, the optimal models for those tasks will be related as well. In the convex optimization setting, this idea has been implemented via various regularization penalties on shared parameter matrices [1,7,18,22]. Evolutionary methods have also had success in MTL, especially in sequential decision-making domains [13,16,19,38,41].

Evolutionary neural AutoML for deep learning

“…In addition to the three goals of evolutionary AutoML demonstrated in this paper, a fourth one is to take advantage of multiple related datasets. As shown in prior work [30], even when there is little data to train a DNN in a particular task, other tasks in a multitask setting can help achieve good performance. Evolutionary AutoML thus forms a framework for utilizing DNNs in domains that otherwise would be impractical due to lack of data.…”

Stochastic Filter Groups for Multi-Task CNNs: Learning Specialist and Generalist Convolution Kernels

“…SFGs probabilistically defines the grouping of kernels and thus the connectivity of features in a CNNs. We use variational inference to approximate the distribution (ii) (i) (iii) (iv) Our method can be considered as a probabilistic form of multi-task architecture learning [34], as the learned posterior embodies the optimal MTL architecture given the data.…”

Many Task Learning With Task Routing

“…A recent step in this direction is made by Liu et al [19] who propose an adaptive MTL model that structurally groups tasks together. Evolutionary algorithms have also been shown to capture task relatedness and create sharing structures [16]. A less architectural solution is proposed by Yang et al [40] who use a factorized space representation to initialize and learn intertask sharing structures at each layer in an MTL model.…”

“…This search duration grows proportionally with the number of tasks and parameters present in the model's structure. Previous works in both MTL and STL rely on evolutionary algorithms [16] or factorization techniques [40] to discover their optimal way of learning, however this takes time and prolongs the training process. In our work, inspired by the efficiency of Random Search [3] we enforce a structured random solution to this problem by regulating the per-task data-flow in our models.…”