EAT-NAS: Elastic Architecture Transfer for Accelerating Large-scale Neural Architecture Search

Fang, Jiemin; Chen, Yukang; Zhang, Xinbang; Zhang, Qian; Huang, Chang; Meng, Guang; Liu, Wenyu; Wang, Xinggang

doi:10.48550/arxiv.1901.05884

Cited by 5 publications

(6 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, Google proposed the ENAS algorithm [12], which avoids inefficient initial training by forcing all submodels to share weights, thereby improving the efficiency of ANS and overcoming the shortcomings of high computational cost and time-consuming of ANS. [20], which simultaneously evolve along network structures and hyper-parameters to generate neural networks. These methods first randomly generate a population (N sets of solutions) and begin to cycle through the following steps: selection, crossover, mutation, until the final condition is met.…”

Section: Automatic Network Searchmentioning

confidence: 99%

Knowledge Graph Representation Learning Based on Automatic Network Search for Link Prediction

Gu¹,

Chen²

2023

Computer Modeling in Engineering &Amp; Sciences

View full text Add to dashboard Cite

Link prediction, also known as Knowledge Graph Completion (KGC), is the common task in Knowledge Graphs (KGs) to predict missing connections between entities. Most existing methods focus on designing shallow, scalable models, which have less expressive than deep, multi-layer models. Furthermore, most operations like addition, matrix multiplications or factorization are handcrafted based on a few known relation patterns in several wellknown datasets, such as FB15k, WN18, etc. However, due to the diversity and complex nature of real-world data distribution, it is inherently difficult to preset all latent patterns. To address this issue, we propose KGE-ANS, a novel knowledge graph embedding framework for general link prediction tasks using automatic network search. KGE-ANS can learn a deep, multi-layer effective architecture to adapt to different datasets through neural architecture search. In addition, the general search space we designed is tailored for KG tasks. We perform extensive experiments on benchmark datasets and the dataset constructed in this paper. The results show that our KGE-ANS outperforms several state-of-the-art methods, especially on these datasets with complex relation patterns. KEYWORDS Knowledge graph embedding; link prediction; automatic network search 1 IntroductionKnowledge Graphs (KGs) are graph-structured information networks. Typical KGs such as Freebase [1] and DBpedia [2], represent the knowledge in the form of the triplet (h, r, t), where the head h and tail t are entities, and the relation r refers to different types of edges between entities. Recently, KGs have been applied in many fields and achieved significant performance, such as question answering [3] and dialog system [4]. Growing efforts from both academia and industry have been made to advance the research on KGs. However, most existing KGs are incomplete and noisy, which severely limits their popularity and usefulness in practice. For example, in Freebase, more than two-thirds of person entities lack relations with the corresponding birthplace entities [5]. To tackle the missing link problem, a link prediction task has been proposed to predict the existence of links between any two entities [6], which quickly becomes a fundamental but challenging task in the KG field.

show abstract

Section: Automatic Network Searchmentioning

confidence: 99%

Knowledge Graph Representation Learning Based on Automatic Network Search for Link Prediction

Gu¹,

Chen²

2023

Computer Modeling in Engineering &Amp; Sciences

View full text Add to dashboard Cite

show abstract

“…Following this manner, EAS (Cai et al, 2018) extends the parameter remapping concept to neural architecture search. Moreover, some NAS works (Pham et al, 2018;Fang et al, 2019a;Elsken et al, 2019) apply parameters sharing on child models to accelerate the search process. Our parameter remapping paradigm extends the mapping dimension with the kernel level.…”

Section: Related Workmentioning

confidence: 99%

Fast Neural Network Adaptation via Parameter Remapping and Architecture Search

Fang,

Sun,

Peng

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Deep neural networks achieve remarkable performance in many computer vision tasks. Most state-of-the-art (SOTA) semantic segmentation and object detection approaches reuse neural network architectures designed for image classification as the backbone, commonly pre-trained on ImageNet. However, performance gains can be achieved by designing network architectures specifically for detection and segmentation, as shown by recent neural architecture search (NAS) research for detection and segmentation. One major challenge though, is that ImageNet pre-training of the search space representation (a.k.a. super network) or the searched networks incurs huge computational cost. In this paper, we propose a Fast Neural Network Adaptation (FNA) method, which can adapt both the architecture and parameters of a seed network (e.g. a high performing manually designed backbone) to become a network with different depth, width, or kernels via a Parameter Remapping technique, making it possible to utilize NAS for detection/segmentation tasks a lot more efficiently. In our experiments, we conduct FNA on MobileNetV2 to obtain new networks for both segmentation and detection that clearly out-perform existing networks designed both manually and by NAS. The total computation cost of FNA is significantly less than SOTA segmentation/detection NAS approaches: 1737× less than DPC, 6.8× less than Auto-DeepLab and 7.4× less than DetNAS. The code is available at https://github.com/JaminFong/FNA.

show abstract

“…These include meta-learning based approaches [36], [37] with application to few-shot learning tasks. XferNAS [38] and EAT-NAS [39] illustrate how architectures can be transferred between similar datasets or from smaller to larger datasets. Some approaches [40], [41] proposed RL-based NAS methods that search on multiple tasks during training and transfer the learned search strategy, as opposed to searched networks, to new tasks at inference.…”

Section: Related Workmentioning

confidence: 99%

Neural Architecture Transfer

Lu,

Sreekumar,

Goodman

et al. 2020

Preprint

View full text Add to dashboard Cite

Neural architecture search (NAS) has emerged as a promising avenue for automatically designing task-specific neural networks. Most existing NAS approaches require one complete search for each deployment specification of hardware or objective. This is a computationally impractical endeavor given the potentially large number of application scenarios. In this paper, we propose Neural Architecture Transfer (NAT) to overcome this limitation. NAT is designed to efficiently generate task-specific custom models that are competitive even under multiple conflicting objectives. To realize this goal we learn task-specific supernets from which specialized subnets can be sampled without any additional training. The key to our approach is an integrated online transfer learning and many-objective evolutionary search procedure. A pre-trained supernet is iteratively adapted while simultaneously searching for task-specific subnets. We demonstrate the efficacy of NAT on 11 benchmark image classification tasks ranging from large-scale multi-class to small-scale fine-grained datasets. In all cases, including ImageNet, NATNets improve upon the state-of-the-art under mobile settings (≤ 600M Multiply-Adds). Surprisingly, small-scale fine-grained datasets benefit the most from NAT. At the same time, the architecture search and transfer is orders of magnitude more efficient than existing NAS methods. Overall, experimental evaluation indicates that, across diverse image classification tasks and computational objectives, NAT is an appreciably more effective alternative to fine-tuning based transfer learning. Code is available at https://github.com/human-analysis/neural-architecture-transfer.

show abstract

EAT-NAS: Elastic Architecture Transfer for Accelerating Large-scale Neural Architecture Search

Cited by 5 publications

References 15 publications

Knowledge Graph Representation Learning Based on Automatic Network Search for Link Prediction

Knowledge Graph Representation Learning Based on Automatic Network Search for Link Prediction

Fast Neural Network Adaptation via Parameter Remapping and Architecture Search

Neural Architecture Transfer

Contact Info

Product

Resources

About