XNAS: Neural Architecture Search with Expert Advice

Nayman, Niv; Noy, Asaf; Ridnik, Tal; Friedman, Itamar; Jin, Rong; Zelnik-Manor, Lihi

doi:10.48550/arxiv.1906.08031

Cited by 6 publications

(8 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the wake of the DARTS's open-sourcing [18], a diverse number of its variants emerge in the neural architecture search community. Some of them extend its use in higher-level architecture search spaces with performance awareness in mind [4,29], some learn a stochastic distribution instead of architectural parameters [29,30,33,7,8], and others offer remedies on discovering its lack of robustness [21,5,16,13,32].…”

Section: Introductionmentioning

confidence: 99%

Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search

Chu

Zhou

Zhang

et al. 2019

Preprint

View full text Add to dashboard Cite

Differential Architecture Search (DARTS) is now a widely disseminated weight-sharing neural architecture search method. However, there are two fundamental weaknesses remain untackled. First, we observe that the wellknown aggregation of skip connections during optimization is caused by an unfair advantage in an exclusive competition. Second, there is a non-negligible incongruence when discretizing continuous architectural weights to a one-hot representation. Because of these two reasons, DARTS delivers a biased solution that might not even be suboptimal. In this paper, we present a novel approach to curing both frailties. Specifically, as unfair advantages in a pure exclusive competition easily induce a monopoly, we relax the choice of operations to be collaborative, where we let each operation have an equal opportunity to develop its strength. We thus call our method Fair DARTS. Moreover, we propose a zero-one loss to directly reduce the discretization gap. Experiments are performed on two mainstream search spaces, in which we achieve new state-of-the-art networks on Ima-geNet. Our code is available here 1 .

show abstract

Section: Introductionmentioning

confidence: 99%

Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search

Chu

Zhou

Zhang

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…= f (x; α, ω), where α and ω indicate the architectural parameters and network weights, respectively. DARTS [22] and its variants [4,25,37] have relied on many manually designed rules to determine the final architecture. Examples include each edge can only preserve one operator, each inner node can preserve two of its precursors, and the architecture is shared by the same type (normal and reduction) of cells.…”

Section: Breaking the Rules: Enlarging The Search Spacementioning

confidence: 99%

GOLD-NAS: Gradual, One-Level, Differentiable

Bi¹,

Xie²,

Chen³

2020

Preprint

View full text Add to dashboard Cite

There has been a large literature of neural architecture search, but most existing work made use of heuristic rules that largely constrained the search flexibility. In this paper, we first relax these manually designed constraints and enlarge the search space to contain more than 10 160 candidates. In the new space, most existing differentiable search methods can fail dramatically. We then propose a novel algorithm named Gradual One-Level Differentiable Neural Architecture Search (GOLD-NAS) which introduces a variable resource constraint to one-level optimization so that the weak operators are gradually pruned out from the supernetwork. In standard image classification benchmarks, GOLD-NAS can find a series of Pareto-optimal architectures within a single search procedure. Most of the discovered architectures were never studied before, yet they achieve a nice tradeoff between recognition accuracy and model complexity. We believe the new space and search algorithm can advance the search of differentiable NAS.Preprint. Under review.

show abstract

“…Many recent works [2,1,3] aim to reduce the search time by training a single over-parameterized network with inherited/shared weights. For example, DARTS [27] and its variants [50,9,30,5,23,58,54,6] relaxed the architecture representation to the continuous domain to make the search differentiable. Recenly, NAS has been explored in 3D [23,43].…”

Section: Related Workmentioning

confidence: 99%

“…Although encouraging in their results, early NAS approaches were limited by their massive computational needs. Recent works like DARTS [27] and its variants [50,9,30,5,23,58,54,6] formulate architecture search as a differentiable problem, which greatly alleviates computational complexity. Further advancements are proposed in recent work like SGAS [23], where optimal architectures show better generalization between search and final tasks.…”

Section: Introductionmentioning

confidence: 99%

LC-NAS: Latency Constrained Neural Architecture Search for Point Cloud Networks

Li,

Xu,

Giancola

et al. 2020

Preprint

View full text Add to dashboard Cite

Point cloud architecture design has become a crucial problem for 3D deep learning. Several efforts exist to manually design architectures with high accuracy in point cloud tasks such as classification, segmentation, and detection. Recent progress in automatic Neural Architecture Search (NAS) minimizes the human effort in network design and optimizes high performing architectures. However, these efforts fail to consider important factors such as latency during inference. Latency is of high importance in time critical applications like self-driving cars, robot navigation, and mobile applications, that are generally bound by the available hardware. In this paper, we introduce a new NAS framework, dubbed LC-NAS, where we search for point cloud architectures that are constrained to a target latency. We implement a novel latency constraint formulation to trade-off between accuracy and latency in our architecture search. Contrary to previous works, our latency loss guarantees that the final network achieves latency under a specified target value. This is crucial when the end task is to be deployed in a limited hardware setting. Extensive experiments show that LC-NAS is able to find state-of-the-art architectures for point cloud classification in Model-Net40 with minimal computational cost. We also show how our searched architectures achieve any desired latency with a reasonably low drop in accuracy. Finally, we show how our searched architectures easily transfer to a different task, part segmentation on PartNet, where we achieve state-of-the-art results while lowering latency by a factor of 10.

show abstract

XNAS: Neural Architecture Search with Expert Advice

Cited by 6 publications

References 40 publications

Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search

Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search

GOLD-NAS: Gradual, One-Level, Differentiable

LC-NAS: Latency Constrained Neural Architecture Search for Point Cloud Networks

Contact Info

Product

Resources

About