Accelerating Neural Architecture Search via Proxy Data

Na, Byunggook; Mok, Jisoo; Choe, Hyeokjun; Yoon, Sungroh

doi:10.48550/arxiv.2106.04784

Cited by 2 publications

(5 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, Liu et al [21] randomly selected a small number of medical images from the entire dataset to train the searched architectures. However, random sampling may lead to the removal of some representative samples, thus greatly weakening the generalization of the architecture trained in the subset [22], [23]. This will lead to an inaccurate evaluation of the performance.…”

Section: A N -Shot Evaluation Methodsmentioning

confidence: 99%

“…Then, they removed the samples that have a small impact on the performance of architectures. Na et al [23] used the data entropy to analyze five sampling methods (i.e., random, entropy top-k, entropy bottom-k, forgetting events, and k-center). They concluded that the low-entropy samples could help search for competitive architectures when the size of the subset is small.…”

Section: A N -Shot Evaluation Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Efficient Evaluation Methods for Neural Architecture Search: A Survey

Xie¹,

Song²,

Lv³

et al. 2023

Preprint

View full text Add to dashboard Cite

Neural Architecture Search (NAS) has received increasing attention because of its exceptional merits in automating the design of Deep Neural Network (DNN) architectures. However, the performance evaluation process, as a key part of NAS, often requires training a large number of DNNs. This inevitably causes NAS computationally expensive. In past years, many Efficient Evaluation Methods (EEMs) have been proposed to address this critical issue. In this paper, we comprehensively survey these EEMs published up to date, and provide a detailed analysis to motivate the further development of this research direction. Specifically, we divide the existing EEMs into four categories based on the number of DNNs trained for constructing these EEMs. The categorization can reflect the degree of efficiency in principle, which can in turn help quickly grasp the methodological features. In surveying each category, we further discuss the design principles and analyze the strength and weaknesses to clarify the landscape of existing EEMs, thus making easily understanding the research trends of EEMs. Furthermore, we also discuss the current challenges and issues to identify future research directions in this emerging topic. To the best of our knowledge, this is the first work that extensively and systematically surveys the EEMs of NAS.

show abstract

Section: A N -Shot Evaluation Methodsmentioning

confidence: 99%

Section: A N -Shot Evaluation Methodsmentioning

confidence: 99%

Efficient Evaluation Methods for Neural Architecture Search: A Survey

Xie¹,

Song²,

Lv³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…EPC-DARTS [17] then introduces the side operation to improve the performance of the searched model under lower sampling rates (e.g., 1/8) of input channels. Proxy Data [15] finds that proxy data constructed by existing data selection methods are not always appropriate for NAS, and designs a new proxy data selection method for efficient NAS. Different from these works, we present the first attempt to investigate the robustness of different NAS under kinds of proxies, which is complementary to the above works that find better proxies for specific NAS.…”

Section: Proxies Are Non-negligible On Nasmentioning

confidence: 99%

“…On the other hand, proxies are non-negligible as they affect the computation and memory cost as well as the search effectiveness and efficiency. Therefore, a few recent works [14], [15], [17] propose to find better proxies to realize more efficient NAS. Different from these works, we propose to investigate the robustness of different NAS under kinds of proxies for the first time.…”

Section: Motivationmentioning

confidence: 99%

“…PC-DARTS [16] and EPC-DARTS [17] explore how to randomly sample a subset of input channels to reduce the memory cost of differentiable NAS. Proxy Data [15] designs a new proxy data selection method for efficient NAS. However, all these works focus on finding better proxies for NAS, while the robustness of different NAS methods under various proxies has not been explored.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

$β$-DARTS++: Bi-level Regularization for Proxy-robust Differentiable Architecture Search

Ye¹,

He²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

Neural Architecture Search (NAS) has attracted increasing attention in recent years because of its capability to design neural networks automatically. Among them, differential NAS approaches such as DARTS, have gained popularity for the search efficiency. However, they still suffer from three main issues, that are, the weak stability due to the performance collapse, the poor generalization ability of the searched architectures, and the inferior robustness to different kinds of proxies (i.e., computationally reduced settings). To solve the stability and generalization problems, a simple-but-effective regularization method, termed as Beta-Decay, is proposed to regularize the DARTS-based NAS searching process (referred as β-DARTS). Specifically, Beta-Decay regularization can impose constraints to keep the value and variance of activated architecture parameters from being too large, thereby ensuring the fair competition among architecture parameters and making the supernet less sensitive to the impact of input on the operation set. In-depth theoretical analyses on how it works and why it works are provided. Comprehensive experiments on a variety of search spaces and datasets validate that Beta-Decay regularization can help to stabilize the searching process and makes the searched network more transferable across different datasets. To address the robustness problem, we first benchmark different NAS methods under a wide range of proxy data, proxy channels, proxy layers and proxy epochs, since the robustness of NAS under different kinds of proxies has not been explored before. We then conclude some interesting findings and find that β-DARTS always achieves the best result among all compared NAS methods under almost all proxies. We further introduce the novel flooding regularization to the weight optimization of β-DARTS (termed as Bi-level regularization), and experimentally and theoretically verify its effectiveness for improving the proxy robustness of differentiable NAS. In summary, our search scheme shows lots of outstanding properties for practical applications, and the code is available at https://github.com/Sunshine-Ye/Beta-DARTS.

show abstract

Accelerating Neural Architecture Search via Proxy Data

Cited by 2 publications

References 17 publications

Efficient Evaluation Methods for Neural Architecture Search: A Survey

Efficient Evaluation Methods for Neural Architecture Search: A Survey

$β$-DARTS++: Bi-level Regularization for Proxy-robust Differentiable Architecture Search

Contact Info

Product

Resources

About