(A) study on architecture search using continuous relaxation

Abstract

DARTS is a simple and efficient architecture search method using continuous relaxation. In this paper, we suggest a couple of problems in DARTS verified by experiment and provide the solution for each. First, we show that DARTS has difficulties in the exploration of large search spaces, which leads to strong dependencies between operations in the one-shot network. Through experiments, we verify that DARTS cannot effectively find optimal solutions in the search space, and that removing operations in the derivation process of DARTS can highly affect other operations due to strong dependencies. Based on these observations, we propose DARTS In DARTS (DID), which is designed to break dependencies between operations. In DID, we randomly drop a certain number of operations from the search space at every iteration, so that operations do not rely on the others but rather learn useful weight parameters by itself. The main advantage of this method lies on its robustness against pruning of operations, which justifies the derivation procedure as in DARTS. Second, DARTS has a critical problem of inconsistency between the relaxed model and sampled model. We solve this problem by our method called STable ARchiTecture Search (STARTS) that directly trains the sampled models and show improved performance through extensive experiments. Also, this method shows better stability by leveraging weight sharing, which helps to keep consistent sampling ability under various search spaces. STARTS finds the image classifier on CIFAR-10 with almost state-of-the-art performance and the recurrent neural network for language modeling on Penn-Treebank which is comparable to the state-of-the-art model.