Summary of Encodings For Prediction-based Neural Architecture Search, by Yash Akhauri et al.
Encodings for Prediction-based Neural Architecture Search
by Yash Akhauri, Mohamed S. Abdelfattah
First submitted to arxiv on: 4 Mar 2024
Categories
- Main: Machine Learning (cs.LG)
- Secondary: Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV); Neural and Evolutionary Computing (cs.NE)
GrooveSquid.com Paper Summaries
GrooveSquid.com’s goal is to make artificial intelligence research accessible by summarizing AI papers in simpler terms. Each summary below covers the same AI paper, written at different levels of difficulty. The medium difficulty and low difficulty versions are original summaries written by GrooveSquid.com, while the high difficulty version is the paper’s original abstract. Feel free to learn from the version that suits you best!
| Summary difficulty | Written by | Summary |
|---|---|---|
| High | Paper authors | High Difficulty Summary Read the original abstract here |
| Medium | GrooveSquid.com (original content) | Medium Difficulty Summary Predictor-based methods have significantly improved Neural Architecture Search (NAS) optimization. The effectiveness of these predictors relies heavily on the method used to encode neural network architectures. While traditional encodings employ an adjacency matrix describing the graph structure, novel approaches include unsupervised pretraining of latent representations and vectors of zero-cost proxies. This paper categorizes and investigates three main types of neural encodings: structural, learned, and score-based. Furthermore, it introduces unified encodings that extend NAS predictors to multiple search spaces. The study draws from experiments conducted on over 1.5 million neural network architectures across various NAS spaces, including NB101, NB201, NB301, NDS, and TransNASBench-101. Building upon this study, the paper presents FLAN: Flow Attention for NAS, a predictor that integrates insights on design, transfer learning, and unified encodings to achieve more than an order of magnitude cost reduction in training NAS accuracy predictors. |
| Low | GrooveSquid.com (original content) | Low Difficulty Summary This paper is about making it easier to find the best ways to design neural networks. Neural networks are used in artificial intelligence and machine learning, but designing them can be a difficult task. The researchers looked at different methods for encoding neural network architectures, which helps computers understand what the network does. They found that some methods work better than others and developed a new way of encoding called unified encodings. This helps reduce the time it takes to train neural networks by more than 10 times. The paper also presents a new predictor called FLAN, which uses this new way of encoding to find the best neural network designs. |
Keywords
* Artificial intelligence * Attention * Machine learning * Neural network * Optimization * Pretraining * Transfer learning * Unsupervised
