Grafové neuronové sítě pro odhad výkonnosti při hledání architektur
| Thesis title in Czech: | Grafové neuronové sítě pro odhad výkonnosti při hledání architektur |
|---|---|
| Thesis title in English: | Graph neural networks for NAS performance prediction |
| Key words: | hledání architektur neuronových sítí|grafové neuronové sítě|AutoML|odhad výkonnosti |
| English key words: | neural architecture search|graph neural networks|AutoML|performance prediction |
| Academic year of topic announcement: | 2020/2021 |
| Thesis type: | diploma thesis |
| Thesis language: | angličtina |
| Department: | Department of Theoretical Computer Science and Mathematical Logic (32-KTIML) |
| Supervisor: | Mgr. Roman Neruda, CSc. |
| Author: | hidden - assigned and confirmed by the Study Dept. |
| Date of registration: | 28.04.2021 |
| Date of assignment: | 28.04.2021 |
| Confirmed by Study dept. on: | 03.09.2021 |
| Date and time of defence: | 02.09.2021 09:00 |
| Date of electronic submission: | 22.07.2021 |
| Date of submission of printed version: | 22.07.2021 |
| Date of proceeded defence: | 02.09.2021 |
| Opponents: | Mgr. Martin Pilát, Ph.D. |
| Guidelines |
| With growing popularity of deep learning and its applications, the demand for efficient design of high-performing architectures of neural networks grew. The neural architecture search (NAS) adresses this problem using optimization algorithms to search for efficient architectures. A key issue in this area is the long training time of deep neural networks leading to long optimization time and excessive resource usage. One solution to this problem is to use a surrogate model that predicts the accuracy of a neural network according to its architecture. Recently, graph neural networks have been used as such models, both in a supervised manner as well as unsupervised embeddings. The goal of this thesis is to extend the graph neural network architectures in a semi-supervised manner by using additional information to guide the training, such as input and output data or the difference between two networks. The efficiency of this approach will be tested by solving the complete NAS problem. |
| References |
| 1. Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang and P. S. Yu, "A Comprehensive Survey on Graph Neural Networks," IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 1, pp. 4-24, 2021.
2. Yan, Shen & Zheng, Yu & Ao, Wei & Zeng, Xiao & Zhang, Mi: Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?. Advances in Neural Information Processing Systems (H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin, Eds), pp 12486-12498, Curran Associates, Inc., vol. 33, 2020 3. Ian J. Goodfellow, Yoshua Bengio and Aaron Courville: Deep Learning, MIT Press, 2016. 4. Xin He, Kaiyong Zhao, Xiaowen Chu: AutoML: A survey of the state-of-the-art, Knowledge-Based Systems, vol. 212, 2021. 5. Martin Wistuba, Ambrish Rawat, Tejaswini Pedapati: A Survey on Neural Architecture Search, arXiv: 1905.01392, 2019. |
| Preliminary scope of work |
| V této práci jsme vytvořili novou metodu embeddingu architektur sítí pro využití při hledání architektury neuronových sítí - info-NAS. Náš model se učí predikovat výstupy trénovaných neuronových sítí na vstupních obrazových datech. Jako vstupní data jsme zvolili dataset sítí NAS-Bench-101 a obrazový dataset CIFAR-10. Pro účely této úlohy jsme rozšířili existující unsupervised grafový variační autoencoder, arch2vec, a rozšířený model trénujeme na označených i neoznačených architekturách sítí semi-supervised způsobem. Pro vyhodnocení našeho přístupu jsme analyzovali, jak se náš model na těchto datech učí, také jsme jej porovnali s původním modelem, a nakonec jsme oba modely vyhodnotili na úloze hledání sítí na NAS-Bench-101 a při predikování výkonnosti sítě. |
| Preliminary scope of work in English |
| In this work we present a novel approach to network embedding for neural architecture search - info-NAS. The model learns to predict the output features of a trained convolutional neural network on a set of input images. We use the NAS-Bench-101 search space as the neural architecture dataset, and the CIFAR-10 as the image dataset. For the purpose of this task, we extend an existing unsupervised graph variational autoencoder, arch2vec, by jointly training on unlabeled and labeled neural architectures in a semi-supervised manner. To evaluate our approach, we analyze how our model learns on the data, compare it to the original arch2vec, and finally, we evaluate both models on the NAS-Bench-101 search task and on the performance prediction task. |