The AI Lecture Hallby Janin A Apurba
🔗 Deep Learning · Lecture 36 of 38

Neural Architecture Search and Automated Machine Learning

Can algorithms design better networks than humans? We review search spaces, reinforcement-learning and evolutionary search, differentiable NAS, weight sharing and hardware-aware search — and the lessons of the NAS era.

JA Janin A Apurba
Advanced ICT Officer, CNRS-UNHCR Advanced ⏱ 5 min read

Designing neural architectures — choosing layer types, widths, depths, connections — was long a craft of human intuition and trial and error. Neural Architecture Search (NAS) automates this: an algorithm searches over a space of possible architectures for the one that performs best. NAS produced several state-of-the-art models, most famously the EfficientNet family, and it belongs to the broader field of AutoML, which also automates hyperparameter tuning and feature engineering.

The three components of NAS#

Elsken, Metzen and Hutter (2019) describe every NAS method by three choices:

  1. Search space — which architectures can be expressed.
  2. Search strategy — how to explore that space.
  3. Performance estimation — how to evaluate a candidate cheaply.

Search spaces#

  • Global / chain-structured: choose each layer's type and hyperparameters in sequence.
  • Cell-based: search for a small repeated building block (a "cell") and stack it; this made search tractable and transferable (NASNet learned cells on CIFAR-10 and transferred them to ImageNet).
  • Hierarchical / macro spaces: choose depths, widths and resolutions per stage.

The search space encodes strong human priors. A space made only of good building blocks makes even random search competitive — an important and humbling lesson.

Search strategies#

Reinforcement learning#

Zoph and Le (2017) used an RNN controller to generate architecture descriptions, trained with policy gradients using validation accuracy as the reward. It produced excellent architectures but required enormous compute (hundreds of GPUs for weeks).

Evolutionary algorithms#

Maintain a population of architectures; mutate the good ones (add a layer, change a kernel); select by fitness. Regularized (aging) evolution (Real et al., 2019) — removing the oldest rather than the worst individuals — produced AmoebaNet, matching RL-based results.

Bayesian optimisation#

Model architecture performance with surrogates (e.g. with graph kernels or neural predictors) to choose promising candidates.

Differentiable NAS (DARTS)#

Liu et al. (2019) relaxed the discrete choice of operation on each edge into a softmax-weighted mixture:

$$ \bar{o}(\mathbf{x}) = \sum_{o \in \mathcal{O}}\frac{\exp(\alpha_o)}{\sum_{o'}\exp(\alpha_{o'})}\,o(\mathbf{x}) $$

Architecture parameters $\alpha$ and network weights are optimised jointly by gradient descent (a bilevel problem), and the strongest operation on each edge is kept at the end. This cut search cost to a few GPU-days, though DARTS can be unstable (e.g. collapsing towards parameter-free skip connections).

Cheap performance estimation#

Training every candidate to convergence is prohibitively expensive. Shortcuts:

  • Low-fidelity estimates — fewer epochs, smaller images, data subsets (combine with Hyperband).
  • Learning-curve extrapolation — stop unpromising runs early.
  • Weight sharing / one-shot supernets — train one large "supernet" containing all candidate architectures as sub-graphs; evaluate candidates by inheriting its weights (ENAS, Once-for-All). Fast, but inherited-weight rankings can correlate imperfectly with stand-alone performance.
  • Zero-cost proxies — score untrained networks by properties at initialisation (gradient statistics, activation patterns).

Hardware-aware NAS#

For deployment, accuracy is not the only objective. Multi-objective NAS includes latency, energy or memory on a specific device:

$$ \max_{a}\;\text{Acc}(a)\cdot\left(\frac{\text{Latency}(a)}{\text{target}}\right)^w $$

(MnasNet's reward). MobileNetV3 and EfficientNet's base network were found with such searches. Once-for-All trains one supernet and then extracts specialised sub-networks for many devices without retraining.

EfficientNet's compound scaling#

Tan and Le (2019) combined NAS with a principled scaling rule. After searching for a small baseline (EfficientNet-B0), they scaled depth $d$, width $w$ and resolution $r$ together:

$$ d = \alpha^\phi, \quad w = \beta^\phi, \quad r = \gamma^\phi, \qquad \alpha\beta^2\gamma^2 \approx 2 $$

so each increment of $\phi$ roughly doubles compute. The family B0–B7 achieved excellent accuracy–efficiency trade-offs.

Lessons from the NAS era#

AutoML in practice#

For most practitioners, AutoML means tools that automate model selection and tuning: Auto-sklearn, AutoGluon, H2O AutoML, FLAML, cloud AutoML services and Optuna-based pipelines. AutoGluon, for example, combines many model families with stacking and often provides a very strong tabular baseline with a few lines of code. Use AutoML as a baseline and accelerator, not a substitute for understanding the problem, the data and the evaluation.

#nas#automl#architecture design#efficientnet#search
© 2026 Janin A Apurba (CSE, AUST), Advanced ICT Officer, CNRS-UNHCR. All rights reserved. This lecture may not be republished without the author's written permission.
JA
Written by

Janin A Apurba

B.Sc. in CSE, AUST · Advanced ICT Officer, CNRS-UNHCR. Teaching AI, ML and Deep Learning to the next generation of engineers and researchers.

Keep learning

Related lectures

🔗 Deep Learning

Model Compression: Pruning and Quantisation

Neural networks are highly redundant. We remove unnecessary weights with pruning, represent the rest with fewer bits via quantisation, and discuss the lottery ticket hypothesis and deployment on edge devices.

Advanced⏱ 6 min#131
🔗 Deep Learning

Double Descent and the Generalisation Mystery of Deep Learning

Over-parameterised networks can fit random labels yet generalise on real data, and test error can fall again beyond the interpolation threshold. We explore double descent, benign overfitting and implicit regularisation.

Advanced⏱ 6 min#133
🔗 Deep Learning

Knowledge Distillation: Teaching Small Models with Large Ones

A large "teacher" model's soft predictions contain rich information that can train a much smaller "student". We derive the distillation loss with temperature, discuss dark knowledge, and survey feature and LLM distillation.

Intermediate⏱ 5 min#130