Evolution of neural architectures

The history of neural networks isn't a smooth ascent. It's a series of breakthroughs separated by winters, where each new architecture solved a specific limitation of the previous one and was eventually retired when its own limits became the bottleneck. Worth understanding the lineage if you want to know why modern architectures look the way they do, and where the field probably goes next.

The Perceptron (1958)

Frank Rosenblatt's Perceptron was the first trainable neural network. A single linear unit: weighted sum of inputs, threshold, output 0 or 1.

output = 1 if (w1*x1 + w2*x2 + ... + wn*xn + bias) > 0 else 0

The Perceptron can learn linearly separable functions like AND and OR. Minsky and Papert proved in 1969 that it can't learn XOR, which needs a nonlinear decision boundary. That result, plus the overpromising of early AI researchers, kicked off the first AI winter.

What it solved: machines could learn from data. Why it was retired: can't represent nonlinear functions. A single layer is a hard ceiling.

Multi-Layer Perceptrons (1986)

The fix for XOR was obvious in hindsight: stack more layers. A multi-layer perceptron (MLP) with at least one hidden layer and nonlinear activation functions is a universal function approximator. Cybenko's 1989 result says an MLP with enough hidden neurons can approximate any continuous function to arbitrary precision.

The practical breakthrough was backpropagation, popularized by Rumelhart, Hinton, and Williams in 1986. Apply the chain rule layer by layer, gradients flow backward through the network, deep architectures become trainable.

"Trainable" is easy to say and easy to underrate, so here's what it actually looks like. This is a two-layer MLP (24 hidden units, tanh) learning a nonlinear curve from scratch — plain-numpy backprop, no library:

The gray dashes are the true function, the faint dots are noisy training samples, and the cyan line is the network's prediction across the input range. Epoch 0 is nearly flat — a random network outputs mush. Each epoch, backprop nudges every weight down the loss gradient, and the prediction bends to fit. By epoch 500 the mean-squared error has dropped from 0.6 to under 0.02. That's the whole idea: the chain rule turns "fit this arbitrary function" into a solvable optimization. ```python # A simple MLP in modern terms import torch.nn as nn model = nn.Sequential( nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 128), nn.ReLU(), nn.Linear(128, 10) )

What it solved: nonlinear decision boundaries, arbitrary function approximation. Why it was retired: MLPs treat every input as a flat vector. No notion of spatial structure for images, no notion of sequential structure for language. An MLP processing a 256x256 image needs 196,608 input weights per first-layer neuron. Wasteful and prone to overfitting.

Convolutional Neural Networks (1989-2012)

Yann LeCun's CNN work, starting with LeNet for handwritten digit recognition, introduced two key ideas: local connectivity and weight sharing. A convolutional filter slides across the input, applying the same weights at every position. That encodes the prior that spatial patterns — edges, textures, shapes — are translation-invariant.

The architecture stacks convolutional layers (feature extraction) with pooling layers (downsampling) and fully connected layers (classification). Each layer learns more abstract features as you go deeper: edges in early layers, textures in middle layers, object parts in deep layers.

The CNN revolution really started in 2012 when AlexNet (Krizhevsky, Sutskever, Hinton) won ImageNet by a huge margin. AlexNet wasn't architecturally novel — three things converged: large-scale labeled data (ImageNet), GPU training, and ReLU activations (which mitigate vanishing gradients better than sigmoid).

The architectures that followed:

VGGNet (2014): showed depth matters. Used only 3x3 convolutions stacked deeply.
GoogLeNet/Inception (2014): parallel pathways with different filter sizes.
ResNet (2015): solved the degradation problem in very deep networks with skip connections.

ResNet deserves a closer look. Intuitively, a 56-layer network should perform at least as well as a 20-layer one, since it could just learn identity mappings for the extra layers. In practice, deeper plain networks performed worse because of optimization issues. Residual connections let the network learn F(x) + x instead of F(x), much easier to optimize because the identity path keeps gradients flowing.

class ResidualBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, 3, padding=1)
        self.bn1 = nn.BatchNorm2d(channels)
        self.conv2 = nn.Conv2d(channels, channels, 3, padding=1)
        self.bn2 = nn.BatchNorm2d(channels)

    def forward(self, x):
        residual = x
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return F.relu(out + residual)  # The skip connection

What CNNs solved: efficient processing of spatial data with translation invariance. Why they weren't enough: CNNs are great for fixed-size grid data, awkward for sequences of variable length. Limited receptive fields, and while you can stack layers to widen them, capturing long-range dependencies in text requires impractically deep networks.

RNNs and LSTMs (1990s-2017)

Recurrent Neural Networks process sequences by carrying a hidden state that gets updated at each timestep:

h_t = tanh(W_hh * h_{t-1} + W_xh * x_t + b)

That gives RNNs a natural mechanism for variable-length sequences. They powered early wins in machine translation, speech recognition, and language modeling.

The problem is the vanishing gradient. Backpropagating through many timesteps shrinks gradients exponentially, so long-range dependencies become impossible to learn. The sentence "The cat, which sat on the mat next to the dog that barked at the mailman, was sleeping" requires connecting "cat" to "was" across a lot of intervening tokens.

The Long Short-Term Memory (LSTM) network, from Hochreiter and Schmidhuber in 1997, fixed this with a gated cell. Three gates (input, forget, output) control information flow, and a cell state acts as a highway for gradients across many timesteps without degradation.

The GRU (Gated Recurrent Unit) by Cho et al. in 2014 simplified the LSTM to two gates (reset and update) with comparable performance and fewer parameters.

For years, encoder-decoder LSTMs with attention (Bahdanau et al., 2014) were state of the art in machine translation. The attention mechanism there was the direct precursor to the Transformer: instead of forcing all source information through a fixed-size hidden state, the decoder could attend to different parts of the input at each generation step.

What RNNs/LSTMs solved: variable-length sequence processing with memory. Why they were retired: sequential processing prevents parallelization. Even with gating, very long-range dependencies stay hard. Training is slow compared to what parallelizable architectures can do on modern hardware.

The Transformer (2017)

Vaswani et al.'s "Attention Is All You Need" dropped recurrence entirely. Self-attention lets every token attend to every other token directly, in a single operation that's fully parallelizable on GPUs.

The insight was that attention, previously bolted onto RNNs, was sufficient on its own. With positional encoding (for sequence order), feed-forward layers, residual connections, and layer normalization, the Transformer matched or beat LSTM performance on translation while training far faster.

The original paper reported 3.5 days on 8 GPUs for their base model. An equivalent LSTM model would have taken weeks.

What the Transformer solved: parallelizable sequence modeling with direct long-range dependencies. Its limit: quadratic complexity in sequence length for self-attention.

The modern era: scaling Transformers

After 2017, progress came less from architectural novelty and more from scale.

BERT (2018) showed that pretraining a Transformer encoder on massive unlabeled text (masked language modeling) produced representations that transferred powerfully to downstream tasks. Fine-tuning BERT beat task-specific architectures across the board.

GPT-2 (2019) and GPT-3 (2020) showed that decoder-only Transformers, trained autoregressively on enough data, develop emergent capabilities. GPT-3's 175 billion parameters could do tasks it was never explicitly trained on, just from in-context examples in the prompt.

Scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) formalized the relationship: model performance improves as a power-law function of compute, data, and parameters. The Chinchilla paper showed most models were undertrained relative to their size, recommending roughly 20 tokens per parameter.

Recent refinements to the Transformer:

RoPE (Rotary Position Embedding): better position encoding that generalizes to unseen sequence lengths.
SwiGLU activation: outperforms ReLU in the feed-forward layers.
Grouped-query attention (GQA): reduces KV cache memory during inference.
FlashAttention: hardware-aware algorithm that cuts memory and compute for attention.
Mixture of Experts (MoE): sparsely activated parameters that scale capacity without proportionally scaling compute.

The pattern

Looking at the timeline, the pattern is clear.

Each generation's architecture encoded a stronger inductive bias: MLPs assumed nothing about input structure, CNNs assumed spatial locality, RNNs assumed sequential order. The Transformer won by assuming less: self-attention makes minimal structural assumptions and lets the model learn what to attend to from data.

The trend is toward architectures that are simpler, more general, more scalable. The computational cost of generality is paid by scale: you need more data and more compute to learn structure that earlier architectures baked in as assumptions. Given enough scale, the learned structure is more flexible and powerful than any hand-designed prior.

Whether the trend continues, and whether Transformers eventually give way to state-space models (Mamba), hybrid architectures, or something we haven't built yet, is open. But the history suggests the next dominant architecture will be simpler than the Transformer, not more complex.