How neural networks evolved

Give a network two switches and ask it to turn a light on when exactly one switch is on. Four possible inputs, one simple rule. A single perceptron cannot learn it, however patiently you train it.

That tiny failure helps explain why neural networks have so many shapes. More training cannot fix every limitation. Sometimes the network needs another layer, a way to recognize the same pattern in different places, or a shorter route between related words. Following those changes makes modern architectures easier to understand, including why the older ones still have useful jobs.

When one boundary is not enough

Frank Rosenblatt's 1958 perceptron paper helped establish learning from examples. The familiar single-unit version computes a weighted sum, then applies a threshold:

output = 1 if (w1*x1 + w2*x2 + ... + wn*xn + bias) > 0 else 0

A single perceptron can learn AND and OR. The switches example is XOR: (0, 1) and (1, 0) should produce 1, while (0, 0) and (1, 1) should produce 0. Plot those four points and the matching answers sit at opposite corners. No straight line separates the two groups. The limit is the shape of the decision boundary, not a shortage of examples.

A hidden layer with nonlinear activations lets a multi-layer perceptron (MLP) combine several boundaries. Stacking linear layers alone would still give a linear function. The nonlinearity is what lets the network bend.

Cybenko's 1989 universal approximation result established that a sufficiently wide hidden layer with suitable activations can approximate any continuous function on a compact domain. It is a statement about what the network can represent, not a promise that training will find it. Backpropagation, popularized by Rumelhart, Hinton, and Williams in 1986, supplies the practical machinery: use the chain rule to work backward from an error and calculate how each weight contributed. Cybenko's paper, the backpropagation paper.

"Trainable" is easy to say and easy to underrate, so here's what it actually looks like. This is a two-layer MLP (24 hidden units, tanh) learning a nonlinear curve from scratch (backpropagation written in NumPy, without a neural-network library):

The gray dashes are the true function, the faint dots are noisy training samples, and the cyan line is the network's prediction across the input range. This is a synthetic teaching example with 60 noisy samples, not a generalization benchmark. The initial prediction misses the curve badly. Each epoch, backprop calculates a direction for updating the weights, and gradient descent bends the prediction toward the samples. The saved trace reports mean-squared training error of 0.200 at epoch 3 and 0.019 at epoch 500. Learning the curve within this interval does not establish accuracy outside it.

# A simple MLP in modern terms
import torch.nn as nn

model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Linear(256, 128),
    nn.ReLU(),
    nn.Linear(128, 10)
)

The PyTorch example is a separate classifier: 784 inputs become ten output scores through two hidden layers. The same basic building block still appears inside Transformers.

Its flexibility has a cost. A fully connected neuron receiving a 256x256 RGB image needs 196,608 input weights, excluding its bias. It must learn relationships between nearby pixels without any built-in preference for nearby pixels. For images, that is an expensive place to start.

Reusing a pattern across an image

Convolutional neural networks (CNNs), including Yann LeCun's work on handwritten digits, build spatial structure into the network. A small filter slides across an image, applying the same weights at each position. Once it learns an edge detector, it can use that detector on both sides of the picture. LeCun and colleagues demonstrated backpropagation for handwritten zip-code recognition in 1989.

Weight sharing makes convolution translation equivariant: move the input pattern and the detected feature moves with it, subject to boundaries and sampling. Pooling can add tolerance to small shifts. Neither property guarantees that an entire CNN ignores every translation.

The architecture stacks convolutional layers (feature extraction) with pooling layers (downsampling) and fully connected layers (classification). Each layer learns more abstract features as you go deeper: edges in early layers, textures in middle layers, object parts in deep layers.

In 2012, AlexNet (Krizhevsky, Sutskever, Hinton) demonstrated how far CNNs could go with large labeled datasets, GPU training and non-saturating ReLU activations. Its training recipe also used dropout to reduce overfitting. Architecture, hardware and training methods worked together. AlexNet paper.

The architectures that followed:

VGGNet (2014): explored depth using stacks of small 3x3 convolutional filters.
GoogLeNet/Inception (2014): parallel pathways with different filter sizes.
ResNet (2015): solved the degradation problem in very deep networks with skip connections.

ResNet deserves a closer look. Intuitively, a 56-layer network should perform at least as well as a 20-layer one, since it could just learn identity mappings for the extra layers. In practice, deeper plain networks performed worse because of optimization issues. Residual connections let the network learn F(x) + x instead of F(x), much easier to optimize because the identity path keeps gradients flowing.

import torch.nn as nn
import torch.nn.functional as F

class ResidualBlock(nn.Module):
    def __init__(self, channels):
        super().__init__()
        self.conv1 = nn.Conv2d(channels, channels, 3, padding=1)
        self.bn1 = nn.BatchNorm2d(channels)
        self.conv2 = nn.Conv2d(channels, channels, 3, padding=1)
        self.bn2 = nn.BatchNorm2d(channels)

    def forward(self, x):
        residual = x
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return F.relu(out + residual)  # The skip connection

The addition in out + residual is the important part: the block can preserve its input and learn a correction. This example keeps the channel count and image dimensions unchanged, so the tensors fit together. Blocks that change dimensions need a corresponding projection on the shortcut.

CNNs make local patterns economical to learn. Connecting distant positions takes additional layers, larger filters or dilation. Language raises another problem: a sentence has an order and a variable length. Recurrent networks approached that problem along a parallel branch of the history.

Remembering earlier words

Recurrent Neural Networks process sequences by carrying a hidden state that gets updated at each timestep:

h_t = tanh(W_hh * h_{t-1} + W_xh * x_t + b)

That gives RNNs a natural mechanism for variable-length sequences. They powered early wins in machine translation, speech recognition, and language modeling.

The problem is the vanishing gradient. Backpropagating through many timesteps can shrink gradients until early inputs receive almost no useful learning signal. Gradients can also grow uncontrollably. The sentence "The cat, which sat on the mat next to the dog that barked at the mailman, was sleeping" requires connecting "cat" to "was" across a lot of intervening tokens.

The Long Short-Term Memory (LSTM) network, from Hochreiter and Schmidhuber in 1997, introduced a cell state designed to preserve information and learning signals over longer intervals. The familiar input, forget and output gates control what enters, stays and leaves. The forget gate was a later addition, described by Gers, Schmidhuber and Cummins in Learning to Forget in 2000. These mechanisms make long dependencies easier to learn, without guaranteeing perfect memory.

The GRU (Gated Recurrent Unit) by Cho et al. in 2014 simplified the LSTM to two gates (reset and update) with fewer parameters at the same hidden width; which performs better depends on the task.

Attention added another way to handle distance. In Bahdanau and colleagues' 2014 work, a recurrent decoder could consult different parts of the source sentence while generating a translation. It no longer had to recover the entire sentence from one fixed-size summary. Think of translating with the original page open instead of relying on a single note about it.

Recurrence still imposes an ordering cost: the next hidden state depends on the previous one. Training can parallelize across examples, but that chain limits parallel work across positions within each sequence.

Giving words a direct connection

The 2017 Transformer removed recurrence from its translation architecture. Self-attention connects positions directly; positional information supplies order, while feed-forward layers, residual connections and normalization help process the result. During training, many positions can be computed in parallel. A causal decoder still masks future tokens, and ordinary text generation still produces tokens sequentially.

In the original experiments, the base model trained for about 12 hours on eight NVIDIA P100 GPUs; the larger model trained for 3.5 days on the same hardware. The paper reported better translation scores at lower estimated training cost than the compared systems. Those are specific experimental results, not a universal speed ratio against every LSTM. Attention Is All You Need.

Direct connections have a bill attached. Dense self-attention compares pairs of positions, so its arithmetic grows quadratically with sequence length. Double the sequence and there are roughly four times as many position pairs. The architecture made distant relationships easier to reach while creating a new scaling problem.

The modern era: scaling Transformers

The Transformer also made it practical to reuse one broad architecture across many language tasks, with pretraining supplying much of the work previously repeated for each task.

BERT (2018) showed that pretraining a Transformer encoder on massive unlabeled text (masked language modeling) produced representations that transferred powerfully to downstream tasks. The BERT paper reported state-of-the-art results on eleven language tasks after fine-tuning.

GPT-2 (2019) and GPT-3 (2020) showed that decoder-only Transformers, trained autoregressively on enough data, can perform a range of tasks through prompting. The GPT-3 paper evaluated its 175-billion-parameter model with tasks specified in the prompt and no task-specific gradient updates. Its training data could still contain related examples.

Scaling-law studies connected language-model loss to compute, data and parameter count. The Chinchilla paper found that the large models it examined were undertrained relative to their size under a fixed training-compute budget. Its 70-billion-parameter model used 1.4 trillion training tokens, roughly 20 per parameter. That balance addressed training efficiency under the study's conditions; it is not a universal prescription for a model that will serve billions of requests.

Recent refinements to the Transformer:

RoPE encodes position through rotations that make relative distance available to attention. Position encoding alone does not guarantee reliable extrapolation to arbitrary lengths.
SwiGLU changes the feed-forward transformation; the paper found improvements over standard activation choices in its experiments.
Grouped-query attention shares key/value heads, reducing the cached information needed during generation.
FlashAttention reduces transfers between GPU memory levels and avoids materializing the full attention matrix. It speeds up exact attention without removing its quadratic arithmetic.
Mixture of Experts (MoE) activates only a subset of expert networks for each token, increasing total parameter capacity without running every parameter on every token.

These changes address different costs. A faster attention kernel, a smaller cache and a different feed-forward layer can all improve the same model without replacing the Transformer.

What this history helps us choose

The switches at the start failed because one straight boundary could never express the rule. Hidden layers solved that representational problem. Convolutions made recurring visual patterns cheaper to learn. Recurrent state carried information through a sequence, and attention gave distant positions a direct route to one another. Residual connections helped make depth trainable across several of these families.

I find that a more useful way to judge an architecture than its place on a timeline. First ask what is failing: representation, learning, access to distant information, or the cost of running the model. Then ask what structure the data already gives you. Spatial locality can be a useful assumption for images; a compact recurrent state can suit a stream. A general attention mechanism offers flexibility, but its data and compute requirements still have to fit the job.

State-space models such as Mamba and hybrid designs continue that search. History does not tell us that the next successful architecture must be simpler. It tells us to look for a concrete bottleneck, an explanation of how the design removes it, and evidence that the resulting tradeoff is worth paying for.