Imagine two portraits of the same person, one facing left and one facing right. Encode each portrait into a short list of numbers, move halfway between those lists, and decode the result. Ideally, you get the person facing forward. You might also get a blurry face with an unfortunate number of eyes.

This is an illustrative example, not a result from a trained model in this article. It captures the question behind latent spaces: how do we learn a representation in which moving between examples produces something useful? Reconstructing the two original portraits is only the beginning. The interesting test is what happens between them.

A compact description with a way back

A 256x256 RGB image contains 196,608 color values. Change those values independently and you're unlikely to produce a convincing face. Real portraits have structure: eyes sit in relation to a nose, lighting affects whole regions, and turning a head changes many pixels together.

An encoder learns to turn those pixel values into a vector, a list of numbers called a latent representation. A decoder learns to turn that vector back into an image. "Latent" means hidden or unobserved: the representation is inferred during training, rather than supplied as a list of labels such as "head angle" and "lighting."

Think of the vector as a compact description written for the decoder. It might have 128 or 256 numbers instead of hundreds of thousands of pixel values. Those numbers needn't be readable instructions. One number won't necessarily mean "turn the head five degrees." The encoder and decoder learn their shared representation through the task they're trained to perform.

In notation, the encoder maps an input x from a space X to a vector z in a space Z. The decoder maps z to a reconstruction x'. Here we're interested in compressed representations for generation; latent representations more broadly needn't always have fewer dimensions than their inputs.

The reason compression can work is often described by the manifold hypothesis. Although images have many dimensions, plausible images occupy a structured subset of that space. A 64x64 RGB face has 12,288 values, yet a change in pose can explain coordinated changes across many of them. A learned latent space tries to capture useful structure of this kind. How well it succeeds depends on the data and the training objective.

Teaching the decoder to reconstruct

An autoencoder trains the encoder and decoder together. Give it an image, reconstruct it, measure the difference, and adjust the networks to reduce that difference. Mean squared error is one option for continuous values; binary cross-entropy fits binary observations or a suitable probabilistic interpretation of the pixels.

This small PyTorch architecture compresses 784 inputs, enough for a flattened 28x28 grayscale image, into 32 numbers before reconstructing them:

import torch import torch.nn as nn class Autoencoder(nn.Module): def __init__(self, input_dim=784, latent_dim=32): super().__init__() self.encoder = nn.Sequential( nn.Linear(input_dim, 256), nn.ReLU(), nn.Linear(256, latent_dim) ) self.decoder = nn.Sequential( nn.Linear(latent_dim, 256), nn.ReLU(), nn.Linear(256, input_dim), nn.Sigmoid() ) def forward(self, x): z = self.encoder(x) return self.decoder(z)

The final sigmoid keeps reconstructed values between zero and one, so the input needs compatible scaling. The class defines the architecture; it doesn't load images, train the model, or establish reconstruction quality.

The 32-number bottleneck forces the network to reuse patterns instead of passing all 784 values straight through. Whether it preserves a digit's shape or loses an important stroke depends on what training rewards and what capacity the network has.

Return to the portraits. An autoencoder can reconstruct each endpoint well while behaving poorly between their encoded vectors. Training on individual reconstructions doesn't directly require the midpoint to be a believable portrait. The decoder can produce an image there, but producing an image and producing a useful image are different tests.

Making the space easier to sample

A variational autoencoder, or VAE, adds a constraint on how the representation is organized. In the common Gaussian formulation, the encoder outputs a mean and log-variance for each input. Training samples a vector from that distribution and decodes it. A regularizer encourages the encoded distributions to stay close to a chosen prior, commonly a standard normal distribution.

The following expression is the evidence lower bound, which training maximizes. Implementations commonly minimize its negative:

L = E[log p(x|z)] - KL(q(z|x) || p(z))

The first term rewards reconstructing the input from its sampled code. The second penalizes differences between the encoder's distribution q(z|x) and the prior p(z). Together, they balance reconstruction against a space from which we can sample without first encoding an input. This is the approach introduced by Kingma and Welling.

Regularization encourages useful neighborhoods; it doesn't guarantee that every coordinate decodes to something plausible. A sample can still be poor, and a path between valid examples can cross a region the decoder handles badly. In the portrait example, the forward-facing midpoint remains something to inspect.

Walking between images

For two encoded portraits z_left and z_right, linear interpolation uses (1 - t) * z_left + t * z_right. At t=0 we have the first code; at t=1 we have the second; at t=0.5 we have their midpoint. Decode each intermediate vector and examine the resulting images.

A useful result would preserve the person's identity while the pose changes. A failure could change the face, distort its features, or merely blend the endpoints. Even smooth changes aren't enough: a face can become steadily less recognizable without any abrupt jump.

Attribute arithmetic asks more of the representation. Suppose we encode a man with glasses, a man without glasses, and a woman without glasses. The intended operation is to use the difference between the first two codes as a "glasses" direction:

# Illustrative latent arithmetic; requires a trained image model and inputs z_glasses = encoder(man_with_glasses) z_no_glasses = encoder(man_without_glasses) z_woman = encoder(woman_without_glasses) z_result = z_woman + (z_glasses - z_no_glasses) generated = decoder(z_result) # inspect whether glasses transferred cleanly

The subtraction also captures any differences in pose, lighting, identity, or background between those first two images. Carefully matched examples help isolate the intended change, but a single successful edit doesn't establish a universal glasses direction.

A representation is called disentangled when its factors can vary independently in a useful way. We would like a control for glasses that leaves the face alone. Ordinary reconstruction training doesn't promise that arrangement, and even specialized methods need assumptions about the model and data. Locatello and colleagues show why unsupervised disentanglement cannot be assumed. The arithmetic is an experiment to evaluate, not a property to grant every latent vector.

Why the representation is worth learning

Image generation makes the benefit concrete. Latent diffusion models perform their repeated denoising operations in the compressed representation of an autoencoder, then decode the result into pixels. This reduces the computational burden while retaining useful image detail, as demonstrated in the latent diffusion paper. The decoder supplies the route back to an image; the diffusion model learns how to generate codes worth decoding.

The same encode, move, decode pattern appears in molecular design. Gómez-Bombarelli and colleagues trained a continuous representation of molecules alongside a property predictor. They could search that representation for desired properties and decode candidates. The promise is a tractable way to propose molecules. A candidate still needs validation; a favorable coordinate isn't a laboratory result.

Other applications use the compact representation without generating a new image or molecule. An autoencoder trained on normal observations can supply reconstruction errors for anomaly detection. Those errors need testing against both ordinary variation and known anomalies: unfamiliar inputs aren't guaranteed to reconstruct badly. A threshold is a decision rule that needs evidence.

In recommendation systems, matrix factorization learns latent vectors for users and items, with their dot product estimating affinity. That illustrates how broadly useful learned representations are, although the prediction task differs from reconstructing an input. In each case, the objective determines which relationships the representation has a reason to preserve.

Test the trip back

I would evaluate a generative latent space by starting with decoded outputs on held-out examples. Can the model reconstruct inputs it didn't train on? Which details disappear? For a digit, a lost stroke can change the answer even if the average pixel error looks small.

Next, interpolate between several pairs and decode ten intermediate steps for each. Check whether the intended change happens while other relevant properties remain stable. Then try a traversal: hold every coordinate fixed except one and vary that coordinate. This reveals what it affects. A coordinate that changes several attributes may still be useful, but it isn't an independent control for any one of them.

Projection methods such as t-SNE and UMAP can help inspect neighborhoods. A digit model doesn't have to produce ten clean islands in a two-dimensional plot to be useful. The projection changes the geometry, and reconstruction training has a different objective from separating labels. Use the plot to choose examples to investigate, then check the original representation and decoded outputs.

For a VAE, also check whether the decoder uses the code. A KL term near zero can signal posterior collapse, where the encoder's distribution approaches the prior and carries little information about the input. Compare this signal with reconstruction behavior and the effect of changing z. KL annealing and limiting decoder capacity are possible interventions, not guaranteed repairs; Bowman and colleagues discuss this training difficulty in sentence VAEs.

The portrait midpoint gives us a practical standard. A useful generative representation must preserve enough information to reconstruct examples and organize that information so chosen movements produce sensible changes. Reconstruction, interpolation, and traversal test different parts of that requirement. Successful examples establish what those paths can do; broader use calls for testing more inputs and more paths. Once you make that distinction, latent space becomes something you can investigate by changing a code and seeing exactly what comes back.