Visualizing High-Dimensional Latent Spaces

Every embedding model maps input into a high-dimensional space where proximity means similarity. Seeing what that space looks like is both useful and tricky.

What a latent space is

A latent space is a learned, compressed representation of data. When a model encodes a sentence, an image, or any other data point, it maps that input to a vector in a continuous space where geometric relationships match semantic ones.

In a well-trained text embedding model, the vector for "king" minus "man" plus "woman" lands near "queen." Sentences about similar topics cluster together. Documents about machine learning sit in a different region than documents about cooking. The model has learned to organize information spatially.

The space is "latent" because it isn't directly observable. It's an internal representation that emerges from training. The model never explicitly learned "put cooking over here and ML over there": that structure showed up because organizing similar things together turned out to help the training objective.

The problem: these spaces are typically 384, 768, or 1536 dimensions. Humans can visualize three. Bridging that gap requires dimensionality reduction, and every technique trades something off.

PCA: the linear baseline

Principal Component Analysis finds the directions of maximum variance in the data and projects onto them. Linear, deterministic, fast.

from sklearn.decomposition import PCA
import numpy as np

# embeddings: shape (n_samples, 768)
pca = PCA(n_components=3)
reduced = pca.fit_transform(embeddings)  # shape (n_samples, 3)

# How much variance is captured?
print(f"Explained variance: {pca.explained_variance_ratio_.sum():.2%}")

PCA's strength is preserving global structure: the overall shape and spread of the data. If two clusters are far apart in the original space, they'll be far apart in the projection.

Its weakness is that it's linear. Real relationships in high-dimensional spaces rarely are. PCA might tell you two clusters exist but blur their boundary. If the first three principal components capture 15% of total variance (common with high-dimensional embeddings), you're looking at a very lossy projection.

When to use PCA: as a first pass for gross structure. As preprocessing before a nonlinear method (PCA to 50 dimensions, then t-SNE to 2). When you need reproducible, deterministic results.

t-SNE: preserving local neighborhoods

t-Distributed Stochastic Neighbor Embedding optimizes a nonlinear mapping that preserves local neighborhoods. Points close in high-dimensional space stay close in the visualization.

from sklearn.manifold import TSNE

tsne = TSNE(
    n_components=2,
    perplexity=30,        # Effective number of local neighbors
    learning_rate='auto',
    n_iter=1000,
    random_state=42
)
reduced = tsne.fit_transform(embeddings)

t-SNE works by constructing probability distributions over pairs of points in both spaces. Neighbors in the original space get high probability, distant points get low. The algorithm minimizes the KL divergence between the two distributions.

Result: tight, well-separated clusters that show local grouping. If your embeddings contain distinct categories, t-SNE finds them and displays them clearly.

The perplexity parameter is the key knob. It roughly maps to the number of nearest neighbors considered for each point:

Low perplexity (5-10): very local. Tight micro-clusters, but global arrangement is meaningless.
Medium perplexity (30-50): the default range. Balances local and moderate-scale structure.
High perplexity (100+): more global structure preserved, but clusters can merge.

Critical caveats:

Distances between clusters in a t-SNE plot are not meaningful. Two clusters that look far apart might not be far apart in the original space.
Cluster sizes are not meaningful. t-SNE expands dense regions and compresses sparse ones.
The algorithm is non-deterministic. Different random seeds produce different layouts. Always set a seed for reproducibility.
It doesn't scale well past ~50,000 points without approximate methods.

UMAP: the modern default

Uniform Manifold Approximation and Projection builds on similar principles to t-SNE but with a stronger mathematical foundation in topological data analysis. In practice, it's faster, scales better, and preserves more global structure.

import umap

reducer = umap.UMAP(
    n_components=2,
    n_neighbors=15,       # Local neighborhood size
    min_dist=0.1,         # Minimum distance between points in embedding
    metric='cosine',      # Match the embedding's native metric
    random_state=42
)
reduced = reducer.fit_transform(embeddings)

n_neighbors controls the balance between local and global structure, similar to perplexity in t-SNE. Low values capture fine-grained local structure, high values preserve more of the global topology.

min_dist controls how tightly points pack. Low values (0.0-0.1) create dense clusters with clear separation. Higher values (0.5-1.0) spread points more evenly, which can be better for understanding continuous gradients in the data.

UMAP's advantages over t-SNE:

Faster. Minutes on datasets that take t-SNE hours.
Preserves global structure better. Relative cluster positions are more meaningful than in t-SNE.
Scales to millions of points with approximate nearest-neighbor methods.
Supports transform on new data. Once fitted, you can project new points without rerunning the full fit.

# UMAP can transform new points; t-SNE can't
new_embeddings = model.encode(["new document text"])
new_reduced = reducer.transform(new_embeddings)

Practical example: visualizing article embeddings

Say you have a collection of blog articles embedded with a sentence transformer. A complete visualization pipeline:

import numpy as np
import umap
import plotly.express as px
from sentence_transformers import SentenceTransformer

# Generate embeddings
model = SentenceTransformer('all-MiniLM-L6-v2')  # 384 dimensions
embeddings = model.encode(articles_text)

# Reduce with UMAP
reducer = umap.UMAP(n_components=2, n_neighbors=10, min_dist=0.1, metric='cosine')
coords = reducer.fit_transform(embeddings)

# Visualize with plotly
fig = px.scatter(
    x=coords[:, 0], y=coords[:, 1],
    color=article_tags,       # Color by category/tag
    hover_name=article_titles,
    title="Article Embedding Space",
    labels={'x': 'UMAP-1', 'y': 'UMAP-2'}
)
fig.update_traces(marker=dict(size=8, opacity=0.7))
fig.show()

In the resulting plot, you'd expect:

Articles about transformers and attention mechanisms clustering together.
RAG articles forming a nearby but distinct group.
Data visualization articles off in their own region.
Some articles bridging clusters (e.g., an article on "visualizing attention" between the transformer and visualization clusters).

Interpreting clusters

What you see in a reduced plot tells you something real, but you need to interpret carefully.

Tight clusters = strong shared signal. If 20 articles about reinforcement learning form a tight blob, the embedding model has captured that they share a common topic with high confidence.

Gradients between clusters = semantic continuum. If the cluster of "neural architecture" articles transitions gradually into "optimization" articles, those topics share vocabulary and concepts that the embedding model represents as proximity.

Outliers = unique content. A point sitting far from any cluster is an article that doesn't share much semantic content with the rest. Either it's off-topic, or it's covering a niche nothing else touches.

Cluster overlap = ambiguous categorization. If "MLOps" and "software engineering" articles overlap, the embedding model considers them semantically similar. That might mean your tagging system draws distinctions the actual content doesn't support.

When visualization misleads

Dimensionality reduction is lossy. Watch for these failure modes:

Phantom clusters. t-SNE can create apparent clusters in uniform data. Project random noise and t-SNE will still show blobs. Always verify clusters with silhouette scores or similar metrics on the original high-dimensional data.

Artificial separation. Two groups that overlap in high-dimensional space can appear separated after projection. The algorithm might find a viewing angle that splits a single cloud into two.

Distance distortion. The Euclidean distance between two points in a UMAP plot isn't proportional to the cosine distance between their original embeddings. Use visualizations for qualitative understanding, not quantitative measurements.

Hyperparameter sensitivity. Different perplexity / n_neighbors values can produce radically different plots from the same data. Always try multiple settings and report which you used.

A good practice: run the same visualization with 3-5 different parameter configurations. Structure that shows up across all of them is likely real. Structure that appears in only one is likely an artifact.

Tools

scikit-learn: PCA and t-SNE implementations, solid for small-medium datasets.
umap-learn: the reference UMAP implementation, pip-installable.
plotly: interactive scatter plots with hover labels, zoom, export.
matplotlib: static plots, good for publications.
TensorBoard Embedding Projector: interactive 3D visualization with built-in PCA, t-SNE, and UMAP, loadable directly from checkpoint data.
Atlas (Nomic): web-based tool for visualizing and exploring large embedding datasets.

Latent spaces are where meaning becomes geometry. Visualization won't show you every relationship encoded in 768 dimensions, but it will show you enough to build intuition, debug models, and explain to other people what your embeddings have learned.