Imagine two floors of an office building. Viewed from the side, the people form two separate groups. Viewed from above, those groups overlap. Nobody moved. The view changed. This is an illustration, but it captures a problem that becomes harder to spot when we flatten hundreds of embedding dimensions into a scatter plot: a convincing picture can hide the relationship we care about.

An embedding map can help you find related articles, discover a strange grouping, or spot a document worth investigating. Before reorganizing your entire library around its colorful islands, though, you need to know which relationships came from the documents and which came from drawing the map.

What we're trying to see

An embedding represents an input, such as a sentence or image, as a vector: a list of numbers. Training shapes those numbers so that geometric relationships help the model perform its task. For a text similarity model, nearby vectors should often correspond to related text. That is learned behavior, with errors and blind spots, rather than a universal definition of meaning.

These internal representations are often called latent spaces. Their dimensions usually don't come with tidy labels such as "cooking" or "machine learning." Relationships emerge across the coordinates. The familiar word-vector example, "king" minus "man" plus "woman" landing near "queen," illustrates the kind of structure a representation can learn; it isn't a promise that arbitrary embedding models support reliable word arithmetic. The separate word embeddings article explores those relationships.

For a running example, consider a collection of articles about transformers, retrieval-augmented generation, and data visualization. We'd like to see which articles the model treats as neighbors, whether our tags make sense, and where an article about visualizing attention belongs. These are questions for an illustrative workflow, not findings from a measured article corpus.

The model used below gives each input 384 coordinates. A screen gives us two. Dimensionality reduction decides which relationships survive that squeeze. PCA, t-SNE, and UMAP make different choices, so agreement between their pictures is useful to investigate, while disagreement can be informative too.

PCA: how much fits in one view?

Principal Component Analysis, or PCA, finds directions along which the data varies most and projects onto a chosen number of them. For our articles, it provides a first look at broad variation without trying to arrange attractive islands. The full SVD solver below avoids a randomized approximation. The PCA documentation explains the solvers and the reported variance ratios.

from sklearn.decomposition import PCA import numpy as np # embeddings: shape (n_samples, 768), with at least 3 samples pca = PCA(n_components=3, svd_solver='full') reduced = pca.fit_transform(embeddings) # shape (n_samples, 3) # How much variance is captured? print(f"Explained variance: {pca.explained_variance_ratio_.sum():.2%}")

The printed percentage matters. If it says 15%, those three axes capture 15% of the input variance. That is a hypothetical reading of the output, not a typical result promised for embeddings. Most variation would lie outside the displayed view, and variance itself is not a measure of semantic importance.

Return to the office building. If the chosen view discards height, it cannot distinguish two people standing directly above one another. PCA has the same limitation: separation along a discarded direction disappears. Two groups far apart in the original space can overlap in the projection. Keeping the greatest overall variance does not guarantee that every distance, category boundary, or useful distinction survives.

PCA is a useful baseline precisely because you can inspect what it retains. It can also reduce a large embedding to, say, 50 components before a nonlinear method, provided the dataset has enough samples and features. That preprocessing introduces its own loss, so record it when comparing results.

t-SNE: who belongs in the neighborhood?

t-SNE focuses on local relationships. It turns pairwise similarities into probabilities, then adjusts a low-dimensional layout to reduce their disagreement, measured by KL divergence. In ordinary language, it works hard to keep likely neighbors together. It doesn't promise a faithful map of the distances between whole neighborhoods.

from sklearn.manifold import TSNE # Requires more than 30 samples for this perplexity. tsne = TSNE( n_components=2, perplexity=30, # Controls the effective neighborhood scale learning_rate='auto', max_iter=1000, metric='cosine', # Use the similarity metric appropriate to the model random_state=42 ) reduced = tsne.fit_transform(embeddings)

For the article collection, a group of transformer articles would give us a promising place to inspect titles and text. The distance from that group to the visualization articles deserves much less confidence.

Perplexity controls the neighborhood scale. Values such as 5 or 10 emphasize smaller neighborhoods; 30 or 50 consider a broader context. Larger values can reveal broader structure, but don't guarantee better preservation. Perplexity must be smaller than the number of samples. Keep the metric consistent across comparisons; the example uses cosine distance. Current scikit-learn uses max_iter for the iteration limit and offers an approximate Barnes-Hut method by default, so there is no universal 50,000-point cutoff. See the t-SNE API.

The striking clusters are also where interpretation gets risky. A compact island doesn't tell you how dense that region was originally. Gaps between islands don't provide a reliable ruler, and a split can appear where the source data has no corresponding groups. Wattenberg, Viégas, and Johnson demonstrate these effects with controlled examples in How to Use t-SNE Effectively. Their examples also show why a single parameter setting is a poor basis for a conclusion.

A fixed seed helps reproduce a layout in the same environment. Trying other seeds and perplexities helps assess whether the relationship you're discussing depends on that particular layout.

UMAP: useful control over the map

UMAP builds a graph of nearby points, then finds a lower-dimensional arrangement that approximates those relationships. Its neighborhood and spacing controls make it useful for exploring both small groups and broader connections.

import umap reducer = umap.UMAP( n_components=2, n_neighbors=15, # Local neighborhood size min_dist=0.1, # Controls how tightly the layout can pack points metric='cosine', # Match the embedding's intended similarity metric random_state=42 ) reduced = reducer.fit_transform(embeddings)

n_neighbors changes the scale of relationships considered. With fewer neighbors, the map concentrates on local detail; with more, it incorporates broader connections. min_dist affects packing in the resulting layout: values around 0.0 to 0.1 allow tight groups, while values around 0.5 to 1.0 produce more spread. Those are settings to explore, not quality grades. The UMAP parameter tutorial shows how each changes the same data.

This gives us a second version of the opening problem. An article collection can look like compact islands at one setting and a more connected arrangement at another. Lowering min_dist has not made the underlying articles more closely related. We have changed how the map draws them.

UMAP can produce false splits and does not fully preserve density, as its own clustering guide explains. Broader neighborhood structure can be useful, but the plotted distance between two groups still isn't a calibrated semantic distance. I would choose between UMAP and t-SNE by the relationships I need to inspect and measured runtime on the actual collection, rather than by a blanket claim that one is always faster or more faithful.

UMAP also supports placing new inputs into an already fitted map:

# UMAP supports transform; scikit-learn's TSNE has no transform method. # model must be the same encoder used for the fitted embeddings. new_embeddings = model.encode(["new document text"]) new_reduced = reducer.transform(new_embeddings)

That is useful for an article library that grows over time. The transform documentation assumes the new inputs follow a distribution compatible with the fitted data. New subject matter may justify refitting and reviewing the map. Also record the seed and library versions: UMAP's reproducibility guide explains the performance tradeoff involved in controlling randomness.

Build a map you can inspect

The following example keeps titles available on hover and uses existing tags as colors. The tags help us compare our catalog with the model's neighborhoods; they are not passed into UMAP to guide the layout.

Supply three aligned lists with more than 10 entries: articles_text, article_titles, and article_tags. Here, articles_text should contain short summaries. The all-MiniLM-L6-v2 model card specifies 384-dimensional outputs and truncation beyond 256 word pieces. Feeding it entire long articles would silently make this a map of their beginnings. For a full-article analysis, choose and document a chunking and aggregation strategy instead.

import numpy as np import umap import plotly.express as px from sentence_transformers import SentenceTransformer # Generate embeddings from short article summaries. model = SentenceTransformer('all-MiniLM-L6-v2') # 384 dimensions embeddings = model.encode(articles_text) # Reduce with UMAP. reducer = umap.UMAP( n_components=2, n_neighbors=10, min_dist=0.1, metric='cosine', random_state=42 ) coords = reducer.fit_transform(embeddings) # Visualize with plotly. fig = px.scatter( x=coords[:, 0], y=coords[:, 1], color=article_tags, # One display category per article hover_name=article_titles, title="Article Embedding Space", labels={'x': 'UMAP-1', 'y': 'UMAP-2'} ) fig.update_traces(marker=dict(size=8, opacity=0.7)) fig.show()

The payoff is being able to investigate individual points. Does the article about visualizing attention sit near transformer explanations, visualization tutorials, or neither? Does its summary explain why? Would changing the summary change the relationship? A hovering title can lead to a more useful question than a perfect-looking cluster.

Plotly provides that interactive view; matplotlib is useful for static figures. The TensorBoard Embedding Projector offers another way to explore embeddings and their metadata, while Nomic Atlas is another option for browsing embedding collections. Whichever interface you use, retain a way to get from a dot back to its source text.

Check the relationship behind the picture

Suppose the attention article looks isolated. Calling it "unique content" immediately would skip several explanations: the projection may have displaced it, its summary may emphasize a different topic, or the input may be malformed. Inspect its nearest neighbors using the original embeddings and the same distance metric. If they are sensible attention articles, the apparent isolation belongs to the picture, not necessarily to the document.

The same reasoning applies to the other shapes:

  • A tight group suggests documents to inspect together. It does not give the embedding model a confidence score.
  • A gradual bridge between groups may reflect shared subject matter. Read the intermediate documents and check their original neighbors before calling it a semantic continuum.
  • Overlapping tags may reveal useful connections, broad labels, or a distinction the model misses. Overlap alone cannot tell you which explanation is right.

For a quantitative neighborhood check, trustworthiness measures how much a projection introduces unexpected neighbors relative to the original space. A high score supports that particular neighborhood comparison; it does not certify semantic correctness. If evaluating proposed clusters with a silhouette score, compute it on the original vectors with an appropriate metric and remember that geometric separation still needs interpretation against the documents.

Try three to five parameter configurations as a practical exploration budget, then vary the seed. Track the relationship you care about, such as whether the attention article keeps the same neighbors, rather than whether an island stays on the left side of the screen. Repeated patterns deserve investigation, but methods can share distortions. Agreement is a reason to check the source space, not permission to skip it.

The two office views disagree because each loses a different direction. Embedding maps add further choices about neighborhoods and spacing. Once you understand those choices, the plots become useful for finding questions worth asking: a surprising neighbor, a questionable tag, a document that needs closer inspection. The conclusion becomes defensible when you can return from the picture to the original distances and the actual text and explain why the relationship makes sense.