latentSource

Visualizing Word Embeddings

Explore the math behind language: see how 'King - Man + Woman = Queen' plays out visually in vector space.

·5 min read
Share
Visualizing Word Embeddings

Visualizing word embeddings

Before 2013, most NLP systems treated words as atomic symbols. In a one-hot representation, "dog" sat as far from "cat" as it did from "quantum." Word embeddings broke that. They learn dense vectors where words with similar meanings end up close in geometric space, and that idea still sits underneath every language model I work with today.

Origins: Word2Vec and GloVe

Tomas Mikolov's Word2Vec (2013) shipped with two training objectives: Skip-gram and CBOW. Skip-gram predicts the surrounding context given a target word. CBOW does the opposite — predict the target from its context. Both produce a weight matrix where each row is a word vector, usually 100 to 300 dimensions.

The trick is simple. Words that show up in similar contexts end up with similar vectors. "Dog" and "cat" both appear next to "pet," "food," and "vet," so they drift together in vector space. No labels, no annotations, just raw text and a sliding window.

GloVe (Pennington et al., 2014) went a different way. Instead of local context windows, it factorizes the global word co-occurrence matrix. The objective: learn vectors whose dot product equals the log of how often two words show up together. GloVe captures global statistics better. Word2Vec is stronger on local syntactic patterns. In practice I've used both and they're roughly interchangeable for downstream tasks.

What embeddings capture

The famous trick is vector arithmetic. "king - man + woman = queen" works because the model has implicitly learned a "gender" direction in the space. Subtract "man" from "king" and you remove the male component, leaving something like "royal." Add "woman" and you put the gender back in with a different value.

Gender isn't the only axis. You also see things like:

  • Paris - France + Italy = Rome (capital city)
  • bigger - big + small = smaller (comparative form)
  • walking - walked + swam = swimming (tense)

These come out of co-occurrence patterns. Nobody supervises any of it. The space ends up encoding relationships as directions, not just locations.

For comparing embeddings, cosine similarity is the standard. Two vectors pointing the same way score near 1, regardless of magnitude. That matches the intuition: what matters is the pattern of associations, not the absolute size of the vector.

Dimensionality reduction for visualization

Embedding vectors live in 100 to 300 dimensions. To plot them you need to project down to 2 or 3 while keeping enough structure to be useful. Three techniques do most of the work.

PCA finds the directions of maximum variance and projects onto the top two or three. Fast, deterministic, linear. It misses nonlinear structure but it's a good first pass and useful when you want to know what the main axes of variation actually are.

t-SNE preserves local neighborhoods. Points that are close in the high-dimensional space stay close in the projection. It's great for revealing clusters but it distorts global distances. The perplexity parameter trades off local versus global structure, and results jitter around between runs.

The projection isn't a one-shot transform — it's an optimization that runs for hundreds of iterations. This is a real t-SNE gradient descent on 90 synthetic 50-dimensional word vectors (five semantic groups). Press play and watch the clusters fall out of a random blob:

Every point is a 50-D vector; t-SNE minimizes the mismatch between neighbor probabilities in 50-D and in 2-D. It starts as a random blob at the center, and the "early exaggeration" phase (first ~60 iterations) blows the groups apart before the layout relaxes into five clean clusters. The colors are the true groups — the algorithm never sees them; it recovers the structure from distances alone. UMAP is newer and what I usually reach for now. It keeps both local and global structure better than t-SNE, runs faster on large datasets, and sits on a more solid mathematical foundation (Riemannian geometry, fuzzy topology). The two parameters that matter are `n_neighbors` (local vs global balance) and `min_dist` (how tightly points pack together). ## Interpreting clusters Once you project embeddings with t-SNE or UMAP, clusters fall out on their own. You'll typically see semantic clusters (animals together, countries together, professions together), syntactic clusters (past-tense verbs in their own region, plurals separate from singulars), and hierarchical structure where "dog" sits closer to "cat" than to "eagle," and "eagle" sits closer to "sparrow." That said, don't read too much into the picture. t-SNE and UMAP are nonlinear projections that can invent visual artifacts. Two clusters that look adjacent in 2D might not be neighbors in the original space at all. Whenever I see something that looks meaningful in a plot, I go back and check it with cosine similarity in the full-dimensional space before I trust it. ## Modern embeddings: from words to sentences Word2Vec and GloVe give every word a single vector regardless of context. "Bank" gets one vector whether it's a financial institution or the side of a river. That's a hard ceiling. BERT (Devlin et al., 2018) produces contextual embeddings. Each occurrence of a word gets a different vector based on its context. "Bank" in "river bank" looks different from "bank" in "bank account." BERT uses the Transformer encoder (12 layers in the base model), and every layer gives you a different embedding. Concatenating or averaging the last four layers tends to work best for most downstream tasks. Sentence-BERT (Reimers and Gurevych, 2019) fine-tunes BERT with a siamese setup so the output is a usable sentence-level embedding. Vanilla BERT isn't great for sentence similarity because the [CLS] token embedding doesn't reliably capture meaning across the whole sentence. Sentence-BERT fixes that and makes it cheap to compare thousands of sentences with cosine similarity, which is what you want for retrieval. ## Practical example: exploring embeddings with Python Here's a complete example that loads pretrained embeddings and plots them: ```python from sklearn.decomposition import PCA from sklearn.manifold import TSNE import matplotlib.pyplot as plt import numpy as np from gensim.models import KeyedVectors # Load pretrained Word2Vec (Google News, 300d) model = KeyedVectors.load_word2vec_format( 'GoogleNews-vectors-negative300.bin', binary=True ) # Select words to visualize categories = { 'animals': ['dog', 'cat', 'horse', 'eagle', 'shark', 'whale'], 'countries': ['france', 'germany', 'japan', 'brazil', 'egypt', 'canada'], 'tech': ['python', 'javascript', 'algorithm', 'database', 'server', 'cloud'], } words = [w for group in categories.values() for w in group] vectors = np.array([model[w] for w in words]) # Reduce to 2D with UMAP (or t-SNE as fallback) try: import umap reducer = umap.UMAP(n_neighbors=5, min_dist=0.3, random_state=42) coords = reducer.fit_transform(vectors) except ImportError: reducer = TSNE(n_components=2, perplexity=5, random_state=42) coords = reducer.fit_transform(vectors) # Plot with color coding colors = {'animals': 'green', 'countries': 'blue', 'tech': 'red'} fig, ax = plt.subplots(figsize=(10, 8)) idx = 0 for category, word_list in categories.items(): n = len(word_list) ax.scatter(coords[idx:idx+n, 0], coords[idx:idx+n, 1], c=colors[category], label=category, s=100) for i, word in enumerate(word_list): ax.annotate(word, (coords[idx+i, 0], coords[idx+i, 1]), fontsize=9, ha='center', va='bottom') idx += n ax.legend() ax.set_title('Word Embeddings Projected to 2D') plt.tight_layout() plt.savefig('embeddings_visualization.png', dpi=150)

Even with this small sample, the categories separate cleanly. Animals cluster, countries cluster, tech terms cluster.

Practical takeaways

Embeddings aren't just a preprocessing step. They're a lens into what the model has actually learned about language. Plotting them surfaces biases (gendered associations with certain professions), domain gaps (a fine-tuned medical model clusters drug names differently from the base model), and data quality problems (if "asdf" sits next to real words, your training data is dirty).

For production systems, the embedding model you pick matters more than the visualization. Sentence-BERT and its successors (E5, GTE, BGE) are what I use for semantic search and retrieval. Word-level embeddings still earn their keep for token-level tasks like named entity recognition, and for poking at the geometry of meaning when you actually want to understand what the model thinks.

Back to articles
Share