Give each word its own numbered slot and "dog" is just as different from "cat" as it is from "quantum." That's what a one-hot representation does: it records identity without representing similarity. Useful for a lookup table. Less useful when you want a computer to recognize that two words belong in similar conversations.

Word embeddings give those relationships a geometry. In Gensim's documented example using the 100-dimensional glove-wiki-gigaword-100 model, the nearest neighbor of "cat" is "dog," with cosine similarity 0.8798. No one added a rule saying that both are pets. Training produced vectors that make this comparison possible. The interesting question is how those numbers acquire that structure, and how much of it survives when we draw a picture. Gensim's word-vector examples

Learning a word from its company

An embedding is a list of numbers assigned to a word. Think of it as a compact profile of the contexts where that word belongs. "Dog" and "cat" can both appear around words such as "pet," "food," and "vet." Their profiles have something in common, even though their spelling doesn't.

The numbers aren't a set of labeled columns called "animal," "furry," and "lives indoors." Training adjusts them together to do a prediction task. A useful relationship can be spread across many coordinates, which is why looking at a single number usually tells you very little.

Word2Vec, introduced by Mikolov and colleagues in 2013, made this approach practical through two training objectives. Skip-gram predicts surrounding words from a target word; CBOW predicts the target from its surrounding words. As the model learns to make those predictions, words used in similar contexts can acquire similar vectors. The text supplies the training examples without someone annotating every word's meaning. The Word2Vec paper

GloVe, introduced by Pennington and colleagues in 2014, starts with aggregated word co-occurrence counts. Its training objective fits relationships between vector dot products and the logarithms of those counts, with weighting and bias terms. Both methods learn from context; they organize that learning differently. The resulting quality depends on the training text and the task you care about. The GloVe project

Cosine similarity compares the direction of two vectors. For nonzero vectors, it ranges from -1 to 1, with 1 meaning they point in the same direction. It ignores their lengths, so a larger vector doesn't automatically count as more similar. The cat/dog score above describes agreement between learned profiles. It isn't an 87.98% probability that the words mean the same thing.

Why the king and queen example works

The familiar expression king - man + woman ≈ queen makes embeddings sound like a calculator for meaning. There is a real result underneath it. With the same GloVe model, Gensim's documented most_similar(positive=['woman', 'king'], negative=['man']) query returns "queen" first, with a cosine score of 0.7699. These are published documentation results, not measurements from the animation below. The documented query and output

Imagine moving from "man" toward "woman," then trying a similar move from "king." If the two relationships have similar directions in the learned space, the second move lands near "queen." That is the geometric idea behind the analogy. Gensim's query combines normalized word vectors and ranks candidate words by cosine similarity; it doesn't calculate an exact equality between dictionary definitions.

The shortcut "remove the male component and leave royalty" goes too far. A word vector contains overlapping associations learned from text. There is no guarantee that subtracting one word neatly removes one concept, or that a successful analogy reveals a universal gender axis. The result depends on the model, its vocabulary, and how the query is scored.

Other relationships can be tested with the same idea: Paris - France + Italy ≈ Rome for capitals, bigger - big + small ≈ smaller for comparative forms, or walking - walked + swam ≈ swimming for verb forms. Treat these as analogy queries to evaluate, not promised outputs for every embedding model. Their appeal is that a relationship can appear as a reusable direction, beyond two words merely occupying nearby positions.

Getting hundreds of coordinates onto a page

The GloVe example has 100 dimensions; the Google News Word2Vec file used in the Python example below has 300. A screen gives us two. Any projection has to give something up.

PCA keeps directions that account for the most variation in the selected vectors. It is a useful linear starting point, but the directions of greatest variation needn't be the distinctions you care about. t-SNE instead builds a map around neighborhood relationships, adjusting the two-dimensional positions so nearby points in the original space tend to remain neighbors. Its perplexity setting changes the neighborhood scale. The optimization and initialization also affect the picture. The t-SNE paper

UMAP provides another way to construct a neighborhood-based map. Its n_neighbors parameter controls the scale of relationships considered; min_dist affects how tightly the displayed points can pack together. Those choices change what you see. Neither method promises to preserve every original distance, and a gap between groups is not a calibrated measure of how different their meanings are. UMAP's parameter guide

A synthetic demonstration of t-SNE

The animation shows the projection process using 90 synthetic vectors with 50 coordinates each, arranged in five groups of 18. The generator places points around five separated centers, then runs 300 iterations of t-SNE optimization. It uses perplexity 12 and an early exaggeration phase during the first 59 iterations. The separation is built into the input; these vectors were not learned from sentences.

Press play to watch the initially scattered positions reorganize. The algorithm receives distances between vectors, while the colors let us follow the groups the generator created. Labels inside the animation call these "words" or "semantic groups"; here they stand for synthetic points and their assigned groups.

Synthetic example: 90 points in five groups, each with 50 dimensions. The recorded optimization starts with a small random 2-D layout and adjusts neighbor relationships over 300 iterations. Early exaggeration strengthens those relationships initially; the later steps refine the layout. Colors identify the constructed groups and are not inputs to the optimization. This demonstrates projection behavior, not semantics learned from language.

The animation helps explain why a map takes shape gradually. For learned word vectors, the separate question is whether useful relationships exist before projection. That is what the GloVe neighbor and analogy scores establish for those particular queries.

Plotting learned vectors with Python

This example loads the pretrained Google News Word2Vec binary, checks the original-space similarity of "dog" with "cat" and "quantum," then plots selected words. It needs that file locally, plus Gensim, NumPy, Matplotlib, and scikit-learn. Installing umap-learn enables the UMAP path; otherwise it uses t-SNE. Gensim documents the binary loading and similarity methods in its KeyedVectors API.

Vocabulary lookup is case-sensitive. The code reports and skips missing words so a missing lowercase spelling doesn't stop the example. Both projection paths use cosine distance to match the comparison we're discussing.

from sklearn.manifold import TSNE import matplotlib.pyplot as plt import numpy as np from gensim.models import KeyedVectors # Load pretrained Word2Vec (Google News, 300d) model = KeyedVectors.load_word2vec_format( 'GoogleNews-vectors-negative300.bin', binary=True ) # Select words to visualize categories = { 'animals': ['dog', 'cat', 'horse', 'eagle', 'shark', 'whale'], 'countries': ['france', 'germany', 'japan', 'brazil', 'egypt', 'canada'], 'tech': ['python', 'javascript', 'algorithm', 'database', 'server', 'cloud'], } # Vocabulary is case-sensitive; skip missing tokens and report them. missing = [w for group in categories.values() for w in group if w not in model] print("Missing words:", missing) categories = { name: [w for w in group if w in model] for name, group in categories.items() } words = [w for group in categories.values() for w in group] if len(words) <= 5: raise ValueError("Need more than five available words for this projection") # Check the original vectors before looking at the plot. for left, right in [('dog', 'cat'), ('dog', 'quantum')]: if left in model and right in model: print(left, right, f"{model.similarity(left, right):.4f}") vectors = np.array([model[w] for w in words]) # Reduce to 2D with UMAP (or t-SNE as fallback) try: import umap reducer = umap.UMAP(n_neighbors=5, min_dist=0.3, metric="cosine", random_state=42) coords = reducer.fit_transform(vectors) except ImportError: reducer = TSNE(n_components=2, perplexity=5, metric="cosine", init="random", random_state=42) coords = reducer.fit_transform(vectors) # Plot with color coding colors = {'animals': 'green', 'countries': 'blue', 'tech': 'red'} fig, ax = plt.subplots(figsize=(10, 8)) idx = 0 for category, word_list in categories.items(): n = len(word_list) ax.scatter(coords[idx:idx+n, 0], coords[idx:idx+n, 1], c=colors[category], label=category, s=100) for i, word in enumerate(word_list): ax.annotate(word, (coords[idx+i, 0], coords[idx+i, 1]), fontsize=9, ha='center', va='bottom') idx += n ax.legend() ax.set_title('Word Embeddings Projected to 2D') plt.tight_layout() plt.savefig('embeddings_visualization.png', dpi=150)

The colors mark categories we selected before plotting. They aren't categories discovered by the model. With only these words on screen, a neat grouping is a useful observation about this sample, not evidence that every animal, country, or technical term would separate cleanly. "Python" also gives us a good reason to inspect vocabulary and context: an animal and a programming language need not have the same representation, especially when capitalization differs.

Read the printed similarities before interpreting the picture. If a surprising neighbor appears on the map, check the original vectors and nearby vocabulary. A plot can suggest a question about a profession's associations or an unfamiliar token, but it can't by itself establish social bias, a domain improvement, or dirty training data. Those conclusions need comparisons that test the proposed explanation.

When one vector per word stops being enough

Static word embeddings have a familiar limitation: "bank" gets one vector whether the sentence concerns a river or an account. Training can blend both uses into the same profile. That makes the representation convenient to store and compare, but ambiguous words carry their contexts together.

BERT introduced representations conditioned on the surrounding text, so the representation of "bank" can differ between those sentences. BERT-base has 12 Transformer encoder layers; which layer or combination to use depends on the task. There is no general rule that averaging the last four layers is best. The BERT paper

Sentence-BERT then trained representations suitable for comparing whole sentences with cosine similarity. This matters for retrieval: each sentence can be encoded and stored, and a query can be compared with those stored vectors without running every sentence pair through BERT together. Models such as E5, GTE, and BGE belong in that broader search for useful text representations, but a word analogy alone won't tell you which model retrieves your documents well. The Sentence-BERT paper

What the picture earns

The cat/dog neighbor and the king/queen analogy show something useful: training on language can produce geometric relationships that we can query directly. Understanding the training objective explains how that happens without imagining that each coordinate contains a neatly labeled piece of meaning.

I would use the plot to find relationships worth checking, then judge those relationships in the original vectors and in the task they need to support. A pair of words can be close on a page because of the projection's compromises. A high cosine score tells us something more specific about the model's representation, while a retrieval test tells us whether that similarity helps a reader find the right document. Keeping those questions separate makes the visualization far more useful.