When meaning misses

One missing letter was enough to make a World Cup project disappear.

I searched my site for "something about sports." Mundial, a project about World Cup predictions, appeared behind Twitter Matcher, whose retrieved passage discussed tweet capture and NLP cleaning. Then I typed "someting about sports." Neither project appeared.

A reader looking for a sports project would have a reasonable reaction: this search is broken. An engineer looking at the service could report that the model returned valid vectors, every document was indexed, and PostgreSQL computed the distances correctly. Both observations were true. That gap is where retrieval gets interesting.

The failure exposes two decisions that are easy to hide inside one search box: which candidate deserves to rank first, and whether any candidate deserves to appear at all. An embedding gives us a similarity score. We still have to turn it into those decisions, and the consequences of getting them wrong are visible to anyone who can misspell a word.

The missing letter didn't erase the meaning

Here are the measured cosine similarities from the test corpus. Each project receives the score of its best matching chunk, using EmbeddingGemma 2's search task.

The typo moved Mundial's score down by about 0.015. It remained very close to the other candidate. The application converted that modest movement into complete disappearance because the cutoff was 0.60.

That's a useful distinction when debugging. The model's representation changed; the threshold made the outcome abrupt. Blaming the empty page entirely on the model would miss the application decision that produced it.

Lowering the cutoff would bring Mundial back. It would also bring Twitter Matcher back, still ahead of it. That gives us a concrete reason to treat admission and ordering separately: a threshold can hide a ranking problem, and relaxing it can expose the same problem to more readers.

The more specific query World Cup put Mundial first at 0.713430. An unrelated banana bread recipe query produced no project above the cutoff. Those are encouraging individual results. They leave a large unanswered question between them: how reliably does the system handle a reader who remembers the subject but doesn't know the project's name?

That reader is exactly who semantic search is supposed to help.

Six decimal places can disguise a weak decision

A score of 0.601943 looks precise. It describes a geometric comparison between vectors, with none of the evidence needed to interpret it as a probability that a reader will find the result useful. A cosine similarity of 0.60 does not mean 60% confidence.

The small margin between these two projects is especially revealing. Twitter Matcher led Mundial by roughly 0.0024 for the correctly spelled query. The interface presented a definite ordering even though that ordering failed an ordinary human judgment about the subject. More decimal places would describe the failure more precisely without making the decision better.

I would calibrate an acceptance threshold against labeled queries from the actual collection. That means including relevant results with low scores and irrelevant results with high scores, then examining the cost of each mistake. A discovery page can tolerate some exploratory matches. An assistant presenting a source as evidence for an answer needs a stronger reason to trust it.

The score distribution also belongs to a particular model, encoding scheme, and corpus. Carrying a convenient cutoff into a different collection is an untested assumption. The sports query is a small example of how expensive that assumption can become at the interface.

Correct prompts are a prerequisite, not a quality verdict

The obvious suspect was the embedding task. Retrieval is asymmetric: the short query and the longer source passage have different roles. EmbeddingGemma 2 provides separate query tasks for search, question answering, and code retrieval, documented in Google's Sentence Transformers guide.

That distinction matters. "Find the sports project" asks for a result. "How does the prediction system work?" asks for material that can support an answer. A code lookup has another purpose again.

In this case, public search already used the correct search prefix. Documents already included their titles, and there was no default prompt silently adding another prefix. Re-encoding the first Mundial chunk produced a cosine self-match of 1.0 against the stored vector. The bad ranking survived those checks.

There was a separate mistake in the question-answering console: it inherited the search task. Selecting the QA task corrected that mismatch without requiring a document rebuild, because the document encoding stayed the same. There is no controlled before-and-after answer-quality result to claim from that correction.

The finding I care about is that a correct embedding contract narrows the diagnosis. It rules out some plumbing errors. It cannot certify that the model's nearest passage is the passage a reader needs. Once those checks pass, the next useful question is what the retrieved passage says and why it deserves its position.

Hybrid search gives the ranking another source of evidence

Exact words still carry information. A project name, a code identifier, or "World Cup" can be a stronger clue than a broad semantic neighborhood. A lexical retriever preserves that clue, while a vector retriever can find passages whose wording differs from the request.

PostgreSQL can support both paths: pg_textsearch supplies BM25 ranking, and pgvector supplies vector similarity search. Keeping them beside the source records makes it practical to apply the same publication, language, and visibility rules to both candidate lists.

The combination still needs a defensible ranking rule. Adding a BM25 score directly to a cosine similarity assigns an arbitrary relationship to two different scoring systems. Reciprocal Rank Fusion uses positions instead: a candidate receives a contribution such as 1 / (60 + rank) from each list. A result found near the top of both lists accumulates support without pretending their raw scores share a scale. The constant and candidate budget remain evaluation choices.

There are two limits worth keeping in view. Fusion cannot recover Mundial if neither candidate list contains it. And even a completely irrelevant collection has a first-place result. A fused ranking therefore needs both candidate-recall checks and a separate decision about whether the available evidence supports returning anything.

For this site, BM25-plus-vector fusion remains a proposed improvement to the new retrieval path. The deployed Hybrid path combines vectors with PostgreSQL's built-in text ranking; it does not yet use the BM25 indexes. The sports results establish a failure to address. They provide no measured proof that RRF would fix it.

My reason to test hybrid retrieval is specific: it gives exact language an independent route into the candidate set. Whether that route helps broad phrases and misspellings is something the evaluation must establish.

More indexed material can make the wrong answer easier to find

The test corpus covered 221 sources or assets and 625 stored vectors, including complete article text, public project READMEs, images, and videos. Coverage checks found no missing or stale sources. The sports failure occurred with that coverage in place.

Those counts answer whether material is available for retrieval. They say nothing about whether the best passage supports the result. With chunked documents, inspecting only the parent title can conceal the problem: a project wins because one passage scored well, and the title makes the match look more convincing than the passage warrants. Evaluation needs to retain the winning excerpt.

Native media embeddings make the same distinction sharper. Image and video self-matches scored 1.0, confirming that stored encodings could retrieve their own assets. That leaves ordinary description-to-media retrieval unproven. The videos were represented by up to eight sampled frames, without soundtrack indexing or timestamp-level retrieval, so a successful match would still offer limited evidence about what happened throughout the clip.

An answering model also needs access to the evidence it describes. A retrieved video link and a source description do not give it the video's pixels. Claims about a particular scene require suitable visual input. Otherwise a larger index can produce a more confident answer with no stronger support.

What would convince me the search works

I would start with judgments a reader could defend: the sports query should retrieve the World Cup project; a misspelling should not casually remove it; an unrelated recipe request should not produce a project presented as relevant. Then I would broaden the set to paraphrases, Spanish queries, exact names, and code identifiers, with a held-out portion that tuning never sees.

The first measurement is candidate recall at a fixed budget. If a relevant source never reaches the candidate set, reranking cannot help. Once it does, MRR or nDCG can track whether useful results move toward the top. Unrelated queries need their own false-positive checks, and latency has to be measured alongside quality. BM25 alone, vectors alone, and fusion should face the same queries and relevance judgments.

For generated answers, I would inspect whether each claim is supported by the retrieved excerpt and whether the citation points to that support. A relevant project at rank one is only the beginning of that test.

This is where I land on embeddings: I want them in the retrieval system, and I want their decisions exposed to evidence. The sports failure shows why. Correct task prompts and complete indexing coexisted with a poor ordering; a small score change crossed an arbitrary boundary and removed a useful result. Another database extension could not change those existing vectors or their cosine scores.

Before spending the next round of effort on a model swap, I would spend it on labeled queries and the passages that won. Then a hybrid retriever has a concrete job to do and a result we can judge. A reader should be able to arrive with an imperfect description and still find the World Cup project. Until that survives evaluation, the vector pipeline's clean bill of health is of little comfort to the person staring at an empty page.