JobMap: relevance, strict matches, and latency

The live evaluation froze 237 eligible postings and 48 predefined requests. Each method ran each request twice for timing; quality uses the first repeat. The requests were written for the benchmark, not sampled from user traffic.

Request → lexical candidates (up to 30) → Jev relevance judgment
Request → query embedding → exact cosine search over 237 postings
Results from four methods → pooled pairs → blind relevance judge

Jev saw the first 1,000 characters of each candidate's search text. MiniLM, BGE-small, and Arctic-embed-s ran locally on CPU and searched representations built from 600-character chunks with 100-character overlap and normalized mean pooling. Exact pgvector cosine search avoids approximate-index recall as a confounder. The methods still have different candidate coverage and text exposure, so the comparison measures complete pipelines.

The pool combines each method's top ten with two random postings where available. Deduplication produced 1,399 request/posting pairs. The judge saw full posting text and explicit metadata without method names, ranks, or retrieval scores.

For a request requiring work from Mexico, a remote posting with no country eligibility cannot receive full credit merely because remote work sounds compatible. Accepted judgments include reasons, evidence, and unmet or unknown requirements. Two HTTP 520 failures succeeded on retry; failed requests were not converted into negative relevance labels.

nDCG uses gains of 3, 1, and 0 for grades 2, 1, and 0, and averages the 45 requests with a relevant item in the judged pool. Strict precision uses all 48 requests and 240 possible top-five positions per method; empty positions are misses. These measures answer different questions.

Jev produced 32 strong and 107 partial top-five appearances. MiniLM produced 33 strong and 81 partial appearances. The graded relevance gain is useful, but it does not establish more fully supported jobs. These are request/job appearances, not distinct postings. The downloadable metrics CSV also contains a graded precision field; do not mistake it for the strict grade-2 precision shown here.

For the request about interviewing users and designing and testing product experiences and prototypes, Jev returned ten partial matches in its first ten results. MiniLM returned one strong, three partial, and six irrelevant matches. A browsing interface can value the first list while a requirements checker values the supported match in the second.

Open the detailed evaluation and inspect the judgments. The blind grading workbench supports a separate review; export ratings to retain them because the sandboxed attachment cannot persist them in site storage.

The judge used 2,634,345 input tokens and 187,458 output tokens, including reasoning. Reported cached input was 2,485,742 tokens. The historical estimate was about $2.48 under the recorded pricing assumptions, excluding unreported usage from failed requests. It is judging cost, not hosting, embedding, or Jev search cost. Usage and assumptions.

A useful reproduction freezes corpus, requests, text preprocessing, candidate limits, model configuration, and rubric. Keep failed calls and repeat timings. Compare equal candidate and text coverage before attributing a gain to the model, and obtain independent human judgments on a sample. Pooled labels cannot establish corpus-wide recall. The 72-posting synthetic test is a separate track and should not be averaged into these live results.

The technical article explains the synthetic comparison and evidence archive. The live ranking difference remains descriptive; this run does not establish statistical significance or a universal search winner.