Building a RAG Pipeline from Scratch

Retrieval-Augmented Generation is the most practical pattern I've found for giving an LLM access to your own data. Instead of fine-tuning a model on your documents — expensive, slow, and stale within weeks — you retrieve the right content at query time and pass it as context. The concept is simple. The pipeline that actually works in production is not.

This walks through every component, the decisions you have to make at each stage, and the failure modes you'll hit on the way.

Architecture

A RAG pipeline has four stages:

There's also an offline ingestion pipeline that processes your source documents:

I'll go through each piece in the order you'll build it.

Document ingestion and chunking

Your documents need to be split into chunks before embedding. This is where most RAG pipelines either succeed or fail, and it's also the step that gets the least attention because it isn't glamorous.

Why chunk

Embedding models have a fixed context window, usually 512-8192 tokens. Even when the model technically supports a long input, embedding quality drops fast on long text — the resulting vector becomes an average of too many concepts to be useful for precise retrieval.

Chunking strategies

Fixed-size chunking is the simplest. Split the text into N tokens with M tokens of overlap.

def fixed_size_chunks(text, chunk_size=512, overlap=50):
    tokens = tokenizer.encode(text)
    chunks = []
    for i in range(0, len(tokens), chunk_size - overlap):
        chunk_tokens = tokens[i:i + chunk_size]
        chunks.append(tokenizer.decode(chunk_tokens))
    return chunks

It works, but it's naive. It splits mid-sentence and mid-paragraph, producing chunks that don't carry coherent meaning on their own.

Recursive character splitting (LangChain's RecursiveCharacterTextSplitter) tries to split on natural boundaries first — paragraphs, then sentences, then words — and only falls back to smaller units when it has to.

Semantic chunking uses an embedding model to detect topic shifts. You compute embeddings for each sentence, then split when cosine similarity between consecutive sentences drops below a threshold. The chunks end up aligned with actual content boundaries instead of arbitrary token counts.

Document-structure-aware chunking respects the structure of the source. For Markdown, split on headings. For HTML, split on semantic tags. For PDFs, use layout analysis to identify sections. It's the most work and produces the best results when your source documents are well-structured.

Practical recommendations

Chunk size of 256-512 tokens is the sweet spot for most use cases. Smaller chunks improve retrieval precision, larger chunks give the LLM more context per retrieval. Use 10-20% overlap to stop information from being cut at boundaries. Always store metadata with each chunk — source document, section heading, page number, creation date. You'll need it for filtering, citations, and debugging the retrieval when things go wrong, which they will.

Embedding models

The embedding model converts text into dense vectors that capture meaning. Embedding quality is the single biggest determinant of retrieval quality.

Options

Model	Dimensions	Max Tokens	Notes
OpenAI text-embedding-3-small	1536	8191	Good cost/quality balance
OpenAI text-embedding-3-large	3072	8191	Best quality from OpenAI
Sentence-BERT (all-MiniLM-L6-v2)	384	512	Free, fast, fine for prototyping
BGE-large-en-v1.5	1024	512	Strong open-source option
Gemini text-embedding-004	768	2048	Competitive, fits Google stack
Cohere embed-v3	1024	512	Strong multilingual

For production I default to OpenAI's embeddings because the quality-to-effort ratio is hard to beat. For cost-sensitive or high-volume cases, self-hosted BGE or Sentence-BERT eliminates per-call cost entirely.

Best practices

Embed queries and documents the same way. Some models, BGE in particular, support instruction-prefixed embeddings — you prepend "Represent this document for retrieval:" to documents and "Represent this query for retrieval:" to queries. When the model supports it, the lift is real.

from openai import OpenAI

client = OpenAI()

def embed(texts, model="text-embedding-3-small"):
    response = client.embeddings.create(input=texts, model=model)
    return [item.embedding for item in response.data]

# Batch embedding for efficiency — up to 2048 texts per call
doc_embeddings = embed(chunks)

Vector store

The vector store holds the indexed embeddings and runs similarity search. Choice depends on scale and infrastructure constraints.

Chroma is embedded, Python-native, zero-config. Excellent for prototyping and applications under 1M documents. Local data.

Pinecone is fully managed and serverless. It scales without ops effort, but you pay for it and you take on a dependency on an external service.

pgvector is a PostgreSQL extension. If you already run Postgres, this is the obvious move because it avoids adding another database to the stack. With proper HNSW indexing, it handles millions of vectors comfortably.

Weaviate, Qdrant, and Milvus are purpose-built vector databases with hybrid search, filtering, multi-tenancy, and so on. Pick one of these when vector search is core to your application rather than a side feature.

For most projects, start with pgvector if you already use Postgres or Chroma if you want the fastest path to a working prototype.

# pgvector example with SQLAlchemy
from pgvector.sqlalchemy import Vector
from sqlalchemy import Column, Integer, Text
from sqlalchemy.orm import declarative_base

Base = declarative_base()

class Document(Base):
    __tablename__ = "documents"
    id = Column(Integer, primary_key=True)
    content = Column(Text)
    embedding = Column(Vector(1536))  # Match your model's dimensions
    source = Column(Text)

Retrieval strategies

Naive top-K retrieval works in a lot of cases but has well-known failure modes once you push it.

Top-K similarity

Compute cosine similarity between the query embedding and every document embedding, return the top K.

query_embedding = embed([user_query])[0]
results = vector_store.similarity_search(
    query_embedding,
    k=5
)

The problem with top-K is that it often returns redundant results. If three of your chunks are paraphrases of the same fact, you've burned 60% of your context window on duplicates.

Maximal Marginal Relevance

MMR balances relevance against diversity. It iteratively picks documents that are similar to the query but dissimilar to documents already chosen.

results = vector_store.max_marginal_relevance_search(
    query_embedding,
    k=5,
    fetch_k=20,       # Fetch 20 candidates
    lambda_mult=0.7,   # 0=max diversity, 1=max relevance
)

A simple change, and it noticeably improves output quality on most applications.

Hybrid search

Combine dense vector search with sparse keyword search (BM25). Dense retrieval is great at semantic similarity ("What causes inflation?"). Sparse retrieval handles exact matches and rare terms better ("error code 0x80070005").

# Hybrid search: BM25 + vector similarity
bm25_results = bm25_index.search(query, k=10)
vector_results = vector_store.search(query_embedding, k=10)

# Reciprocal Rank Fusion
combined = reciprocal_rank_fusion(bm25_results, vector_results)
final_results = combined[:5]

Most of the production RAG systems I've seen end up running hybrid search. pgvector plus Postgres full-text search gives you both in one database with no extra moving parts.

Prompt construction

How you present retrieved context to the model matters about as much as what you retrieve.

def build_prompt(query, retrieved_docs):
    context = "\n\n---\n\n".join([
        f"Source: {doc.metadata['source']}\n{doc.content}"
        for doc in retrieved_docs
    ])

    return f"""Answer the user's question based on the provided context.
If the context doesn't contain enough information to answer fully,
say so rather than making up information.

Context:
{context}

Question: {query}

Answer:"""

Place the context before the question. LLMs attend better to content near the end of the prompt — the "lost in the middle" effect — so the question goes last. Include source attribution so the model can cite, and so you can debug retrieval issues. Set explicit boundaries when factual grounding matters; tell the model not to use anything outside the provided context. And watch the context window. Five chunks of 512 tokens is 2500 tokens of context, and you still need room for the system prompt, the query, and the response.

Evaluation

RAG needs evaluation at two levels: retrieval quality and generation quality.

Retrieval metrics

Recall@K is the fraction of relevant documents that show up in the top K results. MRR (Mean Reciprocal Rank) tells you how high the first relevant result ranks. Precision@K is the fraction of returned results that are actually relevant.

You need an evaluation set. Take 50-100 representative queries, manually identify the correct source documents for each, and measure your pipeline against that ground truth. Without this, every change you make is a guess.

Generation metrics

Faithfulness — does the answer only use information supported by the retrieved context? (No hallucinations.) Relevance — does the answer actually address the user's question? Completeness — does it cover all the important information in the retrieved context?

RAGAS and similar frameworks automate these using an LLM-as-judge approach. It scales fine and gives you a way to measure quality as you iterate.

Common failure modes

Wrong chunks retrieved

The model gives confident but incorrect answers, or it says "I don't have information about that" when the answer is sitting in your corpus. Log the retrieved chunks for every query. Check whether the correct information is in your vector store at all, and if it is, whether the query embedding is actually close to it. The fix is usually better chunking, not a different embedding model.

Relevant chunks exist but rank too low

Increasing K from 5 to 20 fixes it, but blows up your prompt. Try hybrid search, cross-encoder re-ranking, or query expansion (generate multiple phrasings of the user's question and retrieve for each).

The model ignores retrieved context

Answer sounds plausible but doesn't reflect the retrieved documents. Strengthen system prompt instructions. Use a more capable model. Cut the amount of retrieved context so the signal-to-noise ratio is higher.

Chunk boundaries split key information

Answer is partially correct and missing details that span two chunks. Increase overlap. Use document-structure-aware chunking. Or use a parent-document retriever pattern — retrieve small chunks but pass their parent section to the LLM.

Putting it together

A minimal but production-worthy pipeline:

# Full pipeline sketch
from openai import OpenAI
import psycopg2

client = OpenAI()

def ingest(documents):
    for doc in documents:
        chunks = recursive_split(doc.content, chunk_size=512, overlap=50)
        embeddings = embed(chunks)
        for chunk, emb in zip(chunks, embeddings):
            db.execute(
                "INSERT INTO documents (content, embedding, source) VALUES (%s, %s, %s)",
                (chunk, emb, doc.source)
            )

def query(user_question):
    q_emb = embed([user_question])[0]

    # Hybrid retrieval
    results = db.execute("""
        SELECT content, source,
               1 - (embedding <=> %s::vector) as similarity,
               ts_rank(to_tsvector(content), plainto_tsquery(%s)) as text_rank
        FROM documents
        ORDER BY (0.7 * (1 - (embedding <=> %s::vector)) + 0.3 * ts_rank(to_tsvector(content), plainto_tsquery(%s))) DESC
        LIMIT 5
    """, (q_emb, user_question, q_emb, user_question))

    prompt = build_prompt(user_question, results)

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content

Start here, measure with a real evaluation set, and iterate. Most of the wins in RAG come from better chunking and retrieval, not from swapping LLMs or embedding models. Get the data pipeline right first; everything else is downstream.