What Is RAG? Retrieval-Augmented Generation Explained

Beginner 11 min read

What you'll learn

✓Why an LLM trained on the public web cannot answer questions about your data
✓What embeddings are and how vector search works at a high level
✓The retrieve-then-generate pipeline, end to end
✓When RAG is the right tool, and when fine-tuning fits better
✓A small mental-model code outline you can reason about

Prerequisites

•What Is an LLM? covers the model behind the R in RAG
•Light Python familiarity — see What Is Python?

A common moment: you wire up an LLM, ask it a question about your company’s internal docs, and it cheerfully invents an answer that isn’t in any document you’ve ever shipped. The model doesn’t know your data. Retrieval-augmented generation — RAG — is the standard way to fix that.

This post explains what RAG is, how it works, and when to use it.

The problem RAG solves

An LLM’s knowledge lives in its weights. Those weights were frozen at the end of training. That means:

Anything after the training cutoff is unknown.
Anything private — your wiki, your codebase, your customer notes — is unknown.
Anything obscure is fuzzy at best.

You could try to solve this by stuffing every document into the prompt. That fails for two reasons. First, context windows are finite and tokens cost money. Second, even when the window is large enough, models pay less attention to information buried in long contexts.

RAG’s idea is simpler: for each question, fetch only the most relevant chunks of your data, and put just those into the prompt.

Embeddings, briefly

To “fetch the most relevant chunks,” you need a way to measure relevance between a question and a chunk of text. The standard tool is embeddings.

An embedding model takes a piece of text and turns it into a fixed-length vector of numbers — typically 768 or 1536 dimensions. The clever bit: texts with similar meaning end up close together in this vector space, even if they share no words in common.

# Conceptual
embed("How do I reset my password?")
# output: [0.12, -0.04, 0.88, ..., 0.07]   (e.g., 1536 numbers)

embed("I forgot my login")
# output: [0.10, -0.05, 0.85, ..., 0.06]   (close to the first)

embed("How do I file my taxes?")
# output: [0.91, 0.33, -0.12, ..., 0.40]   (far away)

“Close” usually means high cosine similarity — the angle between the two vectors. Once you can measure distance, search becomes a simple sort.

Vector search

A vector database stores documents alongside their embeddings and gives you a fast “find the K closest” query. Common options include pgvector (a Postgres extension), Pinecone, Weaviate, Qdrant, Chroma, and Milvus.

The query pattern is always the same:

Embed the user’s question.
Ask the vector store for the top-K nearest chunks.
Return those chunks.

For small datasets you don’t even need a dedicated database — a NumPy array and a cosine_similarity call will do. For millions of documents, specialised indexes (HNSW, IVF) keep the search fast.

The RAG pipeline, end to end

A typical RAG system has two phases.

Indexing (offline, run once or on update):

Load your documents — wiki pages, PDFs, support tickets, code files.
Chunk them into pieces small enough to fit comfortably in a prompt — often 200 to 1000 tokens each.
Embed each chunk with an embedding model.
Store the chunk text plus its vector in a vector database.

Querying (online, runs per user question):

Embed the question with the same embedding model.
Retrieve the top-K most similar chunks.
Build a prompt that includes the chunks plus the question.
Generate an answer with an LLM, citing the chunks.

That’s the whole architecture. Everything else — re-ranking, hybrid search, query rewriting — is optimisation around this skeleton.

A mental-model code outline

A tiny, runnable-in-spirit version of RAG. Real systems are wordier; the shape is the same.

# Conceptual outline — uses placeholders for the model + DB
from some_llm_sdk import embed, chat
from some_vector_db import VectorStore

store = VectorStore()

# --- Indexing ---
def index_documents(docs: list[str]) -> None:
    for doc in docs:
        for chunk in chunk_text(doc, size=500):
            vector = embed(chunk)
            store.add(text=chunk, vector=vector)

# --- Querying ---
def answer(question: str, k: int = 4) -> str:
    q_vec = embed(question)
    top_chunks = store.search(q_vec, k=k)

    context = "\n\n".join(c.text for c in top_chunks)

    prompt = f"""
Use only the context below to answer the question.
If the answer is not in the context, respond with "I don't know."

Context:
\"\"\"
{context}
\"\"\"

Question: {question}
"""
    return chat(prompt)

# --- Use ---
index_documents(load_wiki_pages())
print(answer("How do I rotate the API key?"))
# output: To rotate the API key, open Settings → API → "Rotate". ...

A few things to notice:

The LLM is instructed to use only the context. This is what reduces hallucinations.
The “I don’t know” out is explicit. Without it the model will fill silence with confidence.
Embedding for indexing and embedding for querying use the same model. Mixing models breaks similarity.

Chunking is half the work

How you split documents matters as much as the model you use.

Too small — chunks lose context. “He resigned in March” is unhelpful without knowing who.
Too large — chunks dilute relevance. Half the retrieved text is unrelated and confuses the model.
Cut mid-sentence — meaning fractures. Prefer paragraph or section boundaries.

A reasonable default: 300–800 tokens per chunk, with 10–20% overlap so concepts that straddle a boundary appear in two chunks. Then evaluate.

For structured sources (PDFs with headings, code repositories) preserve structure in the chunk. Add metadata — title, section, source URL — so you can show citations.

Reflection. Think about a set of documents you’d want to query — internal docs, your notes, a folder of PDFs. What is the natural unit of one “answer”? A paragraph? A code function? A FAQ entry? That unit is your chunk target. Designing the chunker is where most engineers spend their first week on RAG.

RAG vs fine-tuning

A frequent question: should I fine-tune the model on my data instead?

The honest answer is they solve different problems.

RAG is better when:

The data changes often — you don’t want to retrain every Tuesday.
The answer requires citing sources.
You need to add or remove documents dynamically (e.g., per-user data).
You want traceable behaviour — you can see which chunks the model used.

Fine-tuning is better when:

You need to teach the model a format or style it doesn’t naturally produce.
The data is stable and the volume is large enough to justify training.
Latency or token cost from injecting context is a problem.

Most production “AI features” you’ve used are RAG, not fine-tuning. The two also compose well: fine-tune for tone and structure, retrieve for facts.

Where RAG goes wrong

Common failure modes worth knowing:

Retrieval misses. The right chunk exists but isn’t in the top K. Fix with better chunking, hybrid search (combining vector + keyword), or query rewriting.
Bad chunks rank high. Synonyms confuse vector similarity. Re-ranking with a smaller LLM helps.
The model ignores the context. Usually a prompt issue — strengthen the “use only the context” rule and demand citations.
Stale index. Documents updated, embeddings didn’t. Build a re-index pipeline from day one.
No evaluation. You can’t improve what you don’t measure. Build a golden set of question/answer pairs and run it on every change.

What you need to build a real RAG system

A small but real production setup:

A document loader — for PDFs, HTML, Markdown, whatever you have.
A chunker — paragraph-aware, with metadata preserved.
An embedding model — from OpenAI, Anthropic, Cohere, or open-source (e.g., bge, e5).
A vector store — pgvector on a Postgres you already run is a great default.
An LLM API — same provider as your embeddings is convenient but not required.
A web layer — FastAPI is the common Python choice.
Logging and evals. Every query, every retrieved chunk, every generated answer. Eval against a golden set on every prompt or model change.

That stack will get you to a useful internal Q&A tool in a week or two for someone comfortable with Python.

Try it yourself. Take 20 pages of any documentation you have. Split into paragraphs. Use an embedding model to encode each paragraph. Encode a question, sort by cosine similarity, take the top 3, and paste them along with the question into your favourite chatbot. Compare the answer to what the same chatbot says without the retrieved context. The improvement is usually obvious.

Recap

You now know:

RAG injects relevant private data into the LLM’s prompt at query time
Embeddings turn text into vectors where similar meaning = nearby vectors
Vector search finds the top-K chunks most similar to a query
The pipeline is: chunk + embed + store, then retrieve + prompt + generate
RAG suits frequently-changing data with citations; fine-tuning suits stable style/format
Chunking and evaluation matter more than picking the trendiest vector DB

Next steps

RAG bridges LLMs and your data. If you’d like to work with structured data more broadly — tables, CSVs, analytics — the standard Python toolkit is pandas.

→ Next: What Is Pandas?

Questions or feedback? Email codeloomdevv@gmail.com.