Retrieval-augmented generation (RAG) is the most common way companies use AI: a chatbot that answers questions from their own documents, with sources. Here is how it works and how to build one, step by step.
How RAG works
RAG has two phases:
INDEXING (once, and whenever documents change)
documents → clean → split into chunks → embed each chunk → store in a vector index
ANSWERING (every question)
question → embed → find the most similar chunks → (re-rank)
→ prompt = instructions + chunks + question → LLM → answer with sourcesAn embedding is a list of numbers that captures the meaning of text. Texts with similar meaning have similar embeddings, so “How do I return a product?” finds a passage titled “Refund policy” even without matching words.
Step 1: Collect and clean your documents
Pick a small, real set of documents you know well: a product manual, company policies or your college notes. Convert them to clean text. PDFs often contain headers, footers and broken lines; remove them, because messy text leads to poor retrieval.
Tip: Keep each document’s title, section headings and page numbers. You will need them for citations.
Step 2: Split documents into chunks
Models work best with focused passages. Split each document into chunks of roughly a few hundred words, with a small overlap so sentences at the edges are not lost. Better still, split along headings so each chunk covers one topic.
Tip: Add the document title and section heading to the start of each chunk’s text. It noticeably improves retrieval.
Step 3: Create embeddings and store them
Send each chunk to an embedding model and store the vector alongside the text and metadata (source, page, date). For a prototype, a list in memory is fine. For a real app, use a vector database or PostgreSQL with pgvector, which keeps everything in one database. Our PostgreSQL roadmap covers pgvector.
Step 4: Retrieve the best chunks
When a question arrives, embed it with the same model and find the chunks with the highest similarity (cosine similarity is standard). Take the top 3 to 8.
To improve results:
- Hybrid search: combine vector search with keyword search, which catches exact names and codes.
- Re-ranking: reorder the top 20–50 candidates with a more precise re-ranker model and keep the best few.
- Metadata filters: limit search to the right product, date range or user permissions.
Step 5: Write the prompt
Give the model clear rules:
Answer the question using only the context below.
Cite sources in [brackets] after each claim.
If the answer is not in the context, say you don't know.
Context:
[Refund policy, p.2] Items can be returned within 30 days...
[Shipping FAQ, p.1] ...
Question: Can I return a product after 3 weeks?Telling the model to admit when the answer is missing is one of the most effective ways to reduce made-up answers.
Step 6: Show the answer with sources
Display the answer with clickable citations. Sources build trust and make mistakes easy to spot and fix.
Step 7: Evaluate before you tune
Write 20–30 real questions with the passage that should answer each. Then measure two things separately:
- Retrieval: was the right passage in the top results? (hit rate)
- Answers: correct, complete and supported by the sources?
Most RAG problems are retrieval problems. Fix chunking and search before changing prompts or models, and re-run your test set after every change.
Common mistakes
- Chunks too large (vague retrieval) or too small (missing context).
- Ignoring tables and headings in documents.
- No permission filtering, so users see documents they shouldn’t.
- Tuning prompts without a test set.
Going further: agentic RAG
For complex questions, give retrieval to an AI agent as a tool, so it can search several times and compare documents. You can expose your retriever as an MCP tool so any AI app can use it. See how to build an AI agent and how to build an MCP server.
Learn RAG step by step
Our free RAG roadmap breaks this into six steps, each with notes, tips and tricks, code and a practice project. New to AI terms? Start with the AI family tree.
વારંવાર પૂછાતા પ્રશ્નો
RAG in simple words શું છે?
RAG (retrieval-augmented generation) means searching your documents for the passages that match a question and giving them to an AI model, so it answers from your data instead of guessing, and can show its sources.
Do I need a vector database for RAG?
Not at first. For a few hundred chunks, you can keep embeddings in memory. For real apps, use a vector store; PostgreSQL with the pgvector extension works well if you already use PostgreSQL.
RAG or fine-tuning?
Use RAG to give a model knowledge (your documents, which change often). Use fine-tuning to change its behaviour or style. Many products use RAG only.
