Retrieval-Augmented Generation (RAG) is the technique behind most "chat with your data" AI features you've seen in 2024–2026. If you're building a product that needs the LLM to know about YOUR data — not just the internet's data — RAG is probably the pattern.
Here it is in plain English.
The problem
LLMs are trained on public data from a specific cutoff date. Claude Opus 4.7 knows about the world through January 2026. It has no idea what happened last week, and no idea what's in your company's internal documents.
If you want an LLM to answer questions using your data — support docs, product manuals, customer records, legal contracts — you need to give it access to that data somehow.
Two options: fine-tuning or RAG.
Fine-tuning
Fine-tuning bakes your data into the model itself. Expensive, slow, and hard to update. Best for teaching the model a specific style or format, not for teaching it new facts.
Cost: $5,000–$50,000+ per model, plus retraining every time your data changes meaningfully.
RAG
RAG keeps your data separate from the model. When a question comes in, you first RETRIEVE the relevant pieces of your data, then include them in the prompt when you ASK the model.
Roughly:
- User asks: "What's our return policy for damaged items?"
- Your system searches your documents, finds the 3 most relevant paragraphs
- Your system sends the model: "Here are the relevant policies. [pastes them]. Now answer: what's our return policy for damaged items?"
- Model answers with citations to the source paragraphs
Simpler, cheaper, updateable in real-time. This is why RAG has become the default.
When RAG wins
- You have data the LLM doesn't know about (internal docs, customer records, product info)
- Your data changes frequently
- Users need answers grounded in specific sources (not general knowledge)
- You want to show citations (regulated industries often require this)
When RAG isn't enough
- You need the model to write in a very specific style you can't prompt for (consider fine-tuning)
- Your task is more about reasoning than retrieval (better prompting is usually the answer)
- Your data is too small to bother retrieving from (just paste it all in the context)
What a production RAG system looks like
1. Data ingestion pipeline. Every document (PDF, HTML, database record) gets parsed, cleaned, and chunked into passages of 500-1500 tokens each.
2. Embedding. Each chunk gets converted to a vector (list of numbers) using an embedding model (OpenAI's text-embedding-3-large, Cohere Embed, or open source).
3. Vector database. The vectors get stored in a specialised database (Pinecone, Weaviate, pgvector, Qdrant) that can find similar vectors quickly.
4. Retrieval. When a user asks a question, embed the question, find the top N most similar chunks in the vector DB, return them.
5. Reranking (optional). The top N chunks get reranked by a smaller model that's more accurate on relevance. Boosts quality significantly.
6. Generation. The reranked chunks get pasted into the prompt with instructions, sent to the LLM, response comes back.
7. Citation handling. The response includes references to which chunks were used. Users can verify.
The parts founders underestimate
Chunking strategy matters a lot. Bad chunking (splitting mid-sentence, losing context) destroys retrieval quality. Getting this right is 30% of a good RAG system's quality.
Evaluation is hard. You need a test set of question-answer pairs that measures retrieval quality (did we find the right chunks?) AND generation quality (did we answer correctly?). Building this evaluation harness is often 25% of total project effort.
Latency is a real constraint. A full RAG round-trip involves embed → search → rerank → LLM. Total latency is 1–4 seconds. For customer-facing features, this needs to be aggressively optimised.
Cost adds up. Every query does an embedding (cheap), a vector search (cheap), and an LLM call (not cheap). At high volume, cost management matters.
Cost model
For a mid-scale RAG system (100K queries/month, 10K documents):
- Embeddings: ~$50/month (one-time embed of docs + per-query embed)
- Vector DB: $70–$500/month (Pinecone, pgvector on RDS, or self-hosted)
- LLM: $300–$3,000/month depending on model choice
- Reranking (optional): $100–$500/month
Total: $500–$4,000/month for a mid-scale system.
Common use cases
- Customer support bots answering from your help docs
- Internal Q&A letting employees search company knowledge
- Legal research grounded in your firm's own matter history
- Sales enablement pulling from your product docs, case studies, competitive intel
- Semantic search over any large document corpus
Our RAG offering
We ship production RAG systems for clients. Discovery week to scope requirements, 4–8 weeks to build, ongoing retainer for quality tuning. Book a consultation and we'll walk through what your specific use case looks like.