The Problem RAG Solves
A language model’s knowledge is frozen at training time and limited to what it learned during training — it has no inherent access to your company’s documents, your latest data, or anything that happened after its cutoff. Retrieval-augmented generation (RAG) fixes this by fetching relevant information at query time and handing it to the model as context before it generates a response.
How RAG Works, Step by Step
A user query is converted into a vector embedding — a numerical representation of its meaning. That embedding is compared against a database of pre-embedded document chunks to find the most semantically similar ones. Those chunks are inserted into the prompt alongside the original query, and the model generates its answer grounded in that retrieved context rather than relying purely on its training data.
Chunking Strategy Matters More Than You’d Think
How you split documents into chunks has an outsized effect on retrieval quality. Chunks that are too large dilute relevance and waste context; chunks that are too small lose surrounding context needed to make sense of the content. Overlapping chunks and preserving semantic boundaries (like not splitting mid-sentence or mid-table) consistently improve results over naive fixed-length splitting.
Vector Databases
Purpose-built vector databases (like Pinecone, Weaviate, or pgvector as a Postgres extension) handle the similarity search efficiently at scale. For smaller datasets, even an in-memory search can work fine — the right choice depends on your document volume and query latency requirements, not on chasing the most feature-rich option by default.
RAG Isn’t a Silver Bullet
Retrieval can fail silently — if the relevant chunk isn’t retrieved, the model will either say it doesn’t know (good) or hallucinate an answer anyway (bad). Evaluating retrieval quality separately from generation quality is essential: a great model with poor retrieval still produces poor answers.
Practical Tips
- Re-rank retrieved chunks with a smaller, faster model before passing the top results to the main model.
- Include source citations in the prompt so the model can reference where information came from, and so you can verify its answers.
- Regularly evaluate retrieval quality with a labeled set of query-to-relevant-chunk pairs, separate from end-to-end answer evaluation.