RAG Explained: How Retrieval-Augmented Generation Works

The short definition

RAG stands for retrieval-augmented generation: instead of asking an LLM a question and hoping it already knows the answer, you first retrieve relevant text from a document store, then hand that text to the LLM alongside the question so it can generate an answer grounded in real, current information. That's the whole idea in one sentence — retrieve, then generate.

The pattern exists because an LLM's knowledge is frozen at training time and limited by its context window. RAG works around both limits: it can pull in information the model never saw during training, and it only feeds in the specific passages relevant to a given question rather than trying to cram an entire knowledge base into the prompt.

How it actually works

A RAG pipeline has two halves that run in sequence for every query. First, retrieval: your question gets converted into a vector embedding, then compared against a database of pre-embedded document chunks to find the most semantically similar ones. Pinecone is a common choice for this step — it's a fully managed vector database priced on storage plus read and write units, with a free Starter tier covering 2 GB of storage, 2M write units, and 1M read units per month across up to 5 indexes.

Second, generation: the retrieved passages get inserted into a prompt template along with the original question, and that combined prompt goes to an LLM like Claude. Claude Sonnet 5 costs $2 per million input tokens and $10 per million output tokens, which matters directly in a RAG system since every retrieved passage counts as input tokens on every query. Prompt caching is the relevant cost lever here: a cache hit costs 10% of the standard input price, so if your retrieved context or system prompt is reused across queries, caching it cuts that portion of the bill dramatically.

The retrieval step's cost also scales with usage in a specific way: on Pinecone, reads are the more expensive dimension once you're past the free tier, running $16 to $18 per million read units on the Standard plan, so a RAG app that queries the vector database on every user request will hit that cost before it hits write costs.

When you need it — and when you do not

You need RAG when your application has to answer questions using information the LLM wasn't trained on: internal documents, recent data, a specific knowledge base, or anything proprietary. It's also useful for reducing hallucination on factual questions, since the model is working from retrieved text rather than only its training memory.

You don't need RAG for tasks that are self-contained within a single conversation or a short document you can just paste directly into the prompt — if your entire knowledge base fits comfortably in the LLM's context window for a given query, skip the retrieval step entirely and save the added infrastructure and latency. Claude's 4.6 and later models include a full 1M token context window at standard pricing, which raises the bar for when RAG's extra complexity is actually worth it versus just including everything in the prompt directly.

Cost also shapes the decision. A read-heavy RAG app on Pinecone's Standard tier carries a $50/mo minimum spend even if actual usage would cost less, and Pinecone's Builder tier at a flat $20/mo is a reasonable middle ground once you outgrow the free Starter tier but before you're ready for metered Standard pricing.

The tools that do this

Pinecone and Claude are the two pieces with verified pricing here, and they cover the two halves of a RAG pipeline: Pinecone for retrieval, Claude for generation. For more on the retrieval half specifically, see our what is a vector database guide, which goes deeper on how Pinecone and similar tools store and search embeddings.

On the generation side, Claude's pricing tiers span a wide range depending on how much quality you need per query: Claude Haiku 4.5 at $1 per million input tokens for cheap, fast responses, Claude Sonnet 5 at $2 per million for a balance of cost and capability, and Claude Opus 5 at $5 per million input tokens for the highest-quality generation. The Batch API adds a further 50% discount on both input and output tokens for RAG workloads that don't need real-time responses, and that discount stacks with prompt caching — a combination worth using if your RAG system processes documents asynchronously rather than answering live user queries.

Choosing between these two pieces isn't really an either-or decision the way a typical vs-comparison is — a working RAG system needs both a place to store and search embeddings and a model to generate the final answer. The practical question is how to size each one for your workload: Pinecone's free Starter tier or $20/mo flat Builder tier for retrieval at low-to-moderate scale, and Claude Haiku or Sonnet for generation depending on whether you need fast, cheap responses or higher-quality reasoning over the retrieved context. Most teams prototype on the cheapest tier of both and only move to metered Standard pricing on Pinecone or a pricier Claude model once real usage data shows where the bottleneck actually is.

One more detail worth knowing before you budget a RAG project: Claude models from version 4.7 onward use a tokenizer that produces roughly 30% more tokens for the same amount of text than earlier versions. That affects both the retrieval context you're feeding in and the generated output, so a per-token price comparison across model versions isn't quite apples to apples unless you account for the tokenizer difference.