RAG (retrieval-augmented generation)
Last verified
RAG retrieves relevant documents and includes them in the prompt so the model can answer from them.
The cost mechanism is simple and often overlooked: retrieved chunks are input tokens on every request. Increasing top-k from 3 to 10 raises the cost of every single query, permanently, and there is no notification.
The two levers are how much you retrieve and how large each chunk is. A step change in spend with no traffic change is frequently a retrieval parameter someone tuned for quality.