RAG Token Accounting: How Retrieval Costs Compound Across Conversations

The core observation is straightforward but often overlooked: when you retrieve K chunks of ~N tokens for context, that retrieval contributes K*N tokens to each model call - and in multi-turn interactions those tokens are billed repeatedly. For example, retrieving 8 chunks of ~250 tokens results in ~2,000 context tokens per query; in a 10-turn session that's 20,000 tokens of retrieved context alone, not counting model-generated tokens. The arithmetic makes retrieval a recurring cost center rather than a one-off overhead.

This has direct implications for architecture and economics. Systems that rely on high top_k to maximize recall will face compounding costs and increased latency. Conversely, overly aggressive compression or too-small top_k can harm relevance and user satisfaction. Therefore, design trade-offs must be explicit and data-driven: measure retrieval marginal benefit (accuracy lift) versus marginal token cost and latency.

Operational strategies to control token bills include: reduce top_k or chunk size; pre-summarize or compress chunks on ingest; cache retrieved context for session-based reuse; dynamically vary top_k based on query intent or confidence; and offload some logic to smaller grounding models or heuristics. Engineering controls like budget-aware orchestration, adaptive retrieval policies, and token accounting dashboards are essential for cost predictability.

For business leaders: treat RAG design as both a UX and a finance problem. Set clear SLAs and cost KPIs, require cost/benefit testing for retrieval parameters, and fund observability into token usage per user/session. Where appropriate, prioritize architectural changes (summarization, caching, hybrid models) that reduce recurring retrieval costs before resorting to expensive model upgrades.