Agentic RAG: choose your retrieval effort
Amazon Bedrock Managed Knowledge Base now lets the same API return passages only, answer after one search, or plan iterative searches. This guide turns that control into a measurable routing, evidence, and evaluation policy for teams operating enterprise RAG.
- Published
- min read
- 7 min read
- Categoría
- AI Systems
On this page
7 chapters- 01When does a question need agentic RAG?
- 02The mechanism: three outputs, not three products
- 03How do you route a query without asking the model to judge itself?
- 04Hypothetical example: procurement policies with three paths
- 05How do you evaluate quality, latency, and cost together?
- 06The limits: more searches do not repair a broken corpus
- 07The decision: buy evidence, not iterations

Chapter 01
When does a question need agentic RAG?
A question needs agentic retrieval when its answer depends on gathering and checking distributed evidence, not merely finding a similar passage.
On November 9, 2026, one calendar day before this publication in America/Lima, AWS announced three behaviors for AgenticRetrieveStream in Amazon Bedrock Managed Knowledge Base. With maxAgentIteration set to 0, it returns passages without generating an answer; with 1, it performs one retrieval and then answers; with values from 2 through 5, it keeps the agentic flow that plans and refines searches. The right decision is not to enable the deepest mode everywhere. Use passages only when a person or application needs to inspect the source; one search for direct questions whose evidence lives in a well-defined block; and iterations when the task must compare entities, reconcile documents, or cover several parts of a question. Choose the lowest mode that clears your own threshold for answer correctness and sufficient evidence. Latency and model usage then become measurable consequences of documentary difficulty rather than a fixed tax on all traffic.
AWS explains that one iteration equals one model call and that the final one is reserved for answer generation. A limit of n therefore allows at most n minus 1 planning rounds. The initial speculative retrieval does not consume that count. This is a useful operating contract because it turns a vague notion of depth into an observable budget, but it does not classify questions for you or guarantee that more rounds will find better sources. The application remains responsible for selecting the mode, controlling which corpus each identity may query, and deciding when an answer lacks sufficient support.
Chapter 02
The mechanism: three outputs, not three products
The architectural advantage is preserving one input contract while varying the amount of work. Mode 0 performs guaranteed retrieval and returns deduplicated passages; it invokes no generative model and generateResponse must be false. Mode 1 retrieves once and uses one model call to write a grounded answer. Modes 2 through 5 let a planner decompose the question, reformulate searches, evaluate gaps, and finally synthesize. Every mode can query multiple knowledge bases: single-shot describes the absence of iterative planning, not a restriction to one source.
Treat the value as a maximum, not a quality promise. A complex question can stop with incomplete evidence; a direct one can receive too many irrelevant passages when the index or metadata is poor. Effort control acts only after ingestion, chunking, embeddings, filters, and permissions. If those layers fail, more planning can produce a more convincing explanation of the wrong corpus. Routing should record the query, selected mode, corpus version, filters, actual iterations, retrieved sources, and evaluator outcome.
| Mode | System work | Appropriate use | Primary risk |
|---|---|---|---|
| 0 · Passages | One retrieval, no generated answer | Human review or external composition | Relevant but insufficient source |
| 1 · Single-shot | One retrieval and one answer | Direct, well-bounded lookup | One search misses other evidence |
| 2–5 · Agentic | Planning, searches, and synthesis | Comparison and multi-part questions | More work without better support |
Chapter 03
How do you route a query without asking the model to judge itself?
Start with observable rules. An explicit request to see original text goes to mode 0. An exact identifier, one specific policy, and an answer located in a current document can start at 1. Entity comparisons, questions with several conditions, complete summaries, and queries that must reconcile dates begin in a bounded agentic mode. Do not rely only on a semantic classifier: combine workflow type, expected evidence shape, entity count, time span, and coverage requirement. Classification may propose; a versioned policy decides.
Then define escalation. If single-shot fails to retrieve a sufficiently authoritative source, misses expected subtopic coverage, or detects conflicting versions, it can be reissued once with greater effort. Escalation should preserve a request ID and record why it happened. Avoid an automatic loop where the answer grades itself and buys more searches until it sounds confident. Use external signals: effective-date metadata, required-field coverage, citation presence, an entailment verifier, or human review for high-impact domains.
Chapter 04
Hypothetical example: procurement policies with three paths
Imagine a hypothetical internal procurement assistant. The question “show me the current text on advances” requests primary evidence and uses mode 0; the interface presents the passage, document, version, and date without drafting a new rule. “What approval does an advance below S/ 5,000 require?” can use mode 1 when the amount, currency, and authority appear together in one current policy. Before displaying the answer, the application verifies that the citation contains the threshold and approving role, and that the country filter matches the user's identity.
“Compare controls for contractor advances in Peru and Colombia, include exceptions, and flag conflicts between the global policy and local addenda” needs decomposition. Agentic mode searches by country, control, and exception; it keeps sources separate and synthesizes only afterward. If the Colombian addendum is missing or two versions share the same effective date, the correct result is incomplete or requires review, not a smooth comparison. No retrieval path authorizes a payment: retrieving policy and executing a transaction are separate permissions. The assistant may explain requirements; the transaction system retains approval, separation of duties, and idempotency.
Chapter 05
How do you evaluate quality, latency, and cost together?
Build a stratified set with direct, multi-part, ambiguous, out-of-corpus, and adversarial queries. For each one, store the expected class, authorized sources, required facts, and an acceptable answer or abstention state. Run the same cases in modes 0, 1, and at least one agentic limit while keeping corpus, filters, and model constant. Evaluate retrieval before generation: source coverage, passage precision, correct version, and absence of forbidden documents. Then measure support for each claim, completeness, contradictions, and abstention quality.
Record time to first event, total latency, model calls, actual iterations, tokens, searches, expanded documents, and human-review minutes. Calculate cost per accepted answer rather than per request: include escalations, retries, and corrections. The comparison should produce decision boundaries, for example that a direct class uses 1 while it meets support and p95 targets, while a comparison class uses 3 because 2 misses one required source. Do not publish those numbers as universal; they depend on the corpus, Region, model, filters, and query pattern.
Chapter 06
The limits: more searches do not repair a broken corpus
The AWS announcement provides no benchmark proving a universal improvement and does not quantify cost or latency savings. Its banking assistant is illustrative. We also did not test the service in this publication. The documented advantage is explicit control over work, not guaranteed accuracy. If documents are missing, chunking separates conditions from exceptions, effective-date metadata is wrong, or a user can retrieve an unauthorized collection, no amount of planning fixes the foundation. The first investigation of a failure should distinguish ingestion, retrieval, reranking, planning, generation, and citation rendering.
The contract also has limits. Mode 0 returns passages in a stream format that does not necessarily include the same fields as Retrieve; generateResponse true with zero iterations produces a 4xx error. The observability documentation exposes invocations, errors, throttles, and TotalIterationCount, but that metric does not prove correctness. Keep application traces that link every answer to sources and routing policy, remove sensitive content from logs, and test that a configuration change does not broaden sources or permissions. A green dashboard can show availability while the system cites the wrong policy.
Chapter 07
The decision: buy evidence, not iterations
Adopt effort control when you can name query classes, define sufficient evidence, and observe the full cost of each path. Start with a conservative default: mode 1 for well-bounded lookups, 0 for source review, and an agentic limit only for classes that demonstrate better coverage. Add an explicit not-found outcome and one escalation based on external signals. Review the policy whenever the corpus, embeddings, reranker, model, or instructions change because each modification can move the boundary between direct and iterative search.
The useful change is not calling every RAG system agentic, but paying for depth when the structure of the question requires it and skipping it when it adds no evidence. A serious pilot leaves a reproducible matrix of queries, modes, sources, acceptance, latency, and cost. If you need to turn real documents and permissions into that pilot, Wasyra's custom service can design the architecture, evaluation set, and deployment gates. It is an implementation service distinct from the Agents product hosted at agents.wasyra.com.
Written by
Wasyra Lab
AI systems and operations architecture
Wasyra Lab publishes practical frameworks for designing AI agents, automations, and operating flows that survive production.
Series
AI systems that actually reach production
A series on agents, copilots, and guardrails for bringing AI into real work without breaking trust or operations.
Posts in this seriesMore from this author
More from this author
AI Systems
Agent hooks: 7 tests before tools execute
AgentCore's new lifecycle hooks expose four control boundaries. This guide turns them into seven tests for authorization, failure, and auditability.
ArticleAI Systems
Scientific post-training: 7 evidence gates
Nemotron for Science opens scientific post-training. These seven gates prevent proxy optimization, test leakage, and false improvements.
ArticleKeep reading
Keep reading
AI Systems
Agent hooks: 7 tests before tools execute
AgentCore's new lifecycle hooks expose four control boundaries. This guide turns them into seven tests for authorization, failure, and auditability.
ArticleAI Systems
Scientific post-training: 7 evidence gates
Nemotron for Science opens scientific post-training. These seven gates prevent proxy optimization, test leakage, and false improvements.
ArticleAI Systems
Agentic RL: how to read MLPerf Training 6.1
MLPerf now measures agent post-training. Learn how to read time to quality, workload limits, and useful infrastructure decision signals.
Article