Should the agent decide what to retrieve, or should the pipeline?

There is a specific moment when a retrieval pipeline stops being adequate, and it is easy to miss because nothing errors.

Someone asks: "Which of our customers churned after we raised prices, and what did they say on the way out?" The pipeline embeds that sentence, fetches the five nearest chunks, and hands them to the model. The chunks are all about pricing. None of them is about churn, because churn and pricing are discussed in different documents by different teams. The model writes a fluent paragraph about the price change and says nothing about who left.

No stack trace. No empty result. A confident answer to a question nobody asked.

Why one lookup cannot get there

A fixed pipeline does exactly one thing: embed the question, fetch what is near it, answer. That works when the answer lives in one place and the question resembles the text that contains it.

The question above needs two hops. First: who churned after the price change — that is a filter over records. Second: what did those specific people say — that is a lookup keyed on the result of the first hop. There is no single embedding that is near both halves, because the second half cannot be written down until the first half has been answered.

You can pile on chunks and it will not help. Twenty chunks about pricing are still twenty chunks about pricing. This is the pattern worth internalising: when a question's second half depends on the answer to its first half, no amount of retrieval breadth substitutes for a second trip.

What an agentic loop actually changes

Handing retrieval to an agent changes one thing: the decision about what to fetch moves from build time to run time. Instead of a fixed query, the agent gets tools — search, filter, fetch-by-id — and decides in what order to use them, using what it learned from the last call.

On the churn question that looks like: filter the customer table by cancellation date, take the resulting names, search the support corpus for each one, then answer. Three retrievals, chosen in sequence, none of which could have been written in advance.

The cost is real and worth stating plainly. Every hop is a model call. A question a fixed pipeline answers for one call and 300ms becomes four calls and several seconds. Run that on every query, including "what is our refund window", and you have made a fast system slow to fix a problem most questions do not have.

Route by query class

The version that survives production is neither pipeline nor agent. It is a classifier in front of both.

• Single-hop factual — "what is our refund window" — fixed path. One lookup, one answer.
• Relational or comparative — "which customers did X and then said Y" — agent path.
• Temporal or causal — "what led us to decide Z" — agent path, and often a different store entirely.

Classification is itself a model call, but a cheap one against a small model, and it converts an expensive default into an exceptional case. On most corpora the split lands somewhere near eighty–twenty, which means the agent path runs on a fifth of traffic and the average query stays close to fixed-pipeline cost.

The stop condition is the hard part

An agent given retrieval tools and no budget will use them. It will fetch, consider, fetch again, and convince itself that one more document would settle the matter. This is the failure mode that turns a four-second query into a forty-second one, and it appears under load rather than in testing.

Two guards, both necessary:

• A hop budget. Hard cap. When it is reached, answer with what is in hand and say what is missing.
• A sufficiency check after each hop. Explicitly ask whether the question can now be answered. An agent that is not asked will not volunteer that it already has enough.

Say what is missing rather than guessing. An agent that reports "I found who churned but no exit interviews for two of them" is more useful than one that fills the gap with plausible sentences — and it tells you which part of the corpus to fix.

How to tell which you need

Take fifty real queries from your logs — real ones, not the ones you wrote to demo the system. For each, ask whether answering it requires knowing the result of an earlier lookup. If almost none do, an agentic loop will cost you money and latency to solve a problem you do not have. If a meaningful slice do, those are the queries your users are already quietly getting wrong answers to.