When you have several vector stores, how do you send a query to the right one?

The first multi-domain RAG system feels easy. You have product docs, legal docs, and support tickets, each in its own vector store, and a query comes in. Which store do you search?

Most teams reach for one of two answers, and both quietly fail.

The two tempting wrong answers

Merge everything into one store. Now every query searches one giant index. The problem is the signal-to-noise ratio of top-k retrieval: a legal query pulls back the three best legal chunks and several unrelated product chunks that happened to score well, all competing for the same slots in the context window. Precision collapses precisely because you removed the domain boundary that was doing useful work.

Hard-code the routing. An if/else on query keywords works in the demo and breaks the first time a user paraphrases, a synonym appears, or someone renames a domain. Routing logic keyed on surface tokens cannot keep up with how people actually ask questions.

There is also a third non-answer — fan out to every store on every request — which multiplies latency and floods the model with irrelevant context from the wrong domains.

The retriever-tool pattern

The fix is to make routing a decision the model makes, not a rule you maintain. Wrap each vector store as a named retriever tool with a one-line description of what it contains. Hand those tools to an LLM router. The model reads the query, matches it against the tool descriptions, and calls the right retriever — the same tool-calling machinery you already use for everything else.

The descriptions become the routing logic. "Product documentation for the X platform" and "Internal legal contracts and policies" are enough for the model to route reliably, and they survive paraphrase because the model is matching on meaning, not keywords.

LLM router vs. classifier router

You do not always need an LLM in the routing seat. A small trained classifier is cheaper and faster per query, but it needs labeled data and a retrain whenever domains change. An LLM router needs no training and adapts the moment you edit a tool description — at the cost of an extra model call. The right choice is a cost-vs-flexibility tradeoff, not a default.

If your domains are stable and traffic is high, a classifier amortizes well. If domains shift often or you are still discovering them, the LLM router keeps you moving.