When your RAG finds nothing relevant, should it stay silent or fall back to the model?

A RAG pipeline that errors or returns nothing when the vector store has no relevant documents fails silently on exactly the questions where the user most needs a graceful answer — the out-of-domain ones.

Three failure modes show up:

• Empty response on low-relevance queries. When no chunk clears the threshold, the pipeline returns nothing rather than trying an LLM-only answer.
• Parametric knowledge never consulted. The model often already knows the answer from its training, but the pipeline abandons the question — discarding a correct answer it could have given for free.
• Silent irrelevant-chunk pass-through. With no relevance check, unrelated chunks flow into the prompt and the model produces a confident answer grounded in the wrong text.

The root cause

The bug is architectural: treating retrieval as an unconditional prefix to generation. Retrieve, then generate, always — even when retrieval found nothing worth generating from.

The relevance gate

Make generation conditional. Add a relevance check between retrieval and generation and a conditional branch on its result. Relevant context routes to grounded generation. Irrelevant or absent context routes to an LLM-only generation step that answers from the model's parametric knowledge — ideally flagged as a general answer rather than a sourced one.

This is the single-store, non-agentic version of the fallback problem. If your retrieval lives behind an agent that selects among multiple tools, the failure is ambiguous and the gate belongs at the agent's decision layer instead — see the agentic fallback pattern.