Your RAG answers are wrong. Is it the model — or your chunking?

When a RAG chatbot gives a confidently wrong answer, the instinct is to swap in a smarter model. That instinct is usually wrong. In a baseline single-hop pipeline, the model is rarely the bottleneck — the ingestion stage is.

The three-stage pipeline, and where it breaks

A traditional RAG pipeline has three stages: ingest and chunk your documents, embed and store them, then retrieve and generate. The failures cluster at the edges:

• Ungrounded hallucination — a domain query reaches a model that is epistemically blind to the answer, so it fills the gap with fiction.
• No semantic search without vectorization — raw text stored without embeddings can only be keyword-matched, which catches surface tokens, not intent.
• Garbage chunks, garbage answers — the boundaries you cut documents on decide what can ever be retrieved.

Why chunking is the lever

Retrieval can only return what your chunks contain. If the answer to a question spans the bottom of chunk 7 and the top of chunk 8, neither chunk scores well on its own and the answer is effectively unretrievable — no matter how good your embeddings or your model are. Cut chunks too large and the real answer is diluted by surrounding noise, dropping its similarity score below the cutoff.

This is why most "RAG doesn't work" reports are chunking problems wearing a model costume. The query embeds fine, the vector store is healthy, the model is capable — but the chunk that should have matched was split, or buried, at ingestion time.

What to do first

Before touching the model, audit the chunks the retriever actually returns for a failing query. Are the boundaries semantic (sections, complete thoughts) or arbitrary (every N characters)? Does any single chunk contain a complete, self-sufficient answer? Fixing chunk boundaries fixes a class of "wrong answer" bugs that no model upgrade will touch.