Why does RAG keep failing on questions about your team's history?

The first time it happens, you think it's a one-off.

Your VP of Engineering asks the new RAG system: "What chain of decisions led us to standardize on Postgres over MySQL?" The system fetches three relevant documents — the original RFC, the migration retro, the cost analysis — and returns a confident, plausible-sounding answer.

It's wrong.

The actual decision wasn't in any of those three docs. It was made over Slack DMs in October 2023, ratified in a one-line comment on the RFC, and only retroactively justified by the cost analysis. The retrieval system found three things about Postgres-vs-MySQL. None of them captured the chain.

What is the difference between Class A and Class B queries?

Every query against an organizational knowledge base falls into one of two classes. The taxonomy is simple but it predicts almost everything about whether RAG will work.

Class A — Factual lookup. "What does our SLA say about uptime?" "Which version of X is in production?" "Who owns the billing service?" These are point queries. The answer lives in one document. Standard RAG handles them well.

Class B — Relational or temporal. "What sequence of decisions led to our current architecture?" "Why did we choose vendor X over Y?" "What caused the Q3 outage?" These require traversing relationships across documents and time. RAG was not designed for this.

The original RAG paper (Lewis et al, 2020) framed retrieval as a way to ground an LLM's generation in factual content. That's exactly Class A. The architecture works because the answer to a factual lookup is a chunk-shaped object: it has a beginning, an end, and lives near other semantically similar text.

Decision history is not chunk-shaped. It's a graph with timestamps.

Why don't reranking and bigger context windows fix it?

The default reaction to RAG failure is to add tricks. Better embeddings. Cross-encoder reranking. HyDE. A two-million-token context. We've watched teams burn six engineering weeks on these and end up with eval scores that are flat.

The reason is that none of them change the shape of what's being retrieved. Reranking sorts the same chunks differently. HyDE produces a hypothetical document and embeds that — but the document still has to exist in the corpus. Long context lets the model see more chunks, not connect them.

What's missing isn't more chunks. It's the connective tissue between them — the relationships, the causation, the sequence.

What actually solves Class B failures?

The fix is structural, not parametric. You stop asking the embedding space to solve a graph problem and you give it a graph.

For multi-hop queries: a knowledge graph with entity relationships. Microsoft's GraphRAG paper (Edge et al, 2024) showed concrete improvements on questions that span multiple documents — the kind RAG quietly fails on.

For temporal sequencing: a timeline or episodic index. Event nodes with timestamps, causal predecessors, and links to source documents. The model retrieves a slice of the timeline, not a list of unrelated chunks.

For organizational provenance: structured ingestion that captures who decided this, when, why, and what alternatives were considered. Not as prose to be embedded, but as fields to be queried.

How do you tell which kind of failure you're hitting?

There's a five-question diagnostic checklist. Run it on the failing query:

1. Does the answer require joining facts from more than one document?
2. Does the answer require knowing the order in which events occurred?
3. Does the answer require understanding why a decision was made, not just what?
4. Does the answer span a time period longer than a single document's scope?
5. Does the answer require following a causal chain (A caused B, which led to C)?

Three or more "yes" answers and you're in Class B. Adding chunks won't help. The architecture is the bottleneck.

This diagnostic is the heart of the Diagnosing RAG Failure Modes skill — a Claude Code skill that runs the classification, identifies which of four failure patterns the query hits (multi-hop, temporal, organizational context, scale), and prescribes the architecture fix.

The VP from the opening of this post got an answer eventually. It came from the engineer who'd been on the original Slack thread, not the system. That's the institutional memory problem worth solving — and it's worth solving with the right architecture, not a bigger embedding model.