Loreto
Home/Skills/Diagnosing RAG Failure Modes
AI Architecture

Diagnosing RAG Failure Modes

Classifies failing RAG queries and prescribes the architecture fix.

Quick answer

Diagnoses why a RAG pipeline is returning poor results by classifying the failing query as factual-lookup-safe or relational/temporal, then identifies which of four failure patterns it hits — multi-hop, temporal sequencing, organizational context, or scale — and prescribes the architecture remediation.

What this Claude Code skill does

Diagnoses RAG system failures by classifying queries as factual-lookup-safe vs. relational-temporal (where RAG breaks). Use when a RAG pipeline is returning poor results, an agent fails on multi-hop or causal queries, or when a team asks why their retrieval system cannot answer questions about decision histories, event sequences, or organizational causation chains.

Diagnosing RAG Failure Modes is a packaged skill for Claude Code: a SKILL.md plus references and a runnable test that drops into .claude/skills/, so your agent can run this workflow reliably instead of you re-explaining it each time. Install it via MCP, run it as a CLI, or fork it into your own.

Inside the SKILL.md

<h1>Diagnosing RAG Failure Modes</h1>
<h2>The Two Query Classes</h2>
<p>Every query against an organizational knowledge base falls into one of two classes:</p>
<p><strong>Class A — Factual Lookup (RAG-safe)</strong>
- "What is our data retention policy?"
- "What version of X is deployed in production?"
- Point queries. Single-hop. The answer lives in one document.</p>
<p><strong>Class B — Relational/Temporal (RAG danger zone)</strong>
- "What decisions led to the current architecture?"
- "Why did we choose vendor X over vendor Y?"
- "What sequence of events caused the Q3 outage?"
- Multi-hop. Requires traversing relationships across documents and time.</p>
<p>RAG is designed for Class A. When applied to Class B, it retrieves facts but loses the connective tissue between them — relationships, causation, sequence.</p>
<pre class="mermaid">flowchart LR
    Q[Query] --&gt; SCORE{Run diagnostic\nchecklist\n5 questions}

    SCORE --&gt;|0 checks| A[&quot;Class A — Factual Lookup\nRAG works\nCheck embeddings + chunking if failing&quot;]
    SCORE --&gt;|&quot;1–2 checks&quot;| B[&quot;Borderline\nRAG + metadata filtering\n+ reranking may work&quot;]
    SCORE --&gt;|&quot;3–5 checks&quot;| C[&quot;Class B — Relational / Temporal\nRAG will structurally fail\nRequires hybrid architecture&quot;]

    C --&gt; P1[&quot;Pattern 1\nMulti-Hop\n→ Knowledge Graph&quot;]
    C --&gt; P2[&quot;Pattern 2\nTemporal Sequence\n→ Timeline Index&quot;]
…

Frequently asked questions

When should I use diagnosing-rag-failure-modes?
Use it when a RAG pipeline returns poor results, an agent fails on multi-hop or causal queries, or a team asks why retrieval cannot answer questions about decision histories or causation chains.
What is the difference between Class A and Class B queries?
Class A queries are factual lookups answered by a single document — RAG handles them well. Class B queries require traversing relationships across documents and time (multi-hop, causal, sequential) — RAG structurally fails on these.
What are the four RAG failure patterns this skill identifies?
Multi-hop relational failure (fix: knowledge graph), temporal sequencing failure (fix: timeline index), organizational context failure (fix: structured provenance ingestion), and scale failure (fix: tiered retrieval).
When should I NOT use this skill?
Skip it for simple single-document factual lookups, when you already know the failure pattern and need architecture design (use designing-hybrid-context-layers), or when the issue is model reasoning quality rather than retrieval (use auditing-intelligence-context-fit).

Related skills

AI Architecture

Designing Hybrid Context Layers

Designs hybrid AI context architectures that combine RAG, knowledge graphs, episodic memory, and long-context synthesis…

Knowledge Systems

Synthesizing Institutional Knowledge

Builds organizational memory systems that capture decision provenance, causal chains, and institutional context beyond…

AI Reasoning

Temporal Reasoning Sleuth

Engineers temporal reasoning capabilities for AI agents — enabling them to trace decision chains, reconstruct causal se…

Build a skill like this from your own source

Paste a YouTube link, an article, a PDF — or describe your workflow. Loreto turns it into a production-ready Claude Code skill in minutes.

Generate a skill freeBrowse the marketplace