Why Naive RAG Fails on Financial Documents

The Problem: VCs Are Flying Blind

A VC firm sits across from dozens of startups every month. Each one walks in with a pitch deck: polished, confident, and full of big numbers. Revenue growing 3x. Market size $50B. Margins improving every quarter.

Behind those numbers are hundreds of pages of actual financial records: SEC filings, data room spreadsheets, audit reports. The pitch deck and those underlying documents should tell the exact same story. A lot of the time, they do not.

Sometimes it is a mistake. Sometimes it is intentional. Either way, the VC does not have time to manually cross-reference every claim against 200 pages of filings. Nobody reads all of it. They make a call on the deck.

I built ClaimCheck to do the cross-referencing automatically. Feed it the pitch deck and the company's financial documents. It surfaces every conflict, every verified claim, and every claim that has zero evidence anywhere in the filings.

The first thing I reached for was standard RAG. It failed quickly. Here is why, and what I built instead.

Why Naive RAG Falls Apart Here

Standard RAG is straightforward: split documents into chunks, embed them as vectors, then at query time retrieve the most similar chunks and hand them to the LLM.

For general Q&A this works fine. Financial documents are a different story. There are five specific ways naive RAG breaks down.

Problem 1: Table Structure Gets Destroyed

Financial documents are full of tables. When you chunk a table as flat text, numbers get separated from the labels that make them meaningful.

A financial table before and after naive RAG chunking

A table row like Revenue | Q3 2024 | $45M becomes just $45M floating in a text blob with no row label, no column header, and no time period. The LLM sees a number and has no idea what it refers to. It will still answer confidently. That is the real hazard.

Problem 2: Footnotes Get Separated from Their Tables

Financial filings rely heavily on footnotes. A number in a table will have an asterisk pointing to something two pages later that completely changes its meaning:

Revenue: $45M *

* Excludes one-time acquisition costs of $18M.
  Reported GAAP revenue was $27M.

Naive chunking splits the table and the footnote into separate chunks. The model sees $45M and reports it without ever reading the asterisk that invalidates it.

Problem 3: Temporal Confusion

A single filing has Q1, Q2, Q3, Q4, often across multiple years. A retrieved chunk about revenue might not carry enough surrounding context for the model to identify which quarter it came from. It guesses.

Problem 4: Math Claims Need Verification, Not Just Retrieval

A pitch deck says "Revenue grew 3x year over year." Retrieval finds: Revenue 2023: $12M and Revenue 2024: $28M. Is that 3x? No, it is 2.3x. Naive RAG just retrieves and passes to the LLM without calculation. A wrong math claim slips through unchallenged.

Problem 5: Terminology Mismatch

The deck says "ARR." The filing says "Annual Recurring Revenue" or sometimes just "Revenue." The keywords differ enough that retrieval misses the match entirely and reports the claim as not found, even though the evidence was always there.

The Fix: Treat Documents Like Code

When a compiler reads source code, it does not split it into 500-character chunks and hope for the best. It parses the code into an Abstract Syntax Tree (AST), a structured representation that keeps every relationship intact: what belongs to what, what is a parent, what is a child.

I applied the exact same concept to financial documents.

Document parsed into a tree — every number stays with its context

Step 1: Structure-Aware Parsing

Instead of splitting by character count, the parser reads the document's actual structure. Headings become parent nodes. Sections become branches. Table rows become leaf nodes. Footnotes attach to their parent table rather than floating somewhere else in the chunk stream.

A row like Revenue | Q3 2024 | $45M stays as one node. The label, the quarter, and the number never get separated.

Step 2: Summarize Every Node

Each node in the tree gets a short LLM-generated summary of what it contains. Instead of reasoning over dense raw filing text, the model compares a pitch deck claim against a clean 2 to 3 sentence summary. Smaller comparison text means sharper reasoning and higher accuracy.

Step 3: Traverse Every Node With Every Claim

Here is the biggest departure from standard RAG: we do not retrieve at all. We walk the entire document tree, node by node. For every claim from the pitch deck, we ask the LLM to reason against that node's summary: Does this claim appear here? Does it conflict? Is it verified?

# Simplified traversal logic
for node in document_tree.traverse_all():
    for claim in pitch_deck_claims:
        verdict = llm.reason(
            claim=claim,
            node_summary=node.summary,
            node_context=node.path  # e.g. Section 4.2 > Revenue Table > Q3 row
        )
        if verdict.is_relevant:
            results.append(verdict)

No similarity threshold. No hoping the right chunk surfaced. Every node is checked. Nothing gets missed.

Citations Come Free

Citations solve themselves with this approach. Because traversal goes node by node, we always know exactly which node the LLM used to reach its verdict. The citation is a direct byproduct of the traversal path.

Traversal finds the conflict and records the exact node

No hallucinated citations. No vague "I found this somewhere in the document." Every verdict comes with a precise path through the document tree.

The "Not Found" Verdict

When a claim survives the full tree traversal without matching any node, ClaimCheck reports: "This claim has no evidence in any document."

This is its own important verdict and often the most damning one. A startup claiming something that does not appear anywhere in their own financial filings is a red flag about disclosure quality.

How It Compares

What's Next

Closing Thought

The problem with applying AI to financial document verification is not the model. Modern LLMs are capable of reasoning over financial data.

The problem is what you hand them. Give the model a number without its label, a table without its footnotes, or a chunk without its time period, and it will give you a confident wrong answer every time.

Structure-aware parsing fixes this at the source. Treat documents the way compilers treat code: preserve relationships, traverse deliberately, and let the model do what it is actually good at.