Making Document Indexing Idempotent & Incremental

Deterministic path resolution, node versioning, and deferred regeneration in PageIndex

The Demo vs The Pipeline

In a demo, document indexing runs once. You load a clean markdown file, parse it into chunks, build an index, and run a few queries. Everything matches.

In production, documents are never static. People edit paragraphs. CI jobs re-index folders on merge. Webhooks fire twice when an ack drops. A Celery worker gets killed by OOM and retries the batch 10 seconds later.

PageIndex (VectifyAI, 34k+ stars) is an open-source framework that parses documents into hierarchical trees instead of slicing them into arbitrary 500-token chunks. Headings become parent nodes, sections become branches, and tables stay attached to their column headers.

It works much better than flat chunking for complex documents. But when you hook it up to an automated sync job, you hit two real problems:

1. Non-idempotent ingestion (PR #314): Every run minted a fresh UUID, creating duplicate JSON trees on disk and leaving orphaned nodes in vector storage.
2. Full re-index token waste (PR #383): Editing a single line in a 100-page document forced the parser to re-run LLM summarization on all 80 nodes.

I opened PR #314 and PR #383 in VectifyAI/PageIndex to fix both issues. Here is what broke and how we resolved it.

1. The Non-Idempotent Ingestion Bug (PR #314)

Here is the line of code that caused the first issue:

# pageindex/indexer.py (before PR #314)
def index_document(self, file_path: str, metadata: dict = None) -> DocumentTree:
    # A fresh UUID was generated on every call
    doc_id = str(uuid.uuid4())
    
    tree = self.parser.parse_to_tree(file_path)
    tree.doc_id = doc_id
    
    self.storage.save_tree(doc_id, tree)
    self.vector_store.upsert_nodes(tree.get_nodes())
    return tree

Because doc_id was a random UUID, the indexer had no memory of previous runs for that file.

Here is what happened when a nightly cron job re-ingested a folder of 1,000 files:

• Duplicate JSON files on disk: The storage directory grew by 1,000 new JSON files on every run. After 5 syncs, there were 5,000 files for 1,000 documents.
• Orphaned summary nodes: The old child nodes and vector embeddings stayed in the database. Their pointers were severed, but the vectors remained active.
• Split-brain search results: When an agent queried the index, retrieval returned two or three copies of the same section, sometimes mixing old deprecated text with the updated file.
• No query method for existing files: Callers had no get_doc_id_by_path() method to check if a path had already been indexed.

The Fix: Canonical Path Keys and In-Place Overwrites

# pageindex/indexer.py (with PR #314)
def index_document(
    self, 
    file_path: str, 
    doc_id: Optional[str] = None,
    overwrite: bool = True
) -> DocumentTree:
    norm_path = self._normalize_path(file_path)
    existing_id = self.get_doc_id_by_path(norm_path)
    
    if existing_id:
        if not overwrite:
            return self.storage.load_tree(existing_id)
        # Reuse existing ID and purge old subtrees
        target_id = existing_id
        self._purge_document_state(target_id)
    else:
        # First-time indexing: assign stable UUID
        target_id = doc_id or str(uuid.uuid4())
        self._path_to_doc_id[norm_path] = target_id

    tree = self.parser.parse_to_tree(norm_path)
    tree.doc_id = target_id
    self.storage.save_tree(target_id, tree)
    self.vector_store.upsert_nodes(tree.get_nodes())
    return tree

2. Node-Level Versioning & Deferred Regeneration (PR #383)

PR #314 fixed identity and storage leaks. But once pipelines started updating documents in place, a second problem showed up: token waste.

Suppose an engineer fixes a typo or updates a single rate limit in Section 5 of an API guide. The rest of the 50-page document is completely unchanged.

In the original codebase, updating that file re-ran LLM summarization calls for every single node in the tree. You were paying full ingestion cost for a one-line change.

How PR #383 Solves This

# PR #383: Node diffing and selective summary regeneration
def incremental_update_tree(old_tree: DocumentTree, new_ast: DocumentTree) -> DocumentTree:
    dirty_nodes = []
    
    for new_node in new_ast.traverse_all():
        old_node = old_tree.get_node_by_path(new_node.path)
        
        # Hash match: reuse existing summary and embedding
        if old_node and old_node.content_hash == new_node.content_hash:
            new_node.summary = old_node.summary
            new_node.version = old_node.version
            new_node.is_dirty = False
        else:
            # Changed node: increment version and mark dirty
            new_node.version = (old_node.version + 1) if old_node else 1
            new_node.is_dirty = True
            dirty_nodes.append(new_node)
            
    # Regenerate summaries ONLY for dirty nodes and their ancestors
    regenerate_dirty_subtrees(new_ast, dirty_nodes)
    return new_ast

On a benchmark of 50 technical documents undergoing typical edits, this reduced LLM summarization tokens from ~120,000 tokens down to ~9,500 tokens per update—a 92% reduction in re-indexing costs.

Three Invariants for Production Pipelines

These fixes reflect three general rules for building reliable retrieval pipelines:

Business-derived keys, safe retries, and structured error reporting

• 1. Derive keys from the entity, not the transport. In document indexing, that is the canonical file path and content hash. In webhook handlers, it is sha256("order:<id>:v<version>"). Redeliveries must collapse to the exact same key.
• 2. Retries require idempotency. If a worker uses automatic retries, calling a non-idempotent endpoint duplicate-creates records every time a timeout occurs.
• 3. Log structured entities on failure. A failure log should include {"doc_id": "...", "node_path": "...", "attempt": 2} rather than a generic error message.

Why Tree Retrieval Needs Clean State

In a flat vector database, duplicate chunks slightly hurt ranking. In a tree-based retrieval engine like PageIndex, duplicate nodes break search completely.

Flat BM25 retrieval vs structure-aware hierarchical traversal

PageIndex searches documents by evaluating parent section summaries first (Breadth-First Search). It only descends into a branch if the parent summary matches the query.

If orphaned summary trees remain from previous runs, the search traversal wastes tokens expanding dead branches and retrieves outdated procedures.

Summary Table

Closing Thought

Most discussion around RAG focuses on prompt tuning and embedding models. But in production systems, retrieval usually breaks first at the state layer.

If ingestion is not idempotent, your storage leaks memory and returns stale context. If you cannot update nodes incrementally, you burn your token budget re-indexing text that never changed.

Fixing these issues means treating retrieval like a systems problem: canonical keys, atomic overwrites, and content-addressed node caching.