GraphRAG Pipelines

Lesson Learned: Checkpointing and Idempotency in GraphRAG Pipelines with Neo4j

Date: 2026-08-26 Area: Software Architecture / GraphRAG / Neo4j Impact: Avoid costly reprocessing (tokens/time) after interruptions in graph ingestion.

---

1. The Problem: Data Persistence ≠ State Persistence

In graph knowledge indexing pipelines (GraphRAG), it's very common to confuse two fundamental concepts:

1. Saving the data (Nodes and Relationships): Neo4j persists data to disk using MERGE queries for each batch. If the process is interrupted, the data generated up to that point is not lost.
2. Saving the checkpoint (Checkpoint): Knowing the last processed chunk or batch.

The symptom of carelessness

If you stop the indexing process today (e.g., at 86%), when you restart the script, it will start from chunk 0. Although the old data is not duplicated thanks to MERGE, the pipeline will still call the LLM to extract entities and relationships from all the previous text, wasting time and API tokens.

---

2. The Solution Pattern: Idempotent Resume Pattern

For a pipeline to be resilient to interruptions, it must follow an atomic check-in strategy.

       ┌────────────────────────┐
       │     Next Chunk      │
       └───────────┬────────────┘
                   │
                   ▼
      `book_id + chunk_index`
       has status: PROCESSED?
        ┌──────────┴──────────┐
        YES │                     │ NO
        ▼                     ▼
┌──────────────┐     ┌────────────────┐
│ Skip Chunk│     │ Send to LLM  │
│ (0 cost LLM)│     │ (Extraction)  │
└──────────────┘     └───────┬────────┘
                             │
                             ▼
                     ┌────────────────┐
                     │  Transaction   │
                     │   Atomic on    │
                     │     Neo4j      │
                     └────────────────┘

---

3. Key Design Principles

A. Unique Identifier (Positional > Hash)

• Rule: Use a deterministic and natural key composed of book_id + chunk_index.
• Why not use hash(content): Two distinct or repeated sections within the same document can have identical text and result in a collision. The structural position is superior to the content hash.

B. Status Marker (status: PROCESSED)

• It's not enough to simply query if a Chunk node exists (MATCH (c:Chunk)). A crash in the middle of a batch process could leave the node created but its entities/relationships incomplete.
• An explicit status attribute is required to distinguish between complete and incomplete chunks.

C. Atomic Transactionality

To guarantee the integrity of the checkpoint, all operations within a batch must be executed within the same database transaction (session.execute_write):

1. Create or update the structure of the Book and Chunks.
2. Perform the MERGE of Entities and Relationships.
3. Mark the Chunk with status: 'PROCESSED'.
4. Commit the transaction.

Note: If any step within the batch fails, the transaction performs an automatic rollback, and the chunk remains unmarked, allowing a clean retry upon restart.

---

4. Anti-Patterns and Best Practices for Engineering

• ❌ Avoid Over-Engineering: No need for heavy orchestrators like Airflow, Prefect, or Temporal.io for local or single-node pipelines (e.g., scripts on an OrangePi). The in-process checkpoint using Cypher queries to Neo4j is sufficient, lightweight, and highly efficient.
• ❌ Avoid Disjoint Cypher Queries: Do not stream data into multiple independent commits within the same batch (book → nodes → relationships). Everything should be packaged in a single atomic block.

---

5. Checklist for Future Graph Indexing

• Define a natural and unique key for each text fragment (doc_id + chunk_idx).
• Create an index on Neo4j for the Chunk key, to ensure instant verification.
• Implement the query: MATCH (c:Chunk {book_id: $b, chunk_index: $i, status: 'PROCESSED'}) RETURN c.
• Enclose the LLM extraction and persistence within an idempotent flow with status: 'PROCESSED'.