Python — Data Processing Pipelines

Data Pipeline Design for RAG Ingestion

Direct answer

A RAG ingestion pipeline moves documents through five stages — acquire, parse and normalize, chunk, embed, index — with content hashing at each step so unchanged documents are skipped and re-runs cost nothing. Design it as an incremental pipeline keyed on document and chunk hashes, version every vector by embedding model, and treat deletions as first-class so stale chunks can't keep surfacing in retrieval. Most retrieval quality problems I'm hired to fix trace back to ingestion, not the LLM.

When a RAG system gives bad answers, teams tune prompts and swap models, but in most of the systems I've audited the defect was upstream: broken parsing, careless chunking, or stale chunks that were never deleted. Here's the ingestion architecture that prevents those failures.

Key facts, with sources

JetBrains' State of Python 2025 survey of more than 30,000 developers found that 51% of all Python developers are involved in data exploration and processing, with pandas and NumPy the most commonly used tools. (The JetBrains Blog)
Apache Airflow reached more than 77,000 organizations using it as of November 2024, up from about 25,000 in 2020, with monthly downloads growing from 888,000 to over 31 million in the same period. (Astronomer State of Airflow 2025)
The official Apache Airflow 2025 survey collected more than 5,250 responses from 116 countries, and event-driven scheduling introduced in Airflow 3 already showed almost 25% adoption. (Apache Airflow Blog)
pandas has accumulated roughly 15.3 billion total downloads on PyPI, with around 700 million downloads in a single recent month. (pepy.tech)
The Rust-based Polars DataFrame library passed 24 million monthly downloads and over 250 million total downloads as of September 2025, five years after its first commit. (Wikipedia)
Gartner research estimates poor data quality costs organizations an average of at least $12.9 million per year. (Gartner)

The five stages, and where teams actually go wrong

Acquire pulls documents from sources. Parse converts PDFs, HTML, and office formats into clean text with structure preserved. Chunk splits text into retrievable units. Embed turns chunks into vectors. Index writes them to the vector store with metadata. Every RAG pipeline is some version of this, whatever framework wraps it.

The stage that decides quality is parsing, and it's the one teams skip past fastest. A PDF table flattened into word soup, headers and footers bleeding into body text, navigation chrome scraped along with an article — embed garbage and you retrieve garbage, and no reranker downstream rescues it. I spot-check parser output on the ugliest real documents before building anything else, because everything downstream inherits this stage's mistakes.

Incremental by construction: hash everything

Re-embedding an unchanged corpus on every run is the most common money leak in RAG systems. The fix is structural: hash the normalized document content, and skip any document whose hash matches what's already indexed. Within changed documents, hash each chunk too — often an edit touches one section, and only those chunks need re-embedding.

The chunk hash doubles as the vector store ID, which makes indexing idempotent: re-running a partially failed job upserts the same IDs instead of duplicating vectors. This is the same content-addressing idea that makes git cheap, applied to embeddings.

Content-addressed chunking

import hashlib
from dataclasses import dataclass


@dataclass(frozen=True)
class Chunk:
    doc_id: str
    chunk_id: str   # content hash: stable across re-runs
    text: str
    position: int


def chunk_document(
    doc_id: str, text: str, size: int = 1200, overlap: int = 200
) -> list[Chunk]:
    chunks = []
    start, position = 0, 0
    while start < len(text):
        piece = text[start : start + size]
        chunk_id = hashlib.sha256(piece.encode()).hexdigest()
        chunks.append(Chunk(doc_id, chunk_id, piece, position))
        start += size - overlap
        position += 1
    return chunks

Chunking is a product decision, not a constant

Fixed-size chunking with overlap, like the snippet above, is a fine baseline — but the right strategy depends on what your users ask and what your documents look like. Structure-aware splitting on headings keeps a policy clause or API section intact instead of severing it mid-sentence. Contracts, support tickets, and code each reward different boundaries.

Two practices pay off regardless of strategy. First, attach metadata to every chunk — source document, section title, position, timestamps — because retrieval filters and citation UX depend on it. Second, record the chunking parameters used, since changing size or overlap changes chunk hashes and effectively triggers a re-index; you want that to be a deliberate, versioned event rather than an accident.

Embedding versioning and the re-embed path

Vectors from different embedding models live in different spaces — you cannot query one model's index with another model's query vector and expect sane results. So every stored vector carries the embedding model identifier that produced it, and queries always embed with the model matching the index they hit.

This turns the inevitable model upgrade into a routine operation: build the new index by re-embedding in batches (chunk text is already stored, so no re-parsing), serve reads from the old index throughout, and cut traffic over once the new one is complete and spot-checked. Without the version column, upgrades become a guessing game about which vectors came from where — I've seen teams simply rebuild everything from scratch because they couldn't tell.

Deletes and updates: the stage everyone forgets

Documents get deleted, superseded, and rewritten, and a pipeline that only ever adds will happily keep retrieving a policy that was revoked months ago — usually the most embarrassing category of RAG failure, because the system cites the stale source with full confidence.

On each sync I diff the source's current document list against the indexed one: missing documents get their chunks deleted from the vector store, and changed documents get stale chunk IDs removed, not just new ones added — content-addressed IDs make the diff trivial since edited chunks produce new hashes. When hard deletes are risky, soft-delete via a metadata flag that retrieval filters out, then compact later. Either way, deletion handling is a launch requirement, not a backlog item.

Measuring ingestion health

Per-run stage counts are the baseline: documents seen, skipped as unchanged, parsed, chunks produced, embedded, indexed, deleted. A parse-failure spike or a sudden chunk-count jump flags an upstream format change long before users notice degraded answers.

Beyond counts, I keep a small fixed set of retrieval probes — a couple dozen real questions with known source documents — and run them after each significant ingestion change. It's not a full evaluation harness, just a smoke test asserting the right document lands in the top results. Cheap to build, and it has caught chunking regressions and parser breakage for me that stage counts alone never would.

When to hire senior help

Bring in a senior data engineer when pipeline failures start silently corrupting business metrics, or before committing to an orchestration and warehouse architecture, since storage layout and idempotency decisions are expensive to reverse once terabytes flow through them. A few weeks of experienced design work on schemas, retries, and backfill strategy routinely saves months of firefighting later. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships Python — Data Processing Pipelines projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

✕Running pipelines as cron jobs plus scripts with no idempotency or retry semantics, so a mid-run failure leaves half-written tables that corrupt downstream reports
✕Loading entire datasets into pandas in memory instead of chunking or using Polars or DuckDB, causing out-of-memory crashes the first time data volume grows 10x
✕Skipping schema and data-quality checks at ingestion, letting silent schema drift from a source API propagate wrong numbers into dashboards for weeks
✕Designing pipelines that cannot deterministically backfill historical data, making every bug fix or logic change a manual one-off reprocessing project

Frequently asked questions

What chunk size should I use for RAG ingestion?

There's no universal number — it depends on document structure and query style. A common starting point is roughly a few hundred to a thousand tokens with modest overlap, then adjust based on retrieval testing. Structure-aware chunking that respects headings and sections usually beats any fixed size. Treat chunk parameters as versioned configuration, because changing them means re-indexing.

How do I update a RAG index when documents change?

Hash document content and compare against what's indexed: unchanged documents are skipped, changed ones are re-chunked, and only chunks with new hashes get re-embedded. Critically, delete the stale chunk IDs from the old version, not just add new ones — otherwise outdated content keeps appearing in retrieval alongside the current version.

Why does my RAG system retrieve outdated or wrong content?

The usual causes are ingestion defects, not model problems: deleted or superseded documents whose chunks were never removed from the vector store, poor parsing that mangled the source text, or chunks that severed context mid-thought. Audit the pipeline first — check deletion handling, spot-check parser output on real documents, and verify chunk boundaries — before tuning prompts or swapping models.

Do we need an orchestrator like Airflow, or is cron enough?

Cron is fine for one or two independent jobs. Once tasks have dependencies, need retries, backfills, or alerting, an orchestrator pays for itself, which is why Airflow adoption tripled to 77,000+ organizations between 2020 and 2024. Managed options remove most of the operational burden for small teams.

When do we outgrow pandas?

Typically when datasets no longer fit comfortably in one machine's memory or single-threaded transforms become the bottleneck. Polars and DuckDB extend single-machine processing by 10x or more in published benchmarks before you need distributed systems like Spark, which add significant operational cost.

What does bad pipeline engineering actually cost?

Gartner puts the average cost of poor data quality at $12.9 million per year per organization, mostly through bad decisions and wasted rework. For startups the more common cost is losing trust in metrics, which stalls decision-making until someone rebuilds the pipeline with validation and lineage.

Bottom line: Dhairya Senjaliya ships Python — Data Processing Pipelines projects worldwide. Book a scoping call at https://dhairyasenjaliya.com/#book-call.