Data Validation with Pydantic in Pipelines

Key facts, with sources

• JetBrains' State of Python 2025 survey of more than 30,000 developers found that 51% of all Python developers are involved in data exploration and processing, with pandas and NumPy the most commonly used tools. (The JetBrains Blog)
• Apache Airflow reached more than 77,000 organizations using it as of November 2024, up from about 25,000 in 2020, with monthly downloads growing from 888,000 to over 31 million in the same period. (Astronomer State of Airflow 2025)
• The official Apache Airflow 2025 survey collected more than 5,250 responses from 116 countries, and event-driven scheduling introduced in Airflow 3 already showed almost 25% adoption. (Apache Airflow Blog)
• pandas has accumulated roughly 15.3 billion total downloads on PyPI, with around 700 million downloads in a single recent month. (pepy.tech)
• The Rust-based Polars DataFrame library passed 24 million monthly downloads and over 250 million total downloads as of September 2025, five years after its first commit. (Wikipedia)
• Gartner research estimates poor data quality costs organizations an average of at least $12.9 million per year. (Gartner)

Validate at the boundaries, trust the interior

The discipline is simple: every record entering the pipeline from the outside world — an API payload, a Kafka message, a CSV row, a vendor export — passes through a Pydantic model exactly once, at the edge. From that point inward, stages exchange typed models, not dictionaries, so transformation code stops being littered with defensive gets and isinstance checks.

This concentrates all schema knowledge in one place. When a source changes, there is one model to update and one diff to review, instead of a scavenger hunt through every stage that happened to touch the field. In code audits, pipelines that validate everywhere and pipelines that validate nowhere fail the same way — inconsistently. Boundary validation is the maintainable middle.

The quarantine pattern: one bad row never kills the batch

Raising on the first invalid record is the default behavior and the wrong one for pipelines — one malformed row from a vendor shouldn't discard a million good ones. Instead I split every batch into validated models and quarantined rejects, load the good rows, and write rejects to a quarantine table with the original payload and Pydantic's structured errors.

The quarantine count per run becomes a monitored metric with a threshold alert. A slow trickle of rejects is normal life with external data; a sudden spike is almost always an upstream deploy that changed the schema, caught within one run instead of discovered weeks later in an analytics review.

from pydantic import BaseModel, ValidationError, field_validator


class PaymentEvent(BaseModel):
    user_id: str
    amount_cents: int
    currency: str = "USD"

    @field_validator("currency")
    @classmethod
    def normalize_currency(cls, v: str) -> str:
        return v.strip().upper()


def validate_batch(
    rows: list[dict],
) -> tuple[list[PaymentEvent], list[dict]]:
    valid: list[PaymentEvent] = []
    quarantined: list[dict] = []
    for row in rows:
        try:
            valid.append(PaymentEvent.model_validate(row))
        except ValidationError as e:
            quarantined.append({"row": row, "errors": e.errors()})
    return valid, quarantined

Coercion and normalization belong in the model

Pydantic's lax mode already coerces the obvious cases — numeric strings to ints, ISO strings to datetimes — which suits messy real-world feeds. Field validators handle the rest: trimming whitespace, uppercasing currency codes, mapping legacy enum values to current ones. Putting normalization in the model means every consumer of the data gets the same cleaned values, instead of each stage re-implementing slightly different cleanup.

Use strict mode selectively where coercion would mask bugs — IDs are the classic case, where an integer silently becoming a string hides a producer-side type change you wanted to know about. The point is deliberateness: every field is either intentionally lax or intentionally strict, not whatever the default happened to be.

Schema evolution without breaking yesterday's data

Pipelines replay history — backfills, dead-letter reprocessing, restated vendor files — so models must accept every schema version still present in your data, not just today's. The practical rules: new fields arrive optional with defaults, removed fields stay tolerated (Pydantic ignores unknown keys by default), and renames are bridged with validation aliases so old and new payloads both parse.

When a source makes a genuinely breaking change, I add an explicit schema version field and a small dispatch that routes payloads to the matching model rather than contorting one model to accept everything. A model that accepts everything validates nothing — versioned models keep each contract honest while the pipeline handles all of history.

Performance: fast enough, if you use it right

Pydantic v2's validation core is implemented in Rust, and for typical batch pipelines validation is a small fraction of runtime next to I/O, parsing, and database writes. The overhead conversations that matter are about usage patterns, not the library. Validate each record once at the boundary — re-validating between internal stages buys nothing. For homogeneous lists, a single TypeAdapter for the list type validates the whole collection in one call, and building the adapter once at module level avoids repeated schema construction.

If profiling ever shows validation dominating a specific hot path — rare, in my experience — narrow the model for that path or drop to lighter checks there, while keeping full validation everywhere else. Don't preemptively strip validation from an entire pipeline to optimize a stage nobody measured.

When to hire senior help

Bring in a senior data engineer when pipeline failures start silently corrupting business metrics, or before committing to an orchestration and warehouse architecture, since storage layout and idempotency decisions are expensive to reverse once terabytes flow through them. A few weeks of experienced design work on schemas, retries, and backfill strategy routinely saves months of firefighting later. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships Python — Data Processing Pipelines projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Running pipelines as cron jobs plus scripts with no idempotency or retry semantics, so a mid-run failure leaves half-written tables that corrupt downstream reports
• ✕Loading entire datasets into pandas in memory instead of chunking or using Polars or DuckDB, causing out-of-memory crashes the first time data volume grows 10x
• ✕Skipping schema and data-quality checks at ingestion, letting silent schema drift from a source API propagate wrong numbers into dashboards for weeks
• ✕Designing pipelines that cannot deterministically backfill historical data, making every bug fix or logic change a manual one-off reprocessing project

Frequently asked questions

Sources

• The JetBrains Blog
• Astronomer State of Airflow 2025
• Apache Airflow Blog
• pepy.tech
• Wikipedia
• Gartner