Pipeline Orchestration for ML Feature Stores

Key facts, with sources

• JetBrains' State of Python 2025 survey of more than 30,000 developers found that 51% of all Python developers are involved in data exploration and processing, with pandas and NumPy the most commonly used tools. (The JetBrains Blog)
• Apache Airflow reached more than 77,000 organizations using it as of November 2024, up from about 25,000 in 2020, with monthly downloads growing from 888,000 to over 31 million in the same period. (Astronomer State of Airflow 2025)
• The official Apache Airflow 2025 survey collected more than 5,250 responses from 116 countries, and event-driven scheduling introduced in Airflow 3 already showed almost 25% adoption. (Apache Airflow Blog)
• pandas has accumulated roughly 15.3 billion total downloads on PyPI, with around 700 million downloads in a single recent month. (pepy.tech)
• The Rust-based Polars DataFrame library passed 24 million monthly downloads and over 250 million total downloads as of September 2025, five years after its first commit. (Wikipedia)
• Gartner research estimates poor data quality costs organizations an average of at least $12.9 million per year. (Gartner)

What a feature store actually asks of your orchestrator

Beneath the terminology, a feature store is two data stores with a synchronization contract. The offline store holds historical feature values for building training sets; the online store holds current values for millisecond lookups at inference time. The orchestrator owns the contract: compute features on schedule, land them offline, materialize them online, and never let the two drift apart silently.

That translates to concrete orchestration requirements — dependency ordering (materialization must follow computation of the same window), per-feature-group scheduling since different features have different freshness needs, retries that don't double-apply writes, and backfill support that respects event time. Generic orchestrators like Airflow, Prefect, or Dagster handle all of this; the feature-store-specific discipline is in how you structure the jobs.

One definition, two materializations

Training/serving skew — where the feature a model saw in training differs subtly from the one it sees in production — is the defining failure of ML data infrastructure, and it almost always starts as two implementations of the "same" transformation: one in the batch SQL that builds training sets, another in the service code that computes values at request time. They drift, and the model quietly degrades with no error anywhere.

The architectural rule I enforce: each feature transformation is defined exactly once, and both the offline job and the online materialization execute that single definition. Whether it's a shared Python function, a dbt model both paths read from, or a feature framework's declarative definition matters less than the invariant — one definition, two destinations, zero reimplementation.

Schedule around freshness SLAs, not cron habits

Not every feature deserves the same schedule. A user's lifetime aggregate can be a day stale with no model impact; a fraud feature reflecting the last hour of activity cannot. I attach an explicit freshness SLA to each feature group, then derive the schedule from it — rather than defaulting everything to a nightly run because that's what the first job used.

The orchestrator's role is to make the SLA observable and enforced: each feature group's flow records the event-time watermark it computed through, and an independent check alerts when any group's watermark lags its SLA. That converts freshness from an assumption into a monitored contract.

from datetime import datetime

from prefect import flow, task


@task(retries=3, retry_delay_seconds=60)
def compute_user_features(window_end: datetime) -> list[dict]:
    # single source of truth: same transformation code
    # used for offline training data and online serving
    return build_features(window_end)


@task(retries=3, retry_delay_seconds=60)
def write_offline(rows: list[dict], window_end: datetime) -> None:
    upsert_offline_store(rows, watermark=window_end)


@task(retries=3, retry_delay_seconds=60)
def materialize_online(rows: list[dict]) -> None:
    upsert_online_store(rows)  # keyed by entity id: idempotent


@flow
def refresh_user_features(window_end: datetime) -> None:
    rows = compute_user_features(window_end)
    write_offline(rows, window_end)
    materialize_online(rows)  # only after offline write succeeds

Backfills must be point-in-time correct

Backfilling a feature store is more dangerous than backfilling a dashboard, because training sets are assembled with point-in-time joins: for each training example, the feature value must be what was knowable at that example's event time. A naive backfill that computes features from today's data and stamps them onto historical rows leaks future information into training — the model looks brilliant offline and falls apart in production.

So backfill jobs replay history window by window, computing each window's features only from data with event times inside or before it, and writing values tagged with their effective time. It's the same flow as the scheduled run, parameterized by window — which is exactly why the flow above takes window_end as an argument instead of assuming now.

Be honest about whether you need one yet

A feature store is infrastructure for a real problem — multiple models sharing features, online inference needing sub-second lookups, teams reimplementing the same aggregations — and adopting one before those problems exist mostly buys operational burden. For a startup with one or two batch-scored models, a features table in the warehouse plus a cache is often the honest architecture, and it migrates cleanly later.

My adoption triggers: a second model wants features the first already computes, latency requirements force precomputed online lookups, or a skew incident actually bites. Until then, keep transformations in version-controlled, single-definition code — that discipline is the part of the feature store you can't retrofit, and it costs nothing to start with.

When to hire senior help

Bring in a senior data engineer when pipeline failures start silently corrupting business metrics, or before committing to an orchestration and warehouse architecture, since storage layout and idempotency decisions are expensive to reverse once terabytes flow through them. A few weeks of experienced design work on schemas, retries, and backfill strategy routinely saves months of firefighting later. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships Python — Data Processing Pipelines projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Running pipelines as cron jobs plus scripts with no idempotency or retry semantics, so a mid-run failure leaves half-written tables that corrupt downstream reports
• ✕Loading entire datasets into pandas in memory instead of chunking or using Polars or DuckDB, causing out-of-memory crashes the first time data volume grows 10x
• ✕Skipping schema and data-quality checks at ingestion, letting silent schema drift from a source API propagate wrong numbers into dashboards for weeks
• ✕Designing pipelines that cannot deterministically backfill historical data, making every bug fix or logic change a manual one-off reprocessing project

Frequently asked questions

Sources

• The JetBrains Blog
• Astronomer State of Airflow 2025
• Apache Airflow Blog
• pepy.tech
• Wikipedia
• Gartner