Building Data Pipelines for Mobile Analytics

Key facts, with sources

• JetBrains' State of Python 2025 survey of more than 30,000 developers found that 51% of all Python developers are involved in data exploration and processing, with pandas and NumPy the most commonly used tools. (The JetBrains Blog)
• Apache Airflow reached more than 77,000 organizations using it as of November 2024, up from about 25,000 in 2020, with monthly downloads growing from 888,000 to over 31 million in the same period. (Astronomer State of Airflow 2025)
• The official Apache Airflow 2025 survey collected more than 5,250 responses from 116 countries, and event-driven scheduling introduced in Airflow 3 already showed almost 25% adoption. (Apache Airflow Blog)
• pandas has accumulated roughly 15.3 billion total downloads on PyPI, with around 700 million downloads in a single recent month. (pepy.tech)
• The Rust-based Polars DataFrame library passed 24 million monthly downloads and over 250 million total downloads as of September 2025, five years after its first commit. (Wikipedia)
• Gartner research estimates poor data quality costs organizations an average of at least $12.9 million per year. (Gartner)

Why mobile analytics is harder than web

Web analytics enjoys assumptions mobile destroys. Devices go offline for hours or days and then replay everything at once, so "events arrive near their occurrence time" is false. Networks drop acknowledgments after the server already wrote the batch, so retries create duplicates. Device clocks are wrong — occasionally by years — so client timestamps can't be trusted for ordering. And old app versions keep emitting old event schemas for as long as users defer updates, which is effectively forever.

Every downstream design choice follows from taking these seriously: idempotent ingestion, dual timestamps, late-data handling in sessionization, and additive-only schema evolution. Pipelines that ignore them produce dashboards that are subtly wrong in ways nobody can explain.

Client side: assign IDs, buffer locally, flush in batches

The single most important client decision is generating a UUID per event at creation time, on the device. That ID is what makes server-side deduplication possible — without it, a retried batch after a lost acknowledgment becomes indistinguishable duplicate rows. Each event also carries the client timestamp, session identifier, app version, and platform.

Events append to durable local storage immediately — never straight to the network — and a flusher ships batches opportunistically: on batch size, on interval, on app foregrounding. Batches that fail to send stay queued for later; a capped queue with oldest-first eviction keeps a long-offline device from growing unbounded. Sending one network request per event, by contrast, wrecks battery and delivers worse reliability than batching.

Ingestion API: accept fast, validate, land raw

The endpoint has one job: get valid events onto durable storage and acknowledge quickly, deferring anything heavy. I validate the envelope with Pydantic, stamp a server-side received timestamp next to the client's, and append the batch to durable raw storage — a raw events table or a stream — before returning success. Enrichment, sessionization, and aggregation all happen downstream, off the request path.

Keeping the raw layer immutable and append-only is deliberate: when a processing bug surfaces later, you rebuild derived tables from raw instead of losing history.

from datetime import datetime, timezone

from fastapi import FastAPI, status
from pydantic import BaseModel, Field

app = FastAPI()


class MobileEvent(BaseModel):
    event_id: str            # client-generated UUID: dedupe key
    name: str
    client_ts: datetime      # device clock: may be skewed
    session_id: str
    app_version: str
    platform: str            # "ios" | "android"
    props: dict = Field(default_factory=dict)


@app.post("/v1/events/batch", status_code=status.HTTP_202_ACCEPTED)
async def ingest_batch(events: list[MobileEvent]) -> dict:
    received_at = datetime.now(timezone.utc)
    rows = [
        {**e.model_dump(), "received_at": received_at}
        for e in events
    ]
    await append_to_raw_store(rows)  # durable before we ack
    return {"accepted": len(rows)}

Downstream: deduplicate, sessionize, handle the stragglers

Processing starts with deduplication on the client event ID — an insert that ignores conflicts, or a distinct-on view over raw — which quietly absorbs every retry duplicate the network created. Then sessionization: mobile sessions are typically derived from inactivity gaps (a common convention is around half a minute of background time ending a session), computed from event sequences per device, using client timestamps for intra-session ordering but server timestamps for cross-device analysis.

Late arrivals are the part teams underestimate. A device offline for a week delivers a week-old session, which means daily aggregates for closed days change after the fact. Pick a policy explicitly: recompute recent windows on a rolling basis and accept small restatements, or freeze windows after a cutoff and route very late events to a correction table. Either is defensible; silently ignoring the problem is what produces metrics nobody trusts.

Schema governance when old app versions never die

Whatever event schema an app version shipped with, some users will still be running it many months later — mobile pipelines don't get the web's luxury of deploying producers and consumers together. So schema changes are additive only: new fields optional with defaults, old fields tolerated indefinitely, renames handled by accepting both names during a long overlap. Every event carries app version and platform precisely so the pipeline can interpret payloads by their era when needed.

I also keep a versioned event catalog — names, required properties, meaning — checked into the repo, with client tracking calls and the ingestion models generated or reviewed against it. Analytics rot mostly comes from event definitions drifting between platforms and dashboards; one catalog that clients and pipeline both answer to is the cheap prevention.

When to hire senior help

Bring in a senior data engineer when pipeline failures start silently corrupting business metrics, or before committing to an orchestration and warehouse architecture, since storage layout and idempotency decisions are expensive to reverse once terabytes flow through them. A few weeks of experienced design work on schemas, retries, and backfill strategy routinely saves months of firefighting later. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships Python — Data Processing Pipelines projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Running pipelines as cron jobs plus scripts with no idempotency or retry semantics, so a mid-run failure leaves half-written tables that corrupt downstream reports
• ✕Loading entire datasets into pandas in memory instead of chunking or using Polars or DuckDB, causing out-of-memory crashes the first time data volume grows 10x
• ✕Skipping schema and data-quality checks at ingestion, letting silent schema drift from a source API propagate wrong numbers into dashboards for weeks
• ✕Designing pipelines that cannot deterministically backfill historical data, making every bug fix or logic change a manual one-off reprocessing project

Frequently asked questions

Sources

• The JetBrains Blog
• Astronomer State of Airflow 2025
• Apache Airflow Blog
• pepy.tech
• Wikipedia
• Gartner