Python — Data Processing Pipelines

Batch Processing Large Datasets on AWS

Direct answer

For batch processing large datasets on AWS, I default to the smallest architecture that finishes in the required window: a single container on Fargate or EC2 streaming S3 objects through Python generators handles far more than most teams assume, AWS Batch or SQS-driven workers add fan-out when one machine isn't enough, and Spark on EMR or Glue is the last resort for genuinely distributed workloads. The patterns that matter at every tier are partitioned S3 layout, Parquet over row formats, manifest-driven work distribution, and checkpointing so jobs resume instead of restart.

The most expensive mistake in AWS batch processing is reaching for distributed tooling before exhausting what one well-fed container can do. This is the escalation ladder I use, and the S3 and checkpointing patterns that stay constant at every rung.

Key facts, with sources

JetBrains' State of Python 2025 survey of more than 30,000 developers found that 51% of all Python developers are involved in data exploration and processing, with pandas and NumPy the most commonly used tools. (The JetBrains Blog)
Apache Airflow reached more than 77,000 organizations using it as of November 2024, up from about 25,000 in 2020, with monthly downloads growing from 888,000 to over 31 million in the same period. (Astronomer State of Airflow 2025)
The official Apache Airflow 2025 survey collected more than 5,250 responses from 116 countries, and event-driven scheduling introduced in Airflow 3 already showed almost 25% adoption. (Apache Airflow Blog)
pandas has accumulated roughly 15.3 billion total downloads on PyPI, with around 700 million downloads in a single recent month. (pepy.tech)
The Rust-based Polars DataFrame library passed 24 million monthly downloads and over 250 million total downloads as of September 2025, five years after its first commit. (Wikipedia)
Gartner research estimates poor data quality costs organizations an average of at least $12.9 million per year. (Gartner)

Pick the smallest tool that finishes overnight

My escalation ladder has four rungs. First: a scheduled container — Fargate task or a modest EC2 instance — running plain Python that streams data instead of loading it. With generators and Parquet, a single machine chews through datasets that sound big on paper. Second: fan-out with AWS Batch or an SQS queue feeding identical workers, when the window can't be met serially. Third: Spark via EMR or Glue, when a single logical operation — a massive join, a global aggregation — truly exceeds one machine. Fourth rarely comes.

Each rung multiplies operational complexity and debugging difficulty. A failed Python script gives you a stack trace; a failed Spark job gives you an archaeology project. Climb only when the current rung demonstrably can't meet the deadline.

S3 layout decides your costs before any code runs

Batch economics are mostly decided by how data lands in S3. Partition object keys by the dimensions you filter on — typically date, sometimes tenant — so jobs list and read only the slice they need instead of scanning the bucket. Prefer Parquet over CSV or JSON lines for anything analytical: columnar layout means a job reading three columns skips the other forty, and compression is dramatically better.

Two failure modes to avoid. Tiny-file storms — millions of kilobyte objects — make listing and per-request overhead dominate runtime, so compact small files into larger ones during ingestion. And unpartitioned dumps force every job into full scans forever; retrofitting partitioning later means rewriting the lake, so get the layout right when data first arrives.

Stream objects, don't load them

The pattern below — a paginator feeding a generator — is how a modest container processes a bucket far larger than its memory. Nothing is held beyond the current object, and processing starts immediately instead of after a giant download.

Memory-bound crashes in batch jobs almost always trace to someone loading a full dataset into a DataFrame out of habit. Reserve in-memory loading for the aggregation step that genuinely needs it, and stream everything before that point.

Streaming S3 objects with a paginator

import gzip
import json
from collections.abc import Iterator

import boto3

s3 = boto3.client("s3")


def iter_records(bucket: str, prefix: str) -> Iterator[dict]:
    paginator = s3.get_paginator("list_objects_v2")
    for page in paginator.paginate(Bucket=bucket, Prefix=prefix):
        for obj in page.get("Contents", []):
            body = s3.get_object(Bucket=bucket, Key=obj["Key"])["Body"]
            with gzip.open(body, "rt") as lines:
                for line in lines:
                    yield json.loads(line)


for record in iter_records("raw-events", "events/dt=2026-07-01/"):
    process(record)

Fan out with a manifest, not with cleverness

When one worker isn't enough, resist the urge to invent coordination. The robust pattern is dumb: enumerate the work up front into a manifest — a list of S3 prefixes, date ranges, or object keys — push each unit onto SQS or submit it as an AWS Batch array job, and run identical stateless workers that each claim a unit, process it, and record completion.

Workers must be idempotent, because SQS delivers at-least-once and Batch retries failures — write outputs keyed deterministically by work unit so a redelivered unit overwrites its own results harmlessly. Completion records double as checkpoints: rerunning the job skips finished units, so a failure at ninety percent costs you ten percent, not a restart.

The cost levers that actually matter

Batch workloads are the ideal Spot instance customer — interruption-tolerant by design if you've built the checkpointing above — and Spot pricing typically cuts compute costs substantially versus on-demand. Combined with scale-to-zero (Fargate tasks and Batch environments that cost nothing between runs), compute for a nightly job becomes a minor line item.

The sneakier costs are data-shaped: full-bucket scans caused by missing partitioning, S3 request charges from tiny-file storms, cross-region transfer because compute and data live in different regions, and old intermediates nobody deleted — lifecycle policies that expire or archive them are one-time configuration. In my experience the storage-layout fixes usually save more than any instance tuning, because they cut work rather than the price of work.

When to hire senior help

Bring in a senior data engineer when pipeline failures start silently corrupting business metrics, or before committing to an orchestration and warehouse architecture, since storage layout and idempotency decisions are expensive to reverse once terabytes flow through them. A few weeks of experienced design work on schemas, retries, and backfill strategy routinely saves months of firefighting later. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships Python — Data Processing Pipelines projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

✕Running pipelines as cron jobs plus scripts with no idempotency or retry semantics, so a mid-run failure leaves half-written tables that corrupt downstream reports
✕Loading entire datasets into pandas in memory instead of chunking or using Polars or DuckDB, causing out-of-memory crashes the first time data volume grows 10x
✕Skipping schema and data-quality checks at ingestion, letting silent schema drift from a source API propagate wrong numbers into dashboards for weeks
✕Designing pipelines that cannot deterministically backfill historical data, making every bug fix or logic change a manual one-off reprocessing project

Frequently asked questions

Do I need Spark to process large datasets on AWS?

Less often than you'd think. A single container streaming Parquet from partitioned S3 with Python generators handles datasets well into the hundreds of gigabytes, and manifest-driven fan-out across parallel workers extends that much further. Spark earns its complexity when a single logical operation — a huge join or global aggregation — genuinely exceeds what partition-parallel workers can do independently.

How do I make an AWS batch job resumable after failure?

Break the work into units in a manifest — prefixes, date ranges, or key lists — and record each unit's completion durably as workers finish. Write outputs keyed deterministically by unit so retries overwrite rather than duplicate. On rerun, skip completed units. This turns a failure late in a long job into a small incremental rerun instead of a full restart.

How can I reduce AWS batch processing costs?

Attack data layout first: partition S3 keys by the dimensions you filter on, convert to Parquet so jobs read only needed columns, and compact tiny files. Then attack compute: run on Spot instances since checkpointed batch work tolerates interruption, use scale-to-zero services like Fargate or AWS Batch, and add lifecycle policies to expire stale intermediate data.

Do we need an orchestrator like Airflow, or is cron enough?

Cron is fine for one or two independent jobs. Once tasks have dependencies, need retries, backfills, or alerting, an orchestrator pays for itself, which is why Airflow adoption tripled to 77,000+ organizations between 2020 and 2024. Managed options remove most of the operational burden for small teams.

When do we outgrow pandas?

Typically when datasets no longer fit comfortably in one machine's memory or single-threaded transforms become the bottleneck. Polars and DuckDB extend single-machine processing by 10x or more in published benchmarks before you need distributed systems like Spark, which add significant operational cost.

What does bad pipeline engineering actually cost?

Gartner puts the average cost of poor data quality at $12.9 million per year per organization, mostly through bad decisions and wasted rework. For startups the more common cost is losing trust in metrics, which stalls decision-making until someone rebuilds the pipeline with validation and lineage.

Bottom line: Dhairya Senjaliya ships Python — Data Processing Pipelines projects worldwide. Book a scoping call at https://dhairyasenjaliya.com/#book-call.