Real-Time Data Pipelines with Kafka and Python

Key facts, with sources

• JetBrains' State of Python 2025 survey of more than 30,000 developers found that 51% of all Python developers are involved in data exploration and processing, with pandas and NumPy the most commonly used tools. (The JetBrains Blog)
• Apache Airflow reached more than 77,000 organizations using it as of November 2024, up from about 25,000 in 2020, with monthly downloads growing from 888,000 to over 31 million in the same period. (Astronomer State of Airflow 2025)
• The official Apache Airflow 2025 survey collected more than 5,250 responses from 116 countries, and event-driven scheduling introduced in Airflow 3 already showed almost 25% adoption. (Apache Airflow Blog)
• pandas has accumulated roughly 15.3 billion total downloads on PyPI, with around 700 million downloads in a single recent month. (pepy.tech)
• The Rust-based Polars DataFrame library passed 24 million monthly downloads and over 250 million total downloads as of September 2025, five years after its first commit. (Wikipedia)
• Gartner research estimates poor data quality costs organizations an average of at least $12.9 million per year. (Gartner)

When you actually need Kafka

Kafka buys you three things a task queue doesn't: replayable history, multiple independent consumer groups reading the same stream, and ordering within a partition. If your workload is "call this function later," a job queue like Celery with Redis is simpler and cheaper to operate. If your workload is "a stream of events that several systems consume at their own pace, and we may need to reprocess last week," that's Kafka's shape.

In audits I often find Kafka deployed as an expensive job queue — one topic, one consumer, no replay ever used. Be honest about which shape you have before signing up for broker operations or a managed-Kafka bill.

Producer side: keys, acks, and schema discipline

The partition key is the most consequential producer decision because it defines your ordering guarantee. Key by user ID and each user's events stay ordered; key by nothing and events round-robin across partitions with no ordering at all. Choose deliberately, and watch for hot keys — one huge tenant can concentrate load on a single partition no matter how many you provision.

For durability, produce with acks set to all and idempotence enabled, which the client supports natively. And version your payloads from day one — a schema version field in every message at minimum — because the consumer that reads an event may be deployed weeks after the producer that wrote it.

The consumer loop done right

The pattern below is the backbone of most consumers I ship: manual offset commits, committed only after processing succeeds. Auto-commit is the classic silent data-loss bug — offsets get committed on an interval whether or not your handler finished, so a crash mid-batch drops messages without a trace.

Manual commit converts that failure into a redelivery, which is the trade you want: at-least-once delivery with duplicates you handle downstream, instead of at-most-once with holes you can't detect.

import logging

from confluent_kafka import Consumer

conf = {
    "bootstrap.servers": "localhost:9092",
    "group.id": "enrichment-service",
    "enable.auto.commit": False,
    "auto.offset.reset": "earliest",
}

consumer = Consumer(conf)
consumer.subscribe(["events.raw"])

try:
    while True:
        msg = consumer.poll(timeout=1.0)
        if msg is None:
            continue
        if msg.error():
            logging.error("consumer error: %s", msg.error())
            continue
        try:
            process(msg.value())          # idempotent by design
            consumer.commit(message=msg)  # commit only after success
        except PermanentError:
            send_to_dead_letter(msg)
            consumer.commit(message=msg)
finally:
    consumer.close()

Exactly-once is a goal you approximate with idempotency

With manual commits you get at-least-once delivery, which means duplicates are guaranteed eventually — after a rebalance, a crash between processing and commit, or a redeploy. Chasing true exactly-once semantics across arbitrary downstream systems is usually the wrong fight. Making your writes idempotent is the right one.

Concretely: upsert on a deterministic event ID instead of inserting, make external side effects carry an idempotency key, and let duplicate deliveries collapse into no-ops at the sink. Once every write path is idempotent, redeliveries become harmless noise and your consumer logic gets dramatically simpler — you stop trying to be clever about failure timing and just let redelivery happen.

Poison messages and dead-letter topics

One malformed message will otherwise stall a partition forever: the handler throws, the offset never commits, the consumer redelivers the same event in a loop while lag climbs behind it. I distinguish transient errors (a database timeout — retry with backoff) from permanent ones (unparseable payload, schema violation — never retryable), and permanent failures go straight to a dead-letter topic with the original bytes, the error, and enough headers to trace provenance. Then the offset commits and the partition moves on.

The dead-letter topic gets two things: a monitored depth metric, because a sudden spike almost always means an upstream deploy broke the schema, and a replay script for after the fix ships.

Throughput realities for Python consumers

Your parallelism ceiling is the partition count — consumer instances beyond the number of partitions sit idle, so provision partitions with growth headroom because changing the count later reshuffles key-to-partition mappings. Within a single consumer, per-message network round-trips are the usual bottleneck; batching downstream writes typically lifts throughput far more than any code-level tuning.

The confluent-kafka client does its heavy lifting in C, so the client itself is rarely the constraint — your handler is. Watch consumer lag as the primary health metric. Lag trending upward means processing is slower than production, and the fix is batching, more partitions with more consumers, or moving slow enrichment out of the hot path.

When to hire senior help

Bring in a senior data engineer when pipeline failures start silently corrupting business metrics, or before committing to an orchestration and warehouse architecture, since storage layout and idempotency decisions are expensive to reverse once terabytes flow through them. A few weeks of experienced design work on schemas, retries, and backfill strategy routinely saves months of firefighting later. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships Python — Data Processing Pipelines projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Running pipelines as cron jobs plus scripts with no idempotency or retry semantics, so a mid-run failure leaves half-written tables that corrupt downstream reports
• ✕Loading entire datasets into pandas in memory instead of chunking or using Polars or DuckDB, causing out-of-memory crashes the first time data volume grows 10x
• ✕Skipping schema and data-quality checks at ingestion, letting silent schema drift from a source API propagate wrong numbers into dashboards for weeks
• ✕Designing pipelines that cannot deterministically backfill historical data, making every bug fix or logic change a manual one-off reprocessing project

Frequently asked questions

Sources

• The JetBrains Blog
• Astronomer State of Airflow 2025
• Apache Airflow Blog
• pepy.tech
• Wikipedia
• Gartner