Error Recovery in Production Automation

Key facts, with sources

• Grand View Research sized the robotic process automation market at $4.68 billion in 2025 and projects it to reach $35.84 billion by 2033, a 29.0% compound annual growth rate. (Grand View Research)
• Gartner's worldwide market share analysis found RPA software generated about $3.8 billion in revenue in 2024, an 18% year-over-year increase, even as generative AI and agentic tools slowed the segment's growth rate. (Gartner)
• TestGuild's 2025 survey put Playwright at 45.1% adoption among QA professionals with a 94% retention rate, versus 22% and declining for Selenium. (TestDino)
• Playwright job postings grew 180% year over year in 2025, making it the fastest-growing category in QA automation hiring. (TestDino)
• Playwright leads browser automation tooling with roughly 30 million weekly npm downloads compared to Cypress at 6.5 million, after growing from about 1.2 million weekly downloads in January 2022. (Tech Insider)

Classify failures before you handle them

Every error in an automation pipeline belongs to one of three families, and each demands a different response. Transient errors — network timeouts, rate limits, an upstream returning a 503 — will likely succeed on retry, so retrying is correct. Permanent errors — authentication failures, validation rejections, most 4xx responses — will fail identically every time, so retrying just hammers the upstream and delays the real fix. Data errors — a malformed record, a missing required field — aren't system failures at all; the record needs quarantining and a human, while the rest of the batch proceeds.

Code that treats all three the same gets the worst of every world: it retries bugs, gives up on blips, and aborts thousand-record jobs over one bad row. Classification is the foundation every other layer builds on.

Retry the transient, reject the permanent

I use tenacity for retry logic because it makes the policy declarative and visible at the function signature: which exceptions retry, how the wait grows, when to give up. Exponential backoff with a cap is the default — immediate retries against a struggling service pile on load, and a rate-limited client that doesn't back off earns itself stricter limits.

The equally important half is the exclusion list. My HTTP wrappers convert 4xx responses into a PermanentError type that the retry decorator never touches, so a misconfigured auth token fails in seconds with a clear message instead of five minutes of doomed retries obscuring the cause.

import requests
from tenacity import (
    retry,
    retry_if_exception_type,
    stop_after_attempt,
    wait_exponential,
)


class PermanentError(Exception):
    """Do not retry: auth failures, validation rejections, 4xx."""


@retry(
    retry=retry_if_exception_type((requests.ConnectionError, requests.Timeout)),
    wait=wait_exponential(multiplier=1, max=60),
    stop=stop_after_attempt(5),
    reraise=True,
)
def push_record(url: str, record: dict) -> None:
    resp = requests.post(url, json=record, timeout=30)
    if 400 <= resp.status_code < 500:
        raise PermanentError(
            f"rejected {resp.status_code}: {resp.text[:200]}"
        )
    resp.raise_for_status()

Idempotency is what makes retries safe

Retries are only a valid strategy if repeating an operation is harmless, and that property has to be designed in. The retry that duplicates a customer email, double-posts a ledger entry, or creates a second CRM record isn't recovery — it's a new incident. The timeout case is the sneaky one: the request that timed out may have actually succeeded server-side, so the retry is a genuine repeat even though the client saw a failure.

The toolkit: upserts keyed on natural or external IDs instead of blind inserts, idempotency keys passed on any POST that creates something or moves money, and processed-markers checked before side effects like sending notifications. I design the idempotency story before writing any retry logic, because retries without it are strictly worse than failing.

Checkpoint long jobs so crashes resume, not restart

A job that processes ten thousand records and crashes at nine thousand should not start over. I persist a checkpoint — the last successfully processed cursor, ID, or timestamp — to durable storage at batch boundaries, and every job begins by reading its checkpoint and resuming from it. Combined with idempotent writes, this makes crash recovery boring: restart the job, it picks up roughly where it died, and any overlap re-processes harmlessly.

Within a batch, I process record-by-record with per-record error capture rather than failing the whole batch on one bad row. Good records flow through; bad ones are recorded with their error and set aside. A run that ends "9,987 succeeded, 13 quarantined" is a normal Tuesday, not an outage.

Dead-letter queues and the human escalation path

Everything that exhausts its retries or fails validation lands in a dead-letter queue — often just a database table — with the full context a human needs: the record itself, the operation attempted, the error, the attempt count, and timestamps. The alert on the DLQ reports counts and categories, not a page per record, because a hundred identical failures are one problem.

Two practices keep the DLQ from becoming a landfill. First, a replay command: once the underlying cause is fixed, one command re-runs the dead-lettered items through the normal pipeline — recovery uses the same tested code path, not an ad-hoc script. Second, a review cadence with a real owner, because a dead-letter queue nobody reads is just silent failure with extra steps.

When to hire senior help

Bring in senior help when automations move from convenience scripts to business-critical paths, such as billing, order processing, or compliance reporting, where a silent failure has real financial consequences. An experienced engineer will add the monitoring, idempotency, and credential management that separates durable automation systems from fragile scripts. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships Python — Automation Systems projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Automating a broken manual process as-is instead of mapping and simplifying it first, which just makes the inefficiency run faster
• ✕Building UI screen-scraping bots against internal apps that expose APIs, so every minor UI update breaks the automation
• ✕Running unattended automations with no monitoring or alerting, so a silently failing nightly job goes unnoticed until month-end numbers are wrong
• ✕Hardcoding credentials in scripts and running automations under a personal employee account, creating security exposure and a single point of failure when that person leaves

Frequently asked questions

Sources

• Grand View Research
• Gartner
• TestDino
• Tech Insider