Scheduled Scraping with Rate Limiting

Key facts, with sources

• The 2025 Imperva Bad Bot Report found automated traffic surpassed human activity for the first time in a decade, accounting for 51% of all web traffic. (Imperva)
• Bad bots alone made up 37% of all internet traffic in 2024, up from 32% the year before, according to the 2025 Imperva Bad Bot Report. (Business Wire)
• Mordor Intelligence sizes the web scraping market at $1.03 billion in 2025, projected to reach $2.23 billion by 2031 at a 13.78% compound annual growth rate. (Mordor Intelligence)
• Cloudflare's analysis of AI crawler traffic found that about 80% of AI crawling over a recent 12-month period was for model training, versus 18% for search and 2% for user-initiated actions. (Cloudflare)
• Cloudflare data shows Google crawls websites about 14 times per referral click it sends back, while OpenAI's crawl-to-referral ratio was roughly 1,700 to 1 in June 2025, illustrating how much scraping now happens without reciprocal traffic. (Cloudflare)

Pick the scheduler that matches your deployment

For a single-process service, APScheduler is my default: cron-style triggers, async support, and — critically — max_instances and coalesce options that stop a slow crawl from stacking on top of its own next run. For systems that already run Celery, beat is the natural fit and gives you distribution and retries for free. Plain cron still earns its keep for simple containers, but push the actual work through a queue or a lock so two overlapping invocations cannot both crawl.

Whatever you choose, schedule crawls for the target's quiet hours where you can infer them, and stagger jobs across sources rather than launching everything at the top of the hour. A scheduler that fires fifty spiders simultaneously creates a thundering herd against your own proxy of politeness budgets.

from apscheduler.schedulers.asyncio import AsyncIOScheduler
from apscheduler.triggers.cron import CronTrigger

scheduler = AsyncIOScheduler()

scheduler.add_job(
    crawl_catalog,                    # async def crawl_catalog() -> None
    CronTrigger(hour=3, minute=30),
    id="catalog-crawl",
    max_instances=1,                  # never run two crawls at once
    coalesce=True,                    # collapse missed runs into one
    misfire_grace_time=3600,
)

scheduler.start()

Rate limiting belongs in the client, not the spider

The mistake I see most in audits: each scraper implements its own sleep calls, so the moment two jobs touch the same domain, the effective rate doubles. The fix is architectural — one shared client layer owns politeness, and every job goes through it. Per-domain state means a burst against one site cannot borrow budget from another, and jitter prevents the robotic fixed-interval signature that both looks bad and synchronizes badly.

import asyncio
import random

import httpx


class DomainRateLimiter:
    """At most one request per interval per domain, with jitter."""

    def __init__(self, interval: float = 3.0):
        self.interval = interval
        self._locks: dict[str, asyncio.Lock] = {}
        self._next_slot: dict[str, float] = {}

    async def acquire(self, domain: str) -> None:
        lock = self._locks.setdefault(domain, asyncio.Lock())
        async with lock:
            loop = asyncio.get_running_loop()
            wait = self._next_slot.get(domain, loop.time()) - loop.time()
            if wait > 0:
                await asyncio.sleep(wait)
            jitter = random.uniform(0, self.interval * 0.3)
            self._next_slot[domain] = loop.time() + self.interval + jitter


limiter = DomainRateLimiter(interval=3.0)


async def polite_get(client: httpx.AsyncClient, url: str) -> httpx.Response:
    domain = httpx.URL(url).host
    await limiter.acquire(domain)
    return await client.get(url)

Backoff is how you listen to the server

Rate limits are your opening bid; the server's responses are the negotiation. A 429 means slow down now — honor a Retry-After header exactly when present, and otherwise back off exponentially with jitter. Repeated 5xx responses mean the site is struggling, and a polite crawler treats that as a reason to pause the whole domain, not just retry the one URL harder. I cap retries low (two or three attempts) and route persistent failures to a dead-letter list for the next scheduled run rather than grinding in-loop.

Two details worth encoding: after any 429, penalize the domain's rate for the rest of the run, not just the single request — the server told you your baseline is too high. And distinguish retryable failures (timeouts, 429, 503) from permanent ones (404, 410), because retrying permanent failures is pure waste that also looks like probing.

Honor the signals sites publish

Before the first scheduled run against a new domain, my pipeline fetches and parses robots.txt, and the scheduler config records what it found: disallowed paths are excluded from the URL frontier entirely, and any crawl-delay directive overrides my default interval if it is longer. Python's standard library robotparser handles the parsing; the important part is wiring its answer into the limiter and the frontier rather than checking it once and forgetting.

Conditional requests are the other underused signal. Storing ETag and Last-Modified values and sending If-None-Match and If-Modified-Since turns unchanged pages into cheap 304 responses — less bandwidth for them, faster runs for you, and a materially smaller footprint for a recurring job that mostly re-visits pages that have not changed. For a scheduled scraper, cache validation is often the single biggest politeness win available.

Idempotency and monitoring close the loop

Scheduled jobs fail at 3 a.m., so design for safe reruns. Every run gets a run ID; writes are upserts keyed on natural identifiers rather than blind inserts; and a checkpoint of completed URLs lets a restarted run skip finished work instead of re-fetching it. Overlap protection needs to exist even with scheduler-level guards — a distributed lock (Redis works fine) around the run is cheap insurance when someone triggers a manual run during the scheduled window.

Monitor the things that actually degrade: request success rate per domain, 429 frequency, run duration trend, and — most important — items extracted per run. A scraper that runs green while extracting zero items is the classic silent failure; alert on extraction volume deviating from the recent baseline, not just on exceptions. I also log the politeness metrics themselves, because being able to show a source exactly how gently you crawl is occasionally very useful.

When to hire senior help

Bring in senior help when scraped data feeds production features or pricing decisions, because reliability engineering, compliance review, and change monitoring matter far more than the initial extraction script. An experienced engineer will also steer you toward official APIs, licensed feeds, and terms-of-service-respecting designs that avoid legal exposure and rework. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships Python — Web Scraping projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Scraping without first checking the site's terms of service, robots.txt, and whether an official API or licensed data feed already provides the data lawfully and more reliably
• ✕Sending unthrottled concurrent requests with no politeness delays, which looks like an attack, gets IP ranges banned, and can disrupt the target site's service
• ✕Collecting personal data without a lawful basis under GDPR or CCPA, turning a data project into a regulatory liability
• ✕Coupling parsers tightly to page DOM structure with no output validation or monitoring, so a site redesign silently fills the warehouse with empty or wrong records for weeks

Frequently asked questions

Sources

• Imperva
• Business Wire
• Mordor Intelligence
• Cloudflare
• Cloudflare