Building Scraping APIs with FastAPI

Key facts, with sources

• The 2025 Imperva Bad Bot Report found automated traffic surpassed human activity for the first time in a decade, accounting for 51% of all web traffic. (Imperva)
• Bad bots alone made up 37% of all internet traffic in 2024, up from 32% the year before, according to the 2025 Imperva Bad Bot Report. (Business Wire)
• Mordor Intelligence sizes the web scraping market at $1.03 billion in 2025, projected to reach $2.23 billion by 2031 at a 13.78% compound annual growth rate. (Mordor Intelligence)
• Cloudflare's analysis of AI crawler traffic found that about 80% of AI crawling over a recent 12-month period was for model training, versus 18% for search and 2% for user-initiated actions. (Cloudflare)
• Cloudflare data shows Google crawls websites about 14 times per referral click it sends back, while OpenAI's crawl-to-referral ratio was roughly 1,700 to 1 in June 2025, illustrating how much scraping now happens without reciprocal traffic. (Cloudflare)

The synchronous path: fast, single-page extraction

For extractions that finish in a couple of seconds — one page, parse, respond — a direct async endpoint is fine and keeps the client contract simple. The pieces that matter: Pydantic validates the inbound URL, the outbound request carries an honest User-Agent and a hard timeout, upstream failures map to a 502-style response instead of leaking stack traces, and the response is a typed model rather than raw soup. Note the deliberate follow_redirects=False — redirects are re-validated, which the next section explains.

import httpx
from bs4 import BeautifulSoup
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, HttpUrl

app = FastAPI(title="extract-api")


class ExtractRequest(BaseModel):
    url: HttpUrl


class ExtractResult(BaseModel):
    title: str | None
    headings: list[str]


@app.post("/extract", response_model=ExtractResult)
async def extract(req: ExtractRequest) -> ExtractResult:
    ensure_allowed_target(str(req.url))  # allowlist + SSRF checks
    async with httpx.AsyncClient(
        timeout=15.0,
        follow_redirects=False,
        headers={"User-Agent": "internal-extract-bot/1.0"},
    ) as client:
        resp = await client.get(str(req.url))
    if resp.status_code >= 300:
        raise HTTPException(502, f"upstream returned {resp.status_code}")
    soup = BeautifulSoup(resp.text, "html.parser")
    return ExtractResult(
        title=soup.title.get_text(strip=True) if soup.title else None,
        headings=[h.get_text(strip=True) for h in soup.find_all(["h1", "h2"])],
    )

SSRF is your biggest risk, so treat it first

An API that fetches caller-supplied URLs is a server-side request forgery machine unless you constrain it. Without checks, a caller can point your service at cloud metadata endpoints, internal admin panels, or anything else your network position reaches — with your service's credentials and egress. The baseline defenses: resolve the hostname and reject private, loopback, and link-local address ranges; prefer an explicit domain allowlist over a blocklist for internal tools; and disable automatic redirect following, because a public URL that redirects to an internal address defeats a naive pre-check.

There is a subtle time-of-check gap even then — DNS can change between validation and connection — so for higher-security environments, route outbound fetches through an egress proxy that enforces the same policy at connection time. I implement ensure_allowed_target as exactly this stack: allowlist check, DNS resolution, IP-range rejection, and manual redirect handling with re-validation of every hop.

The job pattern for anything slow

Holding an HTTP connection open while you crawl twenty pages is how you get client timeouts, duplicated requests from retries, and a service that falls over under modest concurrency. Anything beyond single-page latency becomes a job: POST validates and enqueues, returns a job ID immediately, and clients poll a status endpoint (or receive a webhook) for completion. Job state lives in Redis or Postgres with statuses like queued, running, succeeded, failed, plus a result pointer and error detail.

FastAPI's built-in BackgroundTasks is acceptable for genuinely small work, but it runs inside the web process — a deploy or crash loses the job, and heavy scraping competes with request handling. For production I use a real worker layer (Celery, ARQ, or RQ) so the API stays thin and the scraping workload scales independently of the web tier. Idempotency keys on job submission prevent double-crawls when clients retry a POST.

One shared client owns politeness

The API surface must not become a way for many callers to unknowingly hammer one site. All outbound fetching flows through a single shared client layer that owns the etiquette: per-domain rate limiting with jitter, robots.txt consultation cached per domain, response caching keyed by URL with sensible TTLs, conditional requests using stored ETags, and backoff on 429s that applies domain-wide rather than per request. Individual endpoints and workers never construct their own httpx clients.

This centralization has a compounding benefit: caching turns popular targets into near-free responses, which simultaneously improves your latency and shrinks your footprint on source sites. If ten internal consumers ask for the same page within the cache window, the origin sees one request. For an internal extraction API, a short TTL cache typically absorbs a large share of traffic — measure it, then tune TTLs per domain based on how often content actually changes.

Auth, quotas, and operational visibility

Even internal scraping APIs need authentication and per-caller quotas, because the failure mode is not malice — it is a well-meaning script in a retry loop multiplying your outbound traffic. API keys per consumer, rate limits per key at the edge, and a per-domain global budget underneath give you three independent brakes. Log every job with caller identity, target domain, page count, and outcome, so when a source site's operator asks what your traffic is, you can answer precisely — and demonstrate the limiter working.

Operationally, the metrics that matter mirror any scraper: upstream status-code distribution per domain, cache hit rate, job queue depth, and extraction success rate. Alert on rising 429s (you are crawling too hard), rising 403s (you are being blocked — stop and reassess, do not escalate), and queue growth (workers are underprovisioned). The API contract also deserves versioning from day one, because extraction schemas evolve and internal consumers break just as loudly as external ones.

When to hire senior help

Bring in senior help when scraped data feeds production features or pricing decisions, because reliability engineering, compliance review, and change monitoring matter far more than the initial extraction script. An experienced engineer will also steer you toward official APIs, licensed feeds, and terms-of-service-respecting designs that avoid legal exposure and rework. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships Python — Web Scraping projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Scraping without first checking the site's terms of service, robots.txt, and whether an official API or licensed data feed already provides the data lawfully and more reliably
• ✕Sending unthrottled concurrent requests with no politeness delays, which looks like an attack, gets IP ranges banned, and can disrupt the target site's service
• ✕Collecting personal data without a lawful basis under GDPR or CCPA, turning a data project into a regulatory liability
• ✕Coupling parsers tightly to page DOM structure with no output validation or monitoring, so a site redesign silently fills the warehouse with empty or wrong records for weeks

Frequently asked questions

Sources

• Imperva
• Business Wire
• Mordor Intelligence
• Cloudflare
• Cloudflare