Scrapy vs Playwright for Modern SPAs

Key facts, with sources

• The 2025 Imperva Bad Bot Report found automated traffic surpassed human activity for the first time in a decade, accounting for 51% of all web traffic. (Imperva)
• Bad bots alone made up 37% of all internet traffic in 2024, up from 32% the year before, according to the 2025 Imperva Bad Bot Report. (Business Wire)
• Mordor Intelligence sizes the web scraping market at $1.03 billion in 2025, projected to reach $2.23 billion by 2031 at a 13.78% compound annual growth rate. (Mordor Intelligence)
• Cloudflare's analysis of AI crawler traffic found that about 80% of AI crawling over a recent 12-month period was for model training, versus 18% for search and 2% for user-initiated actions. (Cloudflare)
• Cloudflare data shows Google crawls websites about 14 times per referral click it sends back, while OpenAI's crawl-to-referral ratio was roughly 1,700 to 1 in June 2025, illustrating how much scraping now happens without reciprocal traffic. (Cloudflare)

Start with the network tab, not the framework

The decision is not really Scrapy versus Playwright — it is HTML versus API. Before choosing tools, I load the target page with the browser dev tools open and watch the XHR/fetch requests. Most React, Vue, and Next.js sites pull their data from JSON endpoints, and those endpoints usually return cleaner, better-structured data than anything you could parse out of rendered HTML. If such an endpoint exists, is publicly reachable without authentication, and the site's robots.txt and terms permit automated access, calling it directly with Scrapy or httpx is faster, cheaper, and more stable than rendering.

Only when data is genuinely embedded in client-rendered DOM — no accessible endpoint, or responses that are obfuscated or signed — does a real browser earn its cost. That is a minority of the SPAs I audit.

What Scrapy brings beyond fetching

People compare Scrapy to Playwright as if both were fetchers, but Scrapy's real value is crawl infrastructure: a scheduler with per-domain concurrency limits, automatic throttling, retry middleware, robots.txt enforcement, item pipelines, and feed exports. When a job involves thousands of pages across many domains, that plumbing is what you would otherwise write by hand.

Scrapy is also where politeness settings live in one obvious place — ROBOTSTXT_OBEY, DOWNLOAD_DELAY, and AUTOTHROTTLE_ENABLED belong in every production spider I ship. A headless browser script has none of this by default; you end up rebuilding a worse version of it.

import scrapy


class ProductSpider(scrapy.Spider):
    name = "products"
    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "DOWNLOAD_DELAY": 2.0,
        "AUTOTHROTTLE_ENABLED": True,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 2,
        "USER_AGENT": "catalog-research-bot/1.0",
    }

    def parse(self, response):
        for card in response.css("article.product"):
            yield {
                "name": card.css("h2::text").get(),
                "price": card.css(".price::text").get(),
            }
        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Where Playwright genuinely earns its keep

When content truly requires JavaScript — infinite scroll that computes items client-side, DOM built from WebSocket messages, or state that only exists after user-like interaction — Playwright is the right tool and a pleasure to use. Its auto-waiting model (wait for a selector, wait for network idle) removes most of the flakiness that plagued older headless setups, and it drives Chromium, Firefox, and WebKit through one API.

The discipline is to treat the browser as a rendering step, not a crawler. Fetch the page, wait for the specific selector you need, grab the HTML or evaluate a small extraction script, and close. Long-lived browser sessions that click around like a ghost user are fragile and much harder to rate-limit responsibly.

from playwright.async_api import async_playwright


async def fetch_rendered(url: str) -> str:
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.goto(url, wait_until="domcontentloaded")
        await page.wait_for_selector("article.product")
        html = await page.content()
        await browser.close()
    return html

The hybrid pattern that usually wins

On most SPA projects I end up with a two-tier architecture. Tier one uses Playwright sparingly — to discover how the site loads data, to render the small subset of pages that genuinely need JavaScript, or to snapshot a page type once so I can study its API calls. Tier two does the volume work over plain HTTP with Scrapy or httpx against whatever server-rendered pages or JSON endpoints exist.

If you want both inside one framework, the scrapy-playwright integration lets individual Scrapy requests opt into browser rendering while everything else stays on the fast path. That keeps Scrapy's scheduling and politeness machinery in charge of every request, rendered or not, which is exactly where you want rate limiting enforced.

Cost, operations, and the ethics constant

A headless browser typically consumes an order of magnitude more CPU and memory per page than an HTTP fetch, and browser fleets bring their own failure modes — zombie processes, memory growth, version drift between the driver and the browser. If your crawl volume is large, that difference decides your infrastructure bill. HTTP-first designs also parallelize more predictably because concurrency is bounded by connections, not by browser instances.

One thing that does not change between tools: the rules. Robots.txt, terms of service, honest identification, and per-domain rate limits apply equally whether the request comes from Scrapy or Chromium. A browser that renders like a human is not a license to crawl like a machine — if anything, rendered crawling should be slower, because each page load triggers many subresource requests on the target's infrastructure.

When to hire senior help

Bring in senior help when scraped data feeds production features or pricing decisions, because reliability engineering, compliance review, and change monitoring matter far more than the initial extraction script. An experienced engineer will also steer you toward official APIs, licensed feeds, and terms-of-service-respecting designs that avoid legal exposure and rework. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships Python — Web Scraping projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Scraping without first checking the site's terms of service, robots.txt, and whether an official API or licensed data feed already provides the data lawfully and more reliably
• ✕Sending unthrottled concurrent requests with no politeness delays, which looks like an attack, gets IP ranges banned, and can disrupt the target site's service
• ✕Collecting personal data without a lawful basis under GDPR or CCPA, turning a data project into a regulatory liability
• ✕Coupling parsers tightly to page DOM structure with no output validation or monitoring, so a site redesign silently fills the warehouse with empty or wrong records for weeks

Frequently asked questions

Sources

• Imperva
• Business Wire
• Mordor Intelligence
• Cloudflare
• Cloudflare