Web Scraping Data Quality and Deduplication

Key facts, with sources

• The 2025 Imperva Bad Bot Report found automated traffic surpassed human activity for the first time in a decade, accounting for 51% of all web traffic. (Imperva)
• Bad bots alone made up 37% of all internet traffic in 2024, up from 32% the year before, according to the 2025 Imperva Bad Bot Report. (Business Wire)
• Mordor Intelligence sizes the web scraping market at $1.03 billion in 2025, projected to reach $2.23 billion by 2031 at a 13.78% compound annual growth rate. (Mordor Intelligence)
• Cloudflare's analysis of AI crawler traffic found that about 80% of AI crawling over a recent 12-month period was for model training, versus 18% for search and 2% for user-initiated actions. (Cloudflare)
• Cloudflare data shows Google crawls websites about 14 times per referral click it sends back, while OpenAI's crawl-to-referral ratio was roughly 1,700 to 1 in June 2025, illustrating how much scraping now happens without reciprocal traffic. (Cloudflare)

Validate at the boundary, fail loudly

Every record crossing from 'parsed HTML' to 'our data' passes through a Pydantic schema, and the schema is deliberately strict: required fields actually required, prices as integers in minor units with sane bounds, enums for anything categorical, and validators that normalize whitespace and reject empty-after-cleaning strings. Loose schemas defer every quality problem to whoever queries the data later, when context is gone.

Validation failures should be loud in aggregate but not fatal per record — one malformed listing must not kill a ten-thousand-page run. I count failures per field per source and fail the run only when the failure rate crosses a threshold, which distinguishes 'one weird page' from 'the site changed and everything is broken'.

from pydantic import BaseModel, field_validator


class Listing(BaseModel):
    source: str
    external_id: str
    title: str
    price_cents: int | None = None
    currency: str | None = None

    @field_validator("title")
    @classmethod
    def normalize_title(cls, value: str) -> str:
        cleaned = " ".join(value.split())
        if not cleaned:
            raise ValueError("title empty after normalization")
        return cleaned

    @field_validator("price_cents")
    @classmethod
    def sane_price(cls, value: int | None) -> int | None:
        if value is not None and not (0 < value < 100_000_000):
            raise ValueError("price outside sane bounds")
        return value

Deduplicate on identity first, content second

Duplicates enter scraped datasets through many doors: the same item reachable via multiple URLs, pagination overlap between runs, tracking parameters making one page look like five, and re-crawls of unchanged pages. I dedup at two levels. Level one is identity: a deterministic key from source plus the site's own stable identifier, enforced as a unique constraint so the database is the last line of defense. Level two is content: a hash of the normalized record that detects when a re-scraped item is byte-identical, so unchanged records update a last-seen timestamp instead of creating version churn.

import hashlib
import json


def identity_key(source: str, external_id: str) -> str:
    raw = f"{source.strip().lower()}:{external_id.strip().lower()}"
    return hashlib.sha256(raw.encode()).hexdigest()


def content_hash(record: dict) -> str:
    # Exclude volatile fields so unchanged content hashes stably
    stable = {k: v for k, v in record.items() if k not in {"fetched_at", "run_id"}}
    canonical = json.dumps(stable, sort_keys=True, ensure_ascii=False)
    return hashlib.sha256(canonical.encode()).hexdigest()

Near-duplicates need similarity, not equality

Hashes only catch exact matches after normalization, and real-world duplication is messier: the same product with a reordered title, the same article with an updated timestamp, listings differing only in a boilerplate suffix. For these I use similarity techniques — MinHash over token shingles for large corpora, or plain normalized-token Jaccard similarity when volumes are modest enough to compare within candidate blocks. Blocking matters: compare records within the same source and category, not everything against everything, or the pairwise cost explodes.

The policy question is as important as the algorithm: when two records are near-duplicates, which survives? My default is a canonicalization rule — prefer the record with more populated fields, then the more recent fetch — and I keep the losing record's identifier as an alias so the merge is reversible. Irreversible merges are how good data gets destroyed by an overconfident similarity threshold.

Drift detection: catching the silent redesign

The most expensive scraping failure is the quiet one: a site ships a redesign, your selectors still match something, and the pipeline happily stores empty strings or wrong fields for three weeks. Exceptions will not save you because nothing throws. The defense is statistical monitoring of the output. For every run I record per-field fill rates — what fraction of records have a non-null price, title, image — plus records-per-page and total volume, and alert when any metric deviates meaningfully from its trailing baseline.

Extraction versioning completes the loop. Every record stores the version of the parser that produced it, so when drift is detected and fixed, you know exactly which records need re-extraction — and if you kept raw HTML (you should), repair is a local reprocessing job rather than a re-crawl that burdens the source site again.

Quarantine, never silently drop

Records that fail validation or land in dedup ambiguity should go to a quarantine store, not the void. Quarantine preserves the raw payload, the failure reason, and the run context, which turns 'the data looks thin this week' from a mystery into a queryable question. A meaningful fraction of quarantined records are usually recoverable after a parser fix, and the failure-reason distribution is itself a quality dashboard — a sudden spike in one validator tells you precisely what changed upstream.

Quarantine also keeps incentives honest. Pipelines that silently drop bad records optimize for looking healthy; pipelines that quarantine make quality problems visible and cheap to triage. I set retention on the quarantine store, review the top failure reasons as part of regular maintenance, and treat a shrinking quarantine as the real signal that data quality work is paying off.

When to hire senior help

Bring in senior help when scraped data feeds production features or pricing decisions, because reliability engineering, compliance review, and change monitoring matter far more than the initial extraction script. An experienced engineer will also steer you toward official APIs, licensed feeds, and terms-of-service-respecting designs that avoid legal exposure and rework. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships Python — Web Scraping projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Scraping without first checking the site's terms of service, robots.txt, and whether an official API or licensed data feed already provides the data lawfully and more reliably
• ✕Sending unthrottled concurrent requests with no politeness delays, which looks like an attack, gets IP ranges banned, and can disrupt the target site's service
• ✕Collecting personal data without a lawful basis under GDPR or CCPA, turning a data project into a regulatory liability
• ✕Coupling parsers tightly to page DOM structure with no output validation or monitoring, so a site redesign silently fills the warehouse with empty or wrong records for weeks

Frequently asked questions

Sources

• Imperva
• Business Wire
• Mordor Intelligence
• Cloudflare
• Cloudflare