Python — Web Scraping

Web Scraping Pipelines for AI Training Data

Direct answer

A scraping pipeline for AI training data should be staged — crawl, extract, clean, deduplicate, filter, version — with each stage writing to durable storage so you can reprocess without re-crawling. The stages that decide corpus quality are boilerplate removal, near-duplicate detection, and licensing checks: duplicated or low-quality documents degrade a fine-tune or a RAG index faster than a smaller, cleaner corpus ever would.

I build ingestion pipelines that feed RAG systems and fine-tuning jobs, and the scraping half is where most corpus quality is won or lost. This is the stage-by-stage architecture I use, including the licensing and consent checks that AI training data specifically demands.

Key facts, with sources

The 2025 Imperva Bad Bot Report found automated traffic surpassed human activity for the first time in a decade, accounting for 51% of all web traffic. (Imperva)
Bad bots alone made up 37% of all internet traffic in 2024, up from 32% the year before, according to the 2025 Imperva Bad Bot Report. (Business Wire)
Mordor Intelligence sizes the web scraping market at $1.03 billion in 2025, projected to reach $2.23 billion by 2031 at a 13.78% compound annual growth rate. (Mordor Intelligence)
Cloudflare's analysis of AI crawler traffic found that about 80% of AI crawling over a recent 12-month period was for model training, versus 18% for search and 2% for user-initiated actions. (Cloudflare)
Cloudflare data shows Google crawls websites about 14 times per referral click it sends back, while OpenAI's crawl-to-referral ratio was roughly 1,700 to 1 in June 2025, illustrating how much scraping now happens without reciprocal traffic. (Cloudflare)

Stage the pipeline and keep the raw HTML

The single most valuable architectural decision: separate crawling from everything downstream, and persist raw HTML (or a compressed archive of it) as the crawl output. Extraction logic changes constantly — you will improve boilerplate removal, fix a parsing bug, or add a new field — and if raw pages are stored, reprocessing is a cheap batch job instead of a fresh crawl that re-burdens the source sites.

My standard layout is a five-stage flow with durable checkpoints between stages: crawl writes raw pages keyed by URL and fetch time; extract produces text plus metadata; clean normalizes it; dedup and filter reduce it; and a versioned export packages the final corpus. Each stage is idempotent and re-runnable, which also makes failures boring — you rerun a stage, not the world. Crawling once and parsing many times is both better engineering and better manners.

Extraction and cleaning decide usable quality

Raw HTML is mostly noise from a training perspective: navigation, footers, cookie banners, share widgets. If that boilerplate survives into the corpus, a fine-tuned model learns to emit it and a RAG index retrieves it. I strip structural chrome tags, extract main content, then normalize unicode and whitespace so downstream hashing behaves.

For high-volume general crawling, purpose-built main-content extractors outperform hand-rolled parsing, but for a known set of sources a targeted BeautifulSoup pass is transparent and debuggable — and transparency matters when you need to explain exactly what went into a training set.

Boilerplate stripping and content fingerprinting

import hashlib
import unicodedata

from bs4 import BeautifulSoup


def extract_text(html: str) -> str:
    soup = BeautifulSoup(html, "html.parser")
    for tag in soup(["script", "style", "nav", "header", "footer", "aside", "form"]):
        tag.decompose()
    text = soup.get_text(separator="\n")
    lines = (line.strip() for line in text.splitlines())
    return "\n".join(line for line in lines if line)


def content_fingerprint(text: str) -> str:
    normalized = unicodedata.normalize("NFKC", text).casefold()
    normalized = " ".join(normalized.split())
    return hashlib.sha256(normalized.encode()).hexdigest()

Deduplication is the highest-leverage stage

Web corpora are full of duplicates: syndicated articles, category pages sharing product blurbs, printer-friendly variants, the same document reached by multiple URLs. Duplicates in training data overweight whatever they repeat, and duplicates in a RAG index waste retrieval slots on redundant chunks. I deduplicate at two levels. Exact duplicates fall to the content fingerprint above — hash the normalized text and keep first-seen.

Near-duplicates need similarity techniques: MinHash over shingles or SimHash both work well at scale, flagging documents that share most of their content despite small edits like dates or bylines. For near-dup clusters I keep the longest or most canonical member. This stage routinely removes a surprisingly large fraction of a raw web crawl, and the corpus is better for it — smaller and cleaner beats bigger and repetitive in every evaluation I have run.

Licensing, consent, and PII are pipeline stages, not afterthoughts

Training-data collection carries obligations beyond ordinary scraping etiquette. Robots.txt and terms of service still govern the crawl itself, but AI use adds more: many sites now publish signals specifically opting out of AI training use, and content licenses (or their absence) determine what you may lawfully include in a corpus. I encode these as machine-checkable pipeline gates — a document that fails a licensing or opt-out check is excluded and the exclusion is logged, so the corpus has an audit trail.

Personal data deserves its own gate. Even incidental PII — emails and phone numbers embedded in page text — should be detected and scrubbed or the document dropped, because a model that memorizes personal data is a liability you cannot easily patch. The legal landscape around scraped training data is genuinely unsettled, so for any commercial model, involve counsel early.

Quality filtering and corpus versioning

After dedup I apply quality filters tuned to the corpus's purpose: minimum and maximum length bounds, language identification, ratio heuristics that catch navigation fragments and link farms, and domain-level allowlists when the project is scoped to trusted sources. For RAG corpora I filter less aggressively — retrieval benefits from coverage — while fine-tuning corpora get stricter gates because every document nudges model behavior.

Finally, version the corpus like code. Every export gets an immutable identifier, a manifest of source documents with fetch timestamps, and the config of every filter that produced it. When someone asks why the model said something odd, or a source requests removal, you can answer precisely which corpus versions contain what — and rebuild without that document from the stored raw pages. That traceability is the difference between a dataset and a pile.

When to hire senior help

Bring in senior help when scraped data feeds production features or pricing decisions, because reliability engineering, compliance review, and change monitoring matter far more than the initial extraction script. An experienced engineer will also steer you toward official APIs, licensed feeds, and terms-of-service-respecting designs that avoid legal exposure and rework. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships Python — Web Scraping projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

✕Scraping without first checking the site's terms of service, robots.txt, and whether an official API or licensed data feed already provides the data lawfully and more reliably
✕Sending unthrottled concurrent requests with no politeness delays, which looks like an attack, gets IP ranges banned, and can disrupt the target site's service
✕Collecting personal data without a lawful basis under GDPR or CCPA, turning a data project into a regulatory liability
✕Coupling parsers tightly to page DOM structure with no output validation or monitoring, so a site redesign silently fills the warehouse with empty or wrong records for weeks

Frequently asked questions

How do I deduplicate web data for LLM training?

Use two passes. First remove exact duplicates by hashing normalized text — unicode-normalize, lowercase, collapse whitespace, then SHA-256 and keep first-seen. Then catch near-duplicates with MinHash or SimHash, which flag documents sharing most of their content despite small differences like dates or bylines. Keep one canonical member per cluster. Deduplication typically removes a large share of a raw crawl and measurably improves training results.

Should I store raw HTML in a scraping pipeline?

Yes, almost always. Compressed raw HTML is cheap to store, and it means every improvement to extraction, cleaning, or filtering becomes a local reprocessing job instead of a new crawl. That saves time, makes experiments reproducible, and reduces load on the source sites — you fetch each page once and parse it as many times as your pipeline evolves. Key pages by URL and fetch timestamp.

Can I legally use scraped data to train AI models?

It is unsettled and jurisdiction-dependent. Beyond standard scraping constraints — robots.txt, terms of service, public non-personal data — AI training raises copyright and licensing questions that courts and regulators are still working through, and many publishers now signal explicit opt-outs from training use. Build licensing and opt-out checks into your pipeline as auditable gates, keep provenance for every document, and involve legal counsel before training a commercial model.

Is web scraping legal for our business?

It depends on what you collect and how: scraping publicly available, non-personal data while respecting terms of service and robots.txt is generally lower risk, while bypassing access controls, violating contracts, or harvesting personal data creates real legal exposure. Get jurisdiction-specific legal advice before building revenue on scraped data, and prefer official APIs or licensed datasets where they exist.

Why do scrapers break so often and what does maintenance cost?

Sites change markup, add bot defenses, and restructure pages; with 51% of web traffic now automated, anti-bot systems are aggressive and constantly updated. Plan for ongoing maintenance as a permanent line item, typically a meaningful fraction of the original build cost per year, plus monitoring that detects breakage within hours instead of weeks.

Should we build scrapers in-house or buy data from a vendor?

For a handful of stable, permissively accessible sources, an in-house Python scraper is cheap and flexible. For large-scale or legally sensitive collection, commercial data providers amortize compliance, proxy infrastructure, and maintenance across many customers, which is why the scraping market is growing at roughly 14% annually. Many teams start with a vendor and only insource once volume justifies it.

Bottom line: Dhairya Senjaliya ships Python — Web Scraping projects worldwide. Book a scoping call at https://dhairyasenjaliya.com/#book-call.