Web Scraping for Price Monitoring Products

Key facts, with sources

• The 2025 Imperva Bad Bot Report found automated traffic surpassed human activity for the first time in a decade, accounting for 51% of all web traffic. (Imperva)
• Bad bots alone made up 37% of all internet traffic in 2024, up from 32% the year before, according to the 2025 Imperva Bad Bot Report. (Business Wire)
• Mordor Intelligence sizes the web scraping market at $1.03 billion in 2025, projected to reach $2.23 billion by 2031 at a 13.78% compound annual growth rate. (Mordor Intelligence)
• Cloudflare's analysis of AI crawler traffic found that about 80% of AI crawling over a recent 12-month period was for model training, versus 18% for search and 2% for user-initiated actions. (Cloudflare)
• Cloudflare data shows Google crawls websites about 14 times per referral click it sends back, while OpenAI's crawl-to-referral ratio was roughly 1,700 to 1 in June 2025, illustrating how much scraping now happens without reciprocal traffic. (Cloudflare)

Product identity is the actual hard problem

Anyone can extract a price. The hard part is knowing which product it belongs to, durably. Within one source, you need a stable key — a SKU, a product ID from the URL structure, or a data attribute — that survives redesigns and does not collide across variants. Sizes, colors, and bundle options frequently share a page while carrying different prices, and conflating them produces phantom price swings that destroy user trust in your alerts.

Across sources the problem gets harder: matching the same physical product between retailers. Identifiers like manufacturer part numbers or barcode-style codes are gold when present. When they are not, fuzzy matching on normalized titles plus attributes (brand, model, capacity) gets you most of the way, but I always keep match confidence as a stored field and route low-confidence matches to human review. A wrong match silently corrupts every comparison downstream.

Extract carefully, normalize ruthlessly

Prices on real pages arrive as strings with currency symbols, thousands separators that differ by locale, strikethrough list prices next to sale prices, unit prices, and financing offers. Extraction must target the right element — usually structured data attributes or the offer markup many retailers embed — and normalization must convert everything to a canonical form, typically minor units (cents) plus an explicit currency code. Floating point has no place in a price pipeline; use integers or Decimal end to end.

import re
from decimal import Decimal, InvalidOperation

PRICE_RE = re.compile(r"\d[\d,]*(?:\.\d{1,2})?")


def parse_price(raw: str) -> Decimal | None:
    """US-style formats; add per-source locale rules as needed."""
    match = PRICE_RE.search(raw.replace("\xa0", " "))
    if not match:
        return None
    try:
        return Decimal(match.group().replace(",", ""))
    except InvalidOperation:
        return None


def meaningful_change(
    old: Decimal, new: Decimal, min_pct: Decimal = Decimal("0.005")
) -> bool:
    if old <= 0:
        return new > 0
    return abs(new - old) / old >= min_pct

Schedule by how fast prices actually move

A uniform crawl frequency wastes requests on stable products and misses movement on volatile ones. I tier sources and products: fast-moving categories or items with recent change history get checked more often; long-tail items that have not moved in months drop to a slow cadence. The tier assignment itself can be data-driven — promote a product after a detected change, demote it after a quiet period — which concentrates your polite request budget where it earns alerts.

This matters for compliance as much as cost. A price monitor that hits every product page hourly is exactly the traffic pattern that gets scrapers blocked and sours the relationship with sources. Tiered scheduling plus per-domain rate limiting keeps aggregate load modest, and honoring each source's robots.txt and crawl-delay is table stakes for a commercial product whose existence depends on continued access.

Change detection that users can trust

The alert is the product, so false positives are the existential risk. Common noise sources: extraction picking up the list price instead of the sale price after a layout tweak, currency or locale flips served by geo-targeting, per-unit prices displacing total prices, and out-of-stock states where some sites show placeholder values. I defend with layered checks — a minimum percentage threshold like the snippet above, sanity bounds that flag implausible jumps for review instead of alerting, and a rule that a change must persist across two consecutive fetches before users hear about it.

Store the full price history as a time series, never just current values. History powers the features customers actually pay for — trend charts, lowest-price-in-period claims, alert digests — and it is your forensic record when someone disputes an alert. Every stored point should carry fetch timestamp, source, extraction version, and raw captured string alongside the normalized value.

Compliance is a product feature, not a tax

A price monitoring business is a repeated game with its sources, so I design compliance in from the start. Each source gets an onboarding review: does robots.txt permit the paths we need, do the terms of service allow automated access, is there an official product or affiliate API that provides prices legitimately? Many retailers actively want their prices distributed through affiliate programs — that channel is often better data and zero legal ambiguity, and I default to it wherever it exists.

For sources that are scraped, the crawler identifies itself, rate limits per domain, and backs off on any distress signal. Only public pages, never personal data, and no circumvention of blocks — if a source objects, they come out of the product. Prospects sometimes see this as competitive weakness; I see it as the only version of this product that is still operating in five years.

When to hire senior help

Bring in senior help when scraped data feeds production features or pricing decisions, because reliability engineering, compliance review, and change monitoring matter far more than the initial extraction script. An experienced engineer will also steer you toward official APIs, licensed feeds, and terms-of-service-respecting designs that avoid legal exposure and rework. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships Python — Web Scraping projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Scraping without first checking the site's terms of service, robots.txt, and whether an official API or licensed data feed already provides the data lawfully and more reliably
• ✕Sending unthrottled concurrent requests with no politeness delays, which looks like an attack, gets IP ranges banned, and can disrupt the target site's service
• ✕Collecting personal data without a lawful basis under GDPR or CCPA, turning a data project into a regulatory liability
• ✕Coupling parsers tightly to page DOM structure with no output validation or monitoring, so a site redesign silently fills the warehouse with empty or wrong records for weeks

Frequently asked questions

Sources

• Imperva
• Business Wire
• Mordor Intelligence
• Cloudflare
• Cloudflare