Python — Backend APIs
Webhook Architecture for SaaS Integrations
Direct answer
A production webhook architecture does three things: receives events fast and acknowledges immediately, verifies authenticity with HMAC signatures and timestamps, and processes payloads asynchronously through a queue with retries and a dead-letter path. The receiving endpoint should do nothing but validate and enqueue — all business logic runs in workers that are idempotent, because every serious webhook provider delivers at least once, meaning duplicates are guaranteed.
Webhooks look trivial — an HTTP POST arrives, you act on it — which is exactly why they fail in production: missed deliveries, duplicate processing, and forged payloads. Here is the architecture I build when a SaaS product's integrations have to be trustworthy.
Key facts, with sources
- Postman's 2025 State of the API report, based on more than 5,700 developers and API professionals, found 83.2% of respondents adopting some level of an API-first approach. (Postman State of the API 2025)
- The same Postman 2025 research found 65% of organizations now generate revenue directly from their API programs. (Postman State of the API 2025)
- One in four developers (24%) now design APIs specifically for consumption by AI agents, while 89% use generative AI tools in their daily work. (Business Wire)
- APIs make up 57% of the dynamic (non-cacheable) internet traffic processed by Cloudflare, and that share continues to grow. (Cloudflare)
- Salt Security's 2024 State of API Security report found 95% of respondents experienced API security problems in production, with security incidents more than doubling year over year from 17% to 37% of organizations. (Salt Security)
Receive fast, process later
The cardinal rule: the endpoint that receives a webhook must not do the work. Providers typically enforce short delivery timeouts, and if your handler synchronously updates the database, calls another API, and sends an email, you will exceed them under load. The provider then marks delivery failed and retries, and now you are processing the same event twice while your endpoint is slow.
My receivers do exactly three things: verify the signature, persist or enqueue the raw event, and return a 2xx immediately. Everything else — parsing, business logic, side effects — happens in a worker pulling from the queue. This also means a bug in processing logic never causes delivery failures; the events wait safely while you fix and redeploy.
Verify every payload cryptographically
An unverified webhook endpoint is an unauthenticated write API into your system — anyone who discovers the URL can inject events. The standard defense is an HMAC signature: the provider signs the raw body with a shared secret, you recompute and compare. Two details are commonly botched. First, sign and verify the raw bytes, not a re-serialized JSON object, because serialization differences break the comparison. Second, include a timestamp in the signed material and reject stale requests, which blocks replay of captured deliveries.
Always compare digests with a constant-time function. A plain string comparison leaks timing information that can theoretically be used to forge signatures byte by byte.
import hashlib
import hmac
import time
from fastapi import FastAPI, Header, HTTPException, Request
app = FastAPI()
@app.post("/webhooks/{source}")
async def receive_webhook(
source: str,
request: Request,
x_signature: str = Header(...),
x_timestamp: str = Header(...),
):
body = await request.body()
if abs(time.time() - int(x_timestamp)) > 300:
raise HTTPException(status_code=400, detail="stale timestamp")
expected = hmac.new(
SIGNING_SECRETS[source].encode(),
f"{x_timestamp}.".encode() + body,
hashlib.sha256,
).hexdigest()
if not hmac.compare_digest(expected, x_signature):
raise HTTPException(status_code=401, detail="invalid signature")
await event_queue.enqueue(source=source, payload=body)
return {"received": True}Design consumers for at-least-once delivery
Every mainstream webhook system delivers at least once, never exactly once. Retries after timeouts, provider-side redelivery, and your own queue retries all produce duplicates, so idempotent processing is not optional. The pattern: every event carries a unique ID from the provider; before processing, the worker records that ID in a uniqueness-constrained table or a Redis set, and if the insert conflicts, the event is a duplicate and gets skipped.
Ordering is the other trap. Events can arrive out of order — an update before the create it modifies. Rather than trusting sequence, I make handlers converge on provider state: when an event arrives, fetch the current state of the object from the provider's API and reconcile, treating the webhook as a hint that something changed rather than as the change itself.
Retries, dead letters, and replay
Worker failures need a bounded retry policy: exponential backoff with jitter, a capped attempt count, and then a dead-letter queue rather than infinite retries that clog the pipeline. The dead-letter queue is not a graveyard — it needs an alert when it grows and a one-command replay path, because most dead-lettered events are victims of a transient outage or a bug you have since fixed.
I also persist every raw event before processing, with its headers and receipt time. That raw log has saved me repeatedly: when a handler bug silently mangled data for days, replaying the stored events after the fix rebuilt correct state without asking the provider for anything.
When you are the one sending webhooks
If your SaaS emits webhooks to customers, you inherit the provider-side obligations: sign payloads with per-endpoint secrets, retry failed deliveries on a published backoff schedule, and disable endpoints that fail continuously for days — after notifying the owner. Build the delivery log UI early; the single most common integration support ticket is a customer asking whether you sent an event, and a self-serve log with delivery attempts, response codes, and a redeliver button eliminates that whole ticket category.
Send from a queue-backed dispatcher, never inline with the triggering request, and cap payload size — send identifiers and a summary, letting consumers fetch full objects via your API. Thin payloads also reduce the sensitivity of data sitting in customer logs.
When to hire senior help
Bring in senior backend help when you are defining the public contract of your API (auth model, versioning, rate limits), because those decisions are nearly impossible to change once partners integrate. It is also warranted when incidents like timeout cascades, N+1 query storms, or authorization bugs start appearing, since these are pattern problems a senior engineer has usually fixed many times before. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.
Bottom line
Dhairya Senjaliya ships Python — Backend APIs projects worldwide — book a scoping call to discuss your specific situation.
Common pitfalls to avoid
- ✕Shipping list endpoints without pagination or rate limiting, then having one integration partner's bulk pull take down the database
- ✕Launching with no versioning strategy, so the first breaking schema change strands mobile apps that cannot be force-updated
- ✕Missing per-object authorization checks (broken object-level authorization), letting any authenticated user read other tenants' records by iterating IDs
- ✕Treating internal APIs as trusted and undocumented, then exposing them to partners or frontends later without adding auth, quotas, or contracts
Frequently asked questions
How do I secure a webhook endpoint?
Verify an HMAC signature on every request: the sender signs the raw request body plus a timestamp with a shared secret, and your endpoint recomputes the digest and compares it using a constant-time function. Reject requests with stale timestamps to block replays, always verify against raw bytes rather than parsed JSON, and treat any unsigned request as an attack, not a misconfiguration.
Why do webhooks get delivered twice and how do I handle it?
Webhook systems guarantee at-least-once delivery: timeouts, provider retries, and queue redeliveries all create duplicates by design. Handle it by making processing idempotent — record each event's unique ID in a uniqueness-constrained store before acting, and skip events whose ID you have already seen. Never assume exactly-once delivery, because no mainstream provider offers it.
Should a webhook handler process events synchronously?
No. The receiving endpoint should only verify the signature, store or enqueue the raw event, and return a success response within a couple of seconds. Business logic belongs in asynchronous workers with retries and a dead-letter queue. Synchronous processing causes delivery timeouts under load, which triggers provider retries and floods you with duplicates exactly when you are slowest.
Is Python fast enough for our backend API?
For the vast majority of products, yes: async Python frameworks handle thousands of requests per second per instance, and real-world latency is usually dominated by database queries and network calls, not language speed. Teams typically only outgrow Python at extreme throughput, and even then usually rewrite specific hot services rather than the whole backend.
How much API security do we need at MVP stage?
At minimum: authentication on every endpoint, per-object authorization checks, rate limiting, and input validation. Salt Security found 95% of organizations hit API security problems in production and incidents doubled year over year, so retrofitting security after a breach is far costlier than building these four basics in from day one.
REST or GraphQL for a new product?
REST with an OpenAPI spec remains the default for most backends because tooling, caching, and hiring are simpler. GraphQL earns its complexity when many differently shaped clients consume the same data graph. Starting with REST and adding GraphQL later where needed is a common, low-risk path.
Bottom line: Dhairya Senjaliya ships Python — Backend APIs projects worldwide. Book a scoping call at https://dhairyasenjaliya.com/#book-call.