Python — Backend APIs
Backend API Monitoring with OpenTelemetry
Direct answer
OpenTelemetry gives a Python backend vendor-neutral traces, metrics, and logs through one SDK: you install the FastAPI auto-instrumentation for inbound requests, add instrumentation for your database and HTTP clients so every dependency call becomes a child span, and export via OTLP to whatever backend you choose. The payoff is end-to-end request traces that show exactly where latency lives, plus the freedom to switch observability vendors by changing an exporter instead of rewriting instrumentation.
When an API is slow, averages and CPU graphs tell you almost nothing — you need to see one request's journey through handler, database, and external calls. OpenTelemetry is how I wire that visibility into FastAPI services without marrying a vendor.
Key facts, with sources
- Postman's 2025 State of the API report, based on more than 5,700 developers and API professionals, found 83.2% of respondents adopting some level of an API-first approach. (Postman State of the API 2025)
- The same Postman 2025 research found 65% of organizations now generate revenue directly from their API programs. (Postman State of the API 2025)
- One in four developers (24%) now design APIs specifically for consumption by AI agents, while 89% use generative AI tools in their daily work. (Business Wire)
- APIs make up 57% of the dynamic (non-cacheable) internet traffic processed by Cloudflare, and that share continues to grow. (Cloudflare)
- Salt Security's 2024 State of API Security report found 95% of respondents experienced API security problems in production, with security incidents more than doubling year over year from 17% to 37% of organizations. (Salt Security)
Why OpenTelemetry over a vendor agent
Every observability vendor ships its own agent, and instrumenting with one means your telemetry code is a migration liability — switching vendors later means touching every service. OpenTelemetry breaks that coupling: it is an open standard for generating traces, metrics, and logs, with the vendor reduced to an export destination. Instrument once with the OTel SDK, and moving from one backend to another is an exporter configuration change.
The ecosystem argument matters just as much for Python specifically: auto-instrumentation packages exist for FastAPI, SQLAlchemy, httpx, requests, Redis clients, Celery, and most of the stack I typically deploy, so the majority of useful spans come free rather than from hand-written timing code.
Instrumenting FastAPI in a few lines
The setup has three parts: a tracer provider identifying the service, a span processor batching spans for export, and the FastAPI instrumentor wrapping the app so every request produces a server span with method, route, and status attributes. The batch processor matters in production — it exports asynchronously in the background, keeping telemetry off the request path.
Set the service name deliberately; it is how every trace, dashboard, and alert will identify this deployable, and renaming it later breaks saved queries.
from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
OTLPSpanExporter,
)
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
provider = TracerProvider(
resource=Resource.create({"service.name": "orders-api"})
)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)
app = FastAPI()
FastAPIInstrumentor.instrument_app(app)Tracing the dependencies, not just the handler
A server span alone tells you a request took eight hundred milliseconds; the value arrives when child spans show that seven hundred of them were one SQL query. Add the SQLAlchemy instrumentation and every query becomes a span with its statement; add httpx or requests instrumentation and every outbound API call appears with its URL and status; the Redis and Celery instrumentations extend the same trace into caches and background jobs.
Context propagation is the quiet superpower: the OTel SDK injects trace headers into outbound HTTP calls automatically, so when service A calls service B, both services' spans join one distributed trace. In a microservices incident, that single stitched timeline replaces an hour of correlating timestamps across log systems.
Custom spans for business context
Auto-instrumentation describes infrastructure; it cannot know that a request was a checkout for a high-value cart. I add a thin layer of custom spans around meaningful business operations, with attributes that make traces searchable by questions people actually ask — find slow checkouts, find requests for this account, find orders above a size threshold.
Be deliberate about attribute hygiene: no secrets, no raw personal data, and keep cardinality sane — attribute values like user IDs are fine on spans but disastrous as metric labels.
from opentelemetry import trace
tracer = trace.get_tracer("orders")
@app.post("/orders")
async def create_order(payload: OrderIn):
with tracer.start_as_current_span("orders.create") as span:
span.set_attribute("order.item_count", len(payload.items))
span.set_attribute("order.channel", payload.channel)
order = await order_service.create(payload)
span.set_attribute("order.id", str(order.id))
return orderSampling, cost, and what to alert on
Tracing every request in a busy service is often unnecessary and always expensive — telemetry bills scale with span volume. Head sampling, deciding at trace start with a fixed probability, is simple and built into the SDK. The more useful strategy is tail sampling at a collector: keep every error and every slow trace, sample the boring successes, so the traces you pay for are the ones you will actually read. Routing telemetry through an OpenTelemetry Collector also centralizes redaction, batching, and exporter credentials outside your application.
For alerting, I keep the signal list short: p95 and p99 latency per route, error rate per route, and saturation of whatever actually limits you — database pool, queue depth, memory. Alerts fire on symptoms users feel; traces then explain the cause.
When to hire senior help
Bring in senior backend help when you are defining the public contract of your API (auth model, versioning, rate limits), because those decisions are nearly impossible to change once partners integrate. It is also warranted when incidents like timeout cascades, N+1 query storms, or authorization bugs start appearing, since these are pattern problems a senior engineer has usually fixed many times before. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.
Bottom line
Dhairya Senjaliya ships Python — Backend APIs projects worldwide — book a scoping call to discuss your specific situation.
Common pitfalls to avoid
- ✕Shipping list endpoints without pagination or rate limiting, then having one integration partner's bulk pull take down the database
- ✕Launching with no versioning strategy, so the first breaking schema change strands mobile apps that cannot be force-updated
- ✕Missing per-object authorization checks (broken object-level authorization), letting any authenticated user read other tenants' records by iterating IDs
- ✕Treating internal APIs as trusted and undocumented, then exposing them to partners or frontends later without adding auth, quotas, or contracts
Frequently asked questions
What does OpenTelemetry actually give me for a Python API?
Three signals through one vendor-neutral SDK: distributed traces showing each request's path through handlers, database queries, and external calls; metrics like request duration and counts; and correlated logs. Auto-instrumentation packages for FastAPI, SQLAlchemy, httpx, Redis, and Celery generate most spans automatically, and you can switch observability vendors by changing the exporter rather than re-instrumenting your code.
Does OpenTelemetry slow down a FastAPI application?
The overhead is modest when configured properly: spans are cheap to create, and the batch span processor exports them asynchronously off the request path. The real costs to manage are network egress and vendor ingestion pricing, which scale with span volume. Sampling — keeping all errors and slow traces while sampling routine successes — controls cost without losing the traces that matter.
Should I sample traces or record every request?
Record everything early on, when traffic is low and every trace is potentially useful. As volume grows, move to tail sampling at a collector: retain all error traces and unusually slow traces, and keep only a percentage of fast successes. That preserves diagnostic power for incidents while cutting telemetry spend, since the discarded traces are the ones nobody would ever read.
Is Python fast enough for our backend API?
For the vast majority of products, yes: async Python frameworks handle thousands of requests per second per instance, and real-world latency is usually dominated by database queries and network calls, not language speed. Teams typically only outgrow Python at extreme throughput, and even then usually rewrite specific hot services rather than the whole backend.
How much API security do we need at MVP stage?
At minimum: authentication on every endpoint, per-object authorization checks, rate limiting, and input validation. Salt Security found 95% of organizations hit API security problems in production and incidents doubled year over year, so retrofitting security after a breach is far costlier than building these four basics in from day one.
REST or GraphQL for a new product?
REST with an OpenAPI spec remains the default for most backends because tooling, caching, and hiring are simpler. GraphQL earns its complexity when many differently shaped clients consume the same data graph. Starting with REST and adding GraphQL later where needed is a common, low-risk path.
Bottom line: Dhairya Senjaliya ships Python — Backend APIs projects worldwide. Book a scoping call at https://dhairyasenjaliya.com/#book-call.