Deploying ML Models with FastAPI

Key facts, with sources

• A 2024 Gartner survey found that on average only 48% of AI projects make it into production, and it takes 8 months to go from AI prototype to production. (Gartner)
• S&P Global's Voice of the Enterprise survey of 1,006 professionals found the share of companies abandoning most of their AI initiatives before production jumped from 17% to 42% year over year, with an average 46% of proofs of concept scrapped before production. (S&P Global Market Intelligence)
• RAND identifies underinvestment in deployment infrastructure as one of five root causes behind an AI project failure rate exceeding 80%, twice the rate of non-AI IT projects. (RAND Corporation)
• CNCF's annual cloud native survey found only 7% of organizations deploy ML models daily while 47% deploy only occasionally, indicating early deployment-automation maturity. (CNCF Annual Cloud Native Survey)
• CNCF reports 66% of organizations hosting generative AI models use Kubernetes to manage some or all of their inference workloads, with Kubernetes production use reaching 82% in the 2025 survey. (CNCF)

Load once, serve many: the lifespan pattern

The most common deployment mistake I see in audits is loading the model inside the endpoint — adding seconds of latency per request or, worse, exhausting memory under concurrency. Models load once, at startup, in FastAPI's lifespan context manager, and live for the process lifetime.

Loading by registry alias rather than a baked-in file path is what makes rollback a config change instead of a redeploy — the serving code never knows which version it's running until startup.

from contextlib import asynccontextmanager

import joblib
from fastapi import FastAPI
from fastapi.concurrency import run_in_threadpool
from pydantic import BaseModel, Field

models = {}


@asynccontextmanager
async def lifespan(app: FastAPI):
    models["churn"] = joblib.load("artifacts/churn-v12.joblib")
    yield
    models.clear()


app = FastAPI(lifespan=lifespan)


class ChurnFeatures(BaseModel):
    tenure_months: int = Field(ge=0, le=600)
    monthly_spend: float = Field(ge=0)
    support_tickets_90d: int = Field(ge=0)


class Prediction(BaseModel):
    churn_probability: float
    model_version: str = "churn-v12"


@app.post("/predict", response_model=Prediction)
async def predict(features: ChurnFeatures):
    row = [[features.tenure_months, features.monthly_spend,
            features.support_tickets_90d]]
    # CPU-bound inference must not block the event loop
    proba = await run_in_threadpool(models["churn"].predict_proba, row)
    return Prediction(churn_probability=round(float(proba[0][1]), 4))

Pydantic is your data contract — use it aggressively

Models fail silently on bad inputs: a missing feature becomes a zero, a swapped unit becomes a confident wrong answer. Pydantic turns those silent failures into 422 responses with exact field-level errors. Constrain everything you know: ranges on numeric features, enums for categoricals, and reject unknown fields so a renamed upstream column breaks loudly at the API boundary instead of quietly in the predictions.

This is also where training-serving skew gets caught: the Pydantic model documents exactly what the model expects, and it lives in the same repo as the training feature definitions. When the two drift, the schema mismatch shows up in a failing request, not in a slow degradation of accuracy that takes a quarter to notice.

Concurrency: the event-loop trap

FastAPI's async is a footgun for ML serving if misunderstood. An async def endpoint that calls model.predict() directly blocks the entire event loop for the duration of inference — under load, every request queues behind the current prediction and p99 latency explodes. Two correct options: declare the endpoint with plain def (FastAPI runs it in a threadpool automatically), or keep async def and wrap inference in run_in_threadpool as above.

For heavier models, cap concurrent inference with a semaphore sized to your CPU cores — accepting unlimited concurrent predictions just trades queue position for memory pressure. And for batch-friendly models, a micro-batching layer that collects requests for a few milliseconds and predicts them together can multiply throughput, at the cost of real complexity — measure before adding it.

Version, health, and the rollback story

Return the model version in every prediction response and expose a health endpoint that actually exercises the model with a known input — a process that's up but serving a corrupted artifact should fail its health check, not its users. Log every prediction with version, latency, input hash, and confidence: that log is your drift-detection dataset and your incident forensics.

Pair the serving layer with a registry (MLflow or equivalent) and load by alias so promoting or rolling back a model is a registry operation plus a rolling restart — no image rebuild. On infrastructure: a CPU model in a container on Railway, Fly, or ECS covers most real workloads; reach for GPU serving infrastructure only when the model actually needs it, because the operational cost is permanent.

When to hire senior help

The pilot-to-production gap is where nearly half of AI projects die, so senior help is most leveraged at the point where a validated prototype needs a serving architecture, rollout plan, and monitoring. An experienced engineer can usually take a working model to a canaried production deployment far faster than a team learning serving infrastructure for the first time, avoiding the 8-month average lag. If your stack includes React Native + Python + AI, a senior engineer who owns the full product beats coordinating multiple juniors.

Bottom line

Dhairya Senjaliya ships ML — Model Deployment projects worldwide — book a scoping call to discuss your specific situation.

Common pitfalls to avoid

• ✕Treating deployment as a final step instead of designing the serving path early, which is how prototypes stall for the 8-month average Gartner measures
• ✕Wrapping a notebook in a Flask endpoint with no load testing, then discovering latency and memory limits under real traffic
• ✕Deploying a new model with no shadow mode or canary phase, so the first regression is discovered by customers
• ✕Rebuilding features at serving time with different code than training used, producing predictions that never match offline evaluation

Frequently asked questions

Sources

• Gartner
• S&P Global Market Intelligence
• RAND Corporation
• CNCF Annual Cloud Native Survey
• CNCF