Source: the original project’s deployment note, re-homed into the MLOps section and trimmed so drift/monitoring depth lives in its own note (monitoring & edge). Getting a trained model out of a notebook and operating it: serving predictions, versioning, safe rollout, and retraining. The infra underneath is DevOps (Docker/k8s/observability) — this is the ML-specific layer on top.
Serving — getting predictions out of a model
At minimum, serving wraps a trained model in something that accepts a request, runs inference (not training — see why these are separate), and returns a response:
from fastapi import FastAPI
import torch
app = FastAPI()
model = torch.load("model.pt")
model.eval()
@app.post("/predict")
def predict(features: list[float]):
with torch.no_grad():
output = model(torch.tensor(features).unsqueeze(0))
return {"prediction": output.item()}For higher throughput or latency-sensitive needs, dedicated serving frameworks (TorchServe, NVIDIA Triton, BentoML, or a managed cloud endpoint) handle request batching, GPU scheduling, and autoscaling that a hand-rolled wrapper doesn’t. Batch (offline, score a big dataset on a schedule) vs online/real-time (low-latency per-request) serving are different problems — pick per use case.
Versioning — track what served what
A production ML system must record which exact model version served which prediction, tied to the data and code version that produced it (model registry) — essential for debugging a bad prediction after the fact and for rollback when a new model underperforms in production despite looking better offline.
A/B testing and gradual rollout
Don’t swap the whole production model at once. Route a small % of real traffic to the new version and compare real-world performance before full rollout — the same canary/blue-green ideas as CI/CD deployment strategies. This matters more in ML than in ordinary software because offline metrics (evaluation) don’t perfectly predict live performance, and a model’s decisions change user behavior in ways only visible on live traffic (feedback loops).
Retraining pipelines
Models decay under drift, so production ML needs a pipeline to retrain on fresh data, evaluate the new model against the current one, and promote only if genuinely better — rather than a one-time training event assumed valid forever. This closes the MLOps loop (fundamentals); orchestration lives in data pipelines.
The gotcha: training-serving skew
The classic silent killer — if serving-time preprocessing doesn’t exactly match training-time (a different scaling formula, a feature computed slightly differently), the model receives different-looking inputs than it trained on and degrades in a way that’s easy to miss. The standard defense is a feature store computing features once for both paths. And deploying with no monitoring at all means the first sign of a problem is a business outcome (missed fraud, bad recommendations), not a metric you caught early.
Related
- Monitoring & Edge — catching the drift that triggers retraining
- Experiment Tracking & Pipelines — model registry, feature stores, orchestration
- Training From Scratch vs Fine-Tuning — what produced the model being served
- DevOps — the operational disciplines this borrows from
Software engineering
Infrastructure & security
Computer science
Hardware & engineering