ML Platform Design

A company has 30 ML models in production: a fraud classifier, a recommender, a search ranker, a churn predictor, a dozen of risk scores, several LLM-based summarizers. Each was built by a different team. One uses Jupyter notebooks for training; another uses a custom Airflow DAG. One serves through a Flask app on EC2; another through SageMaker; another through a Kubernetes deployment of FastAPI. Each team rebuilt monitoring, deployment, model versioning, and feature pipelines from scratch. Now a regulator asks: “Show me which version of which model produced the prediction that denied this loan three months ago, and which training data it used.” Nobody can answer. An ML platform is the shared infrastructure that turns ad-hoc model development into a repeatable, governed, observable process — so that training, serving, monitoring, and rollout look the same regardless of which team or model.

What an ML Platform Provides

flowchart LR
    subgraph "Without ML Platform"
        Team1[Team A
Jupyter + Flask]
        Team2[Team B
SageMaker]
        Team3[Team C
K8s + custom serving]
        Team1 -.-> Chaos[Inconsistent monitoring
no shared lineage
no governance
30× operational burden]
        Team2 -.-> Chaos
        Team3 -.-> Chaos
    end

    subgraph "With ML Platform"
        Pipe[Training pipelines]
        Reg[Model registry]
        Serve[Serving infra]
        Mon[Monitoring]
        Pipe & Reg & Serve & Mon --> Out[Consistent: deploy,
rollback, audit, monitor
across all 30 models]
    end

A platform provides shared infrastructure for the lifecycle stages every model goes through:

Training Pipeline

A reproducible, automated path from data to a trained model artifact.

flowchart LR
    Source[(Data sources
warehouse / lake)] --> Ingest[Ingest
filter, sample]
    Ingest --> FE[Feature engineering
via feature store]
    FE --> Split[Train / val / test split]
    Split --> Train[Distributed training
GPU cluster]
    Train --> Eval[Evaluate
metrics, fairness, slices]
    Eval -->|pass thresholds| Reg[Register in
model registry]
    Eval -->|fail| Stop[Stop & alert]

Orchestration

Reproducibility Requirements

A trained model is reproducible only if all of these are versioned:

For a single training run, the registry records:
  - Code version (git commit hash + branch)
  - Data version (data lake snapshot or query as-of date)
  - Feature definitions version (from feature store registry)
  - Hyperparameters
  - Random seed
  - Container image (ML libraries, CUDA version, OS)
  - Hardware (GPU type, count)
  - Output: model artifact + evaluation metrics + run logs

Without all of these, “rerun this training a year from now and get the same model” is impossible — and impossible reproduction means impossible debugging.

Experiment Tracking

The pattern is consistent: log every experiment’s hyperparameters, metrics, and artifacts; compare runs visually; promote winners.

Model Registry

The single source of truth for every model artifact in the organization.

flowchart TB
    subgraph "Model Registry"
        M1[fraud-classifier
v3 → staging
v2 → production
v1 → archived]
        M2[search-ranker
v8 → production
v7 → archived]
        M3[churn-predictor
v5 → canary
v4 → production]
    end

    subgraph "For each model version"
        Art[Artifact: weights, config
S3 / GCS]
        Meta[Metadata: hyperparams,
metrics, training data hash]
        Lin[Lineage: data sources,
feature versions, code commit]
        App[Approvals: who reviewed,
signed off, when]
    end

    M1 -.-> Art & Meta & Lin & App

What a Registry Tracks

Promotion Workflow

Newly trained → staging
  ↓ (offline metrics pass + automated checks pass)
staging → canary (1–5% of traffic)
  ↓ (online metrics match offline; no guardrail regression)
canary → production (full traffic)
  ↓ (replaced by next version)
production → archived (kept for audit, not served)

Every transition is logged with a reason and approver, generating an audit trail. For regulated domains (lending, healthcare, hiring) this audit trail is a hard requirement.

Tools

Model Serving

The runtime that turns a registered model into served predictions. Three serving modes cover almost every production model.

Online Serving (Real-Time Inference)

Per-request prediction behind a REST or gRPC endpoint. Used for recommendations, fraud scoring, search, ads — anything user-facing.

flowchart LR
    Client --> LB[Load Balancer]
    LB --> R1[Model Replica 1
GPU/CPU]
    LB --> R2[Model Replica 2]
    LB --> R3[Model Replica N]
    R1 & R2 & R3 --> FS[(Feature Store
online)]
    R1 & R2 & R3 --> Logs[Prediction logs
Kafka]

Batch Serving (Offline Scoring)

Score millions of records on a schedule. Used for: nightly user segment refreshes, lookalike audience generation, ML-derived columns in the warehouse.

# Conceptual: Spark batch scoring
df = spark.read.parquet("s3://data/users/")
df_with_features = df.join(feature_table, on="user_id")
predictions = model.transform(df_with_features)
predictions.write.parquet("s3://predictions/churn_score/dt=2025-05-02/")

Streaming Serving (Per-Event Inference)

A streaming consumer applies the model to each event as it arrives. Used for: real-time fraud, content moderation triage, real-time personalization.

flowchart LR
    Kafka[Kafka
events] --> Flink[Flink job
loads model
scores per event]
    Flink --> KafkaOut[Kafka
scored events]
    Flink --> FS[(Feature store
per-event features)]

Choosing the Right Mode

Deployment Strategies

Replacing a production model is a high-risk operation. Four strategies, in order of safety:

1. Shadow Mode

flowchart LR
    Request --> Old[Old model
returns response]
    Request -.->|"copy"| New[New model
response logged, not returned]
    Old --> User[User sees response]
    New --> Compare[Offline comparison
old vs new predictions]

Run the new model in parallel; discard its output. Log predictions and compare distributions to the production model. Catches:

• Latency regressions (new model too slow)
• Distributional drift (new model predicts very differently than old)
• Crashes / errors that wouldn’t appear in offline eval

Costs 2× compute but is the safest first step. Run for a few days before any user-facing test.

2. Canary

Route 1–5% of traffic to the new model; the rest to the old one. Monitor:

• Online metrics (CTR, conversion, revenue per request) match offline expectations
• Guardrails (latency p99, error rate, fairness slices) don’t regress
• Sample ratio is what was configured (sanity check)

Gradually increase: 5% → 10% → 25% → 50% → 100% over hours or days, depending on traffic volume needed for statistical significance.

3. A/B Test

Like canary, but with a longer evaluation window designed for hypothesis testing on a north-star metric. Typical: 50/50 split for 1–2 weeks.

4. Rollback

Every promotion path must have a fast rollback. The previous model version stays warm in the registry; a single config change returns serving traffic to it.

Monitoring: Three Kinds of Drift

Models in production silently degrade. Detecting why requires monitoring three different things.

flowchart TB
    subgraph "What can go wrong"
        D1[Data drift
input distribution
changes]
        D2[Concept drift
relationship between
inputs and outputs
changes]
        D3[Performance drift
actual labels show
accuracy is dropping]
    end

    D1 -->|"early warning"| Alert
    D2 -->|"requires labels"| Alert
    D3 -->|"ground truth"| Alert

1. Data Drift (Input Distribution Drift)

The features the model sees at inference time differ from what it trained on. Doesn’t require labels — just compare distributions.

Example: PSI on `user_country` feature
  Training: US 60%, IN 20%, BR 10%, other 10%
  Production (week 4): US 30%, IN 50%, BR 5%, other 15%
  PSI = sum over buckets of (prod% - train%) × ln(prod% / train%) ≈ 0.4
  Alert: model trained mostly on US users now serves mostly IN users

2. Concept Drift (Output Drift)

The relationship between features and the label has changed — even with the same input distribution, the right answer is now different. Examples: a fraud pattern evolves, a price-elasticity model sees post-pandemic shopping behavior, an LLM’s expected response changes after a UX update.

Detection requires labels, which usually arrive late. Proxies:

• Drift in the predicted label distribution (e.g., fraud rate predicted by model rises 3×)
• Drift in prediction confidence (model becomes less confident)
• A/B-tested online metrics (CTR, conversion) decline

3. Performance Drift (Ground Truth Available)

Once labels arrive (clicks, fraud confirmations, returns), measure accuracy directly.

Alerting Discipline

End-to-End Architecture

flowchart TB
    subgraph "Sources"
        Data[(Data warehouse /
data lake)]
        Stream[Event streams
Kafka]
    end

    subgraph "Feature Platform"
        FS[(Feature store
online + offline)]
    end

    subgraph "Training"
        Pipe[Training pipeline
Kubeflow / Airflow]
        Track[Experiment tracking
MLflow]
        Pipe --> Track
    end

    subgraph "Registry"
        Reg[Model registry
versions + lineage
+ approvals]
    end

    subgraph "Serving"
        Online[Online serving
KServe / Triton]
        Batch[Batch scoring
Spark]
        Stream2[Streaming scoring
Flink]
    end

    subgraph "Observability"
        Mon[Drift + perf
monitoring]
        Logs[Prediction logs
+ feature logs]
    end

    Data --> FS
    Stream --> FS
    FS --> Pipe
    Pipe --> Reg
    Reg --> Online
    Reg --> Batch
    Reg --> Stream2

    Online --> Logs
    Batch --> Logs
    Stream2 --> Logs

    Logs --> Mon
    FS --> Mon
    Mon -->|retrain trigger| Pipe

The closed loop matters: monitoring feeds back into retraining. When drift is detected, a new training run is automatically triggered (or queued for human approval), and the cycle repeats.

Build vs. Buy

Rule of thumb: for the first 5–10 models, lean entirely on managed/open-source pieces wired together. Build custom platform components only when a specific pain point can’t be solved by configuration of existing tools.

Production Examples

Test Your Understanding