Last week, a ML team in Barcelona rolled out their recommendation model on Kubernetes. Everything was perfect locally; it passed the tests, and staging was a success. However, three days later in production, the system collapsed: 4-second latencies, pods restarting in loops, and GKE costs skyrocketing by 340%. The issue? It wasn't the model. It was the architecture.
Kubernetes doesn't automatically optimize your AI models. If you just wrap TensorFlow Serving in a container and launch a basic Deployment, you are setting up a bottleneck ready to explode under real traffic. This article breaks down the architecture that Uber, Spotify, and Airbnb use to serve millions of predictions per hour without a hitch.
The Mistake 80% of Teams Make When Deploying Models in Kubernetes
Most tutorials say: "Package your model in a container, create a Deployment, expose a Service, and you're good to go." It's technically true, but operationally, it's a disaster.
The key issue is that TensorFlow loads the entire model graph into memory at startup. If your model is 2GB and your pod restarts every time it gets 100 concurrent requests, Kubernetes will take 30-40 seconds to restart each time just to load the model. Meanwhile, your users experience timeouts.
Cold Start: The Invisible Killer
Pods in production restart for various reasons: updates, rebalancing, memory errors, failed health checks. Each restart means loading the model from scratch.
Companies that scale implement model warming right from the container. Before Kubernetes marks the pod as "Ready," the container runs dummy predictions to fully load the graph and main operations on GPU/CPU.
# warm_model.py
import tensorflow as tf
import numpy as np
import time
def warm_model(model_path, input_shape, iterations=50):
"""
Pre-warm the model before Kubernetes marks it as ready
"""
model = tf.saved_model.load(model_path)
infer = model.signatures['serving_default']
# Generate realistic dummy inputs
dummy_input = tf.constant(
np.random.randn(*input_shape).astype(np.float32)
)
print(f"Warming model with {iterations} iterations...")
start = time.time()
for i in range(iterations):
_ = infer(dummy_input)
if i % 10 == 0:
print(f"Completed {i} warm-up iterations")
elapsed = time.time() - start
print(f"Model warmed in {elapsed:.2f}s. Ready for traffic.")
return model
if __name__ == "__main__":
model = warm_model(
model_path="/models/recommender/1",
input_shape=(1, 128), # Adjust to your model
iterations=50
)
This script should run before your inference server is ready. In the Dockerfile:
FROM tensorflow/serving:2.13.0-gpu
COPY ./model /models/recommender/1
COPY ./warm_model.py /scripts/warm_model.py
# Run warming before starting TF Serving
CMD python /scripts/warm_model.py && \
tensorflow_model_server \
--rest_api_port=8501 \
--model_name=recommender \
--model_base_path=/models/recommender
But there's a detail: if warming takes 40 seconds and your readinessProbe has an initialDelaySeconds of 10, Kubernetes will kill the pod before it finishes warming up. Proper probe configuration is crucial.
Resource Configuration: The Balance Google Doesn't Document Well
Allocating resources for AI models is not the same as a REST API. TensorFlow models have specific consumption patterns that break conventional rules.
CPU vs GPU: When GPU Isn't the Answer
Interestingly, not all models benefit from GPU for inference. If your model is small (<500MB) and processes individual requests instead of large batches, an optimized CPU can be 3x more cost-effective and offer comparable latencies.
Uber discovered this in 2024 when it migrated recommendation models from GPU instances to high-performance CPUs. The overhead of moving data between CPU and GPU for small requests nullified the advantage.
The rule of thumb:
- Use GPU if you process large batches (>32 inputs at once) or your model has >1B parameters.
- Use CPU for small models with ultra-low-latency inference (<50ms) on individual requests.
In Kubernetes, this translates to different resource configurations:
# For GPU models
resources:
requests:
memory: "8Gi"
cpu: "2"
nvidia.com/gpu: 1
limits:
memory: "12Gi"
cpu: "4"
nvidia.com/gpu: 1
# For CPU-optimized models
resources:
requests:
memory: "4Gi"
cpu: "4"
limits:
memory: "6Gi"
cpu: "8"
The Memory Limits Problem
TensorFlow tends to reserve memory aggressively. By default, it will occupy all possible GPU memory, and on CPU, it will try to use all the RAM to cache operations.
If you define limits.memory: "8Gi" but your model and TensorFlow runtime really need 9Gi under load, the pod will be OOMKilled (Out Of Memory Killed) and restarted. This can create a death loop: restart β cold start β max load β OOM β restart.
Spotify solved this for its music recommendation models with:
env:
- name: TF_GPU_MEMORY_FRACTION
value: "0.7" # Reserve only 70% of available GPU memory
- name: TF_FORCE_GPU_ALLOW_GROWTH
value: "true" # Allow gradual growth, don't reserve all at startup
This approach prevents TensorFlow from hogging unnecessary resources and provides room for traffic spikes.
Smart Autoscaling: HPA with Custom Metrics
The standard Horizontal Pod Autoscaler (HPA) scales based on CPU or memory. For AI models, these metrics can be misleading.
A model at 90% CPU could be performing well thanks to a large batch, while another at 40% might be saturated processing 500 small requests sequentially.
Metrics That Matter
Those scaling AI models in production use application-level metrics:
- Request queue depth: Number of requests in the queue
- P95 latency: 95th percentile latency
- Throughput: Predictions per second per pod
Implementation with Prometheus and custom metrics:
# metrics_server.py with Flask + prometheus_client
from prometheus_client import Counter, Histogram, Gauge
from flask import Flask, request, jsonify
import tensorflow as tf
import time
app = Flask(__name__)
# Custom metrics
REQUEST_COUNT = Counter('model_requests_total', 'Total requests')
REQUEST_LATENCY = Histogram('model_request_duration_seconds', 'Inference latency')
QUEUE_SIZE = Gauge('model_queue_depth', 'Requests in queue')
model = None
request_queue = []
@app.route('/v1/predict', methods=['POST'])
def predict():
global request_queue
QUEUE_SIZE.set(len(request_queue))
REQUEST_COUNT.inc()
start = time.time()
# Your inference logic
data = request.json
predictions = run_inference(data)
duration = time.time() - start
REQUEST_LATENCY.observe(duration)
return jsonify(predictions)
def run_inference(data):
# Actual inference implementation
pass
HPA configuration with these metrics:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: model-hpa
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: tf-model
minReplicas: 3
maxReplicas: 20
metrics:
- type: Pods
pods:
metric:
name: model_queue_depth
target:
type: AverageValue
averageValue: "10" # Scale when queue > 10 per pod
- type: Pods
pods:
metric:
name: model_request_duration_seconds
target:
type: AverageValue
averageValue: "0.2" # Scale when P50 > 200ms
behavior:
scaleDown:
stabilizationWindowSeconds: 300 # Wait 5min before scaling down pods
policies:
- type: Percent
value: 50
periodSeconds: 60
The key is in stabilizationWindowSeconds. ML models have expensive warmup costs, so aggressive scaling down can destroy performance. Spotify uses between 5 and 10 minutes of stabilization before removing pods.
Dynamic Batching: The Technique That Cuts Latency by 60%
One of the best-kept secrets to serving millions of predictions is dynamic batching. You accumulate multiple requests in milliseconds and process them as a batch.
TensorFlow Serving has this feature, but it's disabled by default. Configuring it correctly can reduce average latency by up to 60% under high load.
# batching_config.txt
max_batch_size { value: 32 }
batch_timeout_micros { value: 5000 } # Max wait of 5ms
max_enqueued_batches { value: 100 }
num_batch_threads { value: 4 }
pad_variable_length_inputs: true
Mount this file in the container:
volumes:
- name: batching-config
configMap:
name: tf-batching-config
volumeMounts:
- name: batching-config
mountPath: /config/batching_config.txt
subPath: batching_config.txt
And start TensorFlow Serving with:
tensorflow_model_server \
--model_name=recommender \
--model_base_path=/models/recommender \
--enable_batching=true \
--batching_parameters_file=/config/batching_config.txt
The Latency Tradeoff
Dynamic batching adds latency up to batch_timeout_micros. If your SLA requires less than 50ms, a 5ms timeout is manageable. However, if you need less than 10ms, batching might not be the best option.
Airbnb uses a hybrid approach: for search models, it disables batching. For background recommendations, it uses batches of up to 64 with a 10ms timeout.
Logging and Observability: When the Model Fails Silently
Models in production fail differently from traditional APIs. You won't see a 500 or a stack trace; you get erratic predictions, silent drifts, and gradual degradation that goes unnoticed until business metrics plummet.
Structured Logging with Context
Implement structured logging that captures not just the request/response, but also the model context:
import logging
import json
from datetime import datetime
class ModelLogger:
def __init__(self, model_version, deployment):
self.model_version = model_version
self.deployment = deployment
self.logger = logging.getLogger(__name__)
def log_inference(self, request_id, input_features, prediction, confidence, latency):
log_entry = {
'timestamp': datetime.utcnow().isoformat(),
'request_id': request_id,
'model_version': model_version,
'deployment': deployment,
'input_shape': input_features.shape,
'prediction': prediction.tolist(),
'confidence': float(confidence),
'latency_ms': latency * 1000,
'input_hash': hash(input_features.tobytes()) # To detect duplicates
}
self.logger.info(json.dumps(log_entry))
# Alerts if confidence is low
if confidence < 0.6:
self.logger.warning(f"Low confidence prediction: {confidence} for request {request_id}")
# Usage
model_logger = ModelLogger(model_version="v2.3.1", deployment="production-us-east")
model_logger.log_inference(
request_id="req_abc123",
input_features=input_data,
prediction=pred,
confidence=conf,
latency=0.082
)
These logs are sent to systems like Google Cloud Logging or Elasticsearch, where you can create dashboards to detect anomalies:
- Sudden drop in average confidence
- Increase in P99 latency
- Prediction distribution deviating from historical data
Circuit Breaker for Degraded Models
If the model starts failing systematically (e.g., 20% of requests with errors), implement a circuit breaker that automatically rolls back to the previous version:
class ModelCircuitBreaker:
def __init__(self, error_threshold=0.15, window_size=100):
self.error_threshold = error_threshold
self.window = []
self.window_size = window_size
self.state = "closed" # closed = normal, open = fallback active
def record_request(self, success):
self.window.append(success)
if len(self.window) > self.window_size:
self.window.pop(0)
error_rate = 1 - (sum(self.window) / len(self.window))
if error_rate > self.error_threshold and self.state == "closed":
self.state = "open"
self.trigger_rollback()
def trigger_rollback(self):
# Change the Kubernetes Service to point to the previous Deployment
print("Circuit breaker activated! Rolling back to v2.2.0")
The Complete Architecture: Uber's Blueprint for 15M Predictions/Hour
With all this said, here's the real production architecture:
-
Multi-versioning: Separate Deployments for each model version (v2.2.0, v2.3.0, canary). A Service with weighted routing distributes 95% of traffic to stable, 5% to canary.
-
Resource isolation: Critical models (e.g., real-time pricing) operate in dedicated node pools with taints, preventing other workloads from interfering.
-
Caching layer: Redis caches predictions for frequent inputs, using a hash of the input as a key. TTL of 1-5 minutes, depending on the model's sensitivity to changes.
-
Native A/B testing: The ingress can route specific users to specific model versions, allowing controlled experiments.
-
Automated retraining pipeline: Cloud Composer (Airflow) orchestrates weekly retraining. If the new model's metrics outperform the current one in staging, it is automatically promoted to production.
This kind of architecture separates a model that "works locally" from one that scales in production without wrecking your GKE budget or operational peace of mind.
Is your model already in production on Kubernetes? What was the metric that most surprised you to discover misconfigured? Honestly, many learn about batching or resource limits only after their first incident.