Model Optimization for Production
Explore hardware-aware strategies to optimize machine learning models for production deployment. Understand quantization options, pruning impacts, and model compilation frameworks. Evaluate edge versus cloud inference decisions based on latency, privacy, model size, connectivity, and cost to build efficient, scalable ML serving systems.
We'll cover the following...
You have a recommendation ranker that meets its offline accuracy targets. When it is deployed to GPU inference servers handling 50,000 queries per second, p99 latency exceeds the SLO budget. The model is accurate enough offline, but it is too slow and expensive to serve at production scale. This is where hardware-aware model optimization becomes the next lever. With system-level strategies such as caching, load balancing, and autoscaling already in place from the previous lesson, the next way to reduce per-request latency and cost is to optimize the model for the target hardware.
Interviewers at L5 and above expect candidates to articulate not just what model to serve but how to make it fast and cheap on specific hardware. This lesson covers four optimization axes that directly address that expectation: quantization, pruning, model compilation, and edge vs. cloud placement. Applied correctly, these techniques can cut inference cost by 2–4× without retraining the model.
Quantization for inference
Quantization reduces the numerical precision of model weights and activations. A standard trained model stores parameters in FP32 (32-bit floating point). Quantization converts these to lower-bit representations like FP16 (16-bit half-precision) or INT8 (8-bit integer), which consume less memory and execute faster on hardware with native support for reduced-precision arithmetic.
Post-training quantization vs. quantization-aware training
Two primary approaches exist, and each fits different production constraints.
Post-training quantization (PTQ): This method converts a trained FP32 model to lower precision without any retraining. A representative
calibration dataset A small, representative subset of production data passed through the model to determine optimal per-layer scaling factors that map FP32 value ranges to INT8 ranges. Quantization-aware training (QAT): This approach simulates quantization noise during the training loop itself, so the model learns weight distributions that are robust to reduced precision. ...