Low Latency Serving at Scale
Explore how to design machine learning serving systems that meet strict low latency requirements at scale. Understand how model caching, pre-computation, tail latency SLAs, stateless server design, load balancing, and autoscaling work together to ensure reliable and performant production ML deployments under heavy traffic.
A recommendation system that ranks candidates in 20 ms in a local benchmark can exceed 500 ms under high concurrent traffic. The gap between “works in development” and “works in production” often comes down to latency, reliability, and resource management. Consider a large-scale video ranking pipeline or ride-hailing ETA service, where even small p99 latency regressions can affect a large number of user requests. This lesson covers how to keep each stage of the retrieval and ranking funnel within its latency budget under production load.
The strategies here rest on four pillars: model caching, pre-computation, tail latency SLAs, and horizontal scaling. Industry systems like Snowflake’s two-layer serving architecture, which separates a controller layer from inference engines, exemplify the microservices approach to this problem. This lesson focuses on system-level serving strategies. The next lesson on Model Optimization for Production covers hardware-level techniques like quantization and compilation that complement what you learn here.
Model caching and pre-computation
The fastest inference call is the one you never make. Two complementary strategies make this possible, and understanding when to apply each one is essential for any ML serving design.
Result and feature caching
Result caching stores inference outputs keyed by input features in low latency stores like Redis or Memcached. When a user-item pair score has already been computed, the system returns the cached value in sub-millisecond time instead of running the model again. The effectiveness of this approach depends on the
Cache entries need a TTL (time-to-live) policy that balances freshness against hit rate. Short TTLs keep results fresh but reduce hit rates. Long TTLs improve hit rates but serve stale scores. When user context changes rapidly, such as a user switching locations in a ride-hailing app, cache invalidation becomes complex because the cached score no longer reflects the current state.
Feature caching operates one layer deeper. Instead of caching final scores, it caches expensive intermediate feature computations. Uber’s ETA system, for example, caches precomputed geographic features to avoid redundant feature store lookups on every request. This reduces per-request latency even when the final inference must run online.
Practical tip: In an interview, specify...