Search⌘ K
AI Features
Home
Courses
Grokking the Machine Learning System Design Interview

Back-of-the-Envelope Estimation for ML Systems

Explore back-of-the-envelope estimation methods crucial for ML system design interviews. Understand how to break down complex questions into manageable parts, state assumptions clearly, and produce order-of-magnitude estimates for model size, storage, throughput, and training time. This lesson helps you develop the reasoning skills needed to plan infrastructure, balance latency versus throughput, and avoid common pitfalls in production ML system design.

We'll cover the following...

The previous lesson showed how metric cannibalization and guardrail design require quantitative reasoning. The same reasoning applies to infrastructure planning, which many candidates underprepare for. When an interviewer asks, “How much storage do we need for 1 billion user embeddings?”, they are not just testing arithmetic. They are testing whether you can break an ambiguous infrastructure question into smaller subproblems, state reasonable assumptions, and produce an order-of-magnitude estimate that can inform a real design decision. A candidate who can reason through this quickly demonstrates the production awareness expected in senior-level interviews.

Back-of-the-envelope estimation is one of the highest-signal skills in ML system design interviews. Interviewers evaluate your engineering judgment, not your arithmetic precision. The primary failure mode is oversimplified assumptions that ignore real-world complexities like mixed-precision training, replication overhead, and data distribution drift, leading to underprovisioned resources and system bottlenecks. This lesson covers four estimation categories that form the quantitative backbone of any ML system design discussion: model size, storage, throughput, and training time.

Note: Interviewers care about your reasoning chain far more than the final number. Stating assumptions explicitly before computing is what distinguishes senior candidates.

The following mindmap provides a structural overview of these four estimation pillars and their key variables:

Four estimation categories and their key variables for ML system design interviews

With this structural overview in place, let’s work through each estimation type with concrete formulas and worked examples.

Model size and storage estimation

These first two estimation types address the most common interview questions about memory footprints and disk requirements. They share a common structure: count the entities, multiply by the bytes per entity, then account for overhead factors that candidates frequently forget.

Model size estimation

The core formula for model memory is straightforward:

total memory=parameters×bytes per parameter\text{total memory} = \text{parameters} \times \text{bytes per parameter}

The bytes per parameter depend on the numerical precision used.

  • FP32 (full precision): Each parameter occupies 4 bytes, used in traditional training setups.

  • FP16 / BF16 (half precision): Each parameter occupies 2 bytes, now standard for inference and mixed-precision training.

  • INT8 (quantized): Each parameter occupies 1 byte, common in optimized serving deployments.

For inference, the calculation stops here. But training memory is dramatically larger because the optimizer maintains additional state.A widely used optimization algorithm that maintains two momentum terms (first and second moment estimates) per parameter in addition to the weights and gradients themselves. With Adam optimizerA widely used optimization algorithm that maintains two momentum terms (first and second moment estimates) per parameter in addition to the weights and gradients themselves. in FP32, each parameter requires roughly 16 bytes total: 4 bytes for the weight, 4 bytes for the gradient, and 4 bytes for the two momentum terms each.

Consider a 7B parameter model. For inference in FP16, the footprint is 7×109×2=147 \times 10^9 \times 2 = 14 ...