Machine Foundations

A datasheet reports peak throughput; a workload almost never reaches it. Closing that gap requires knowing which physical limit binds execution first: latency, memory bandwidth, arithmetic intensity, or energy. This appendix collects the durable hardware models and order-of-magnitude values that anchor single-node reasoning throughout the book, providing the mathematical baselines where chapter estimates can be audited or extended. The models presume an undergraduate background in computer architecture, particularly the memory hierarchy.

How to Use This Appendix

This appendix is designed as a reference. When diagnosing performance issues, use this appendix to translate a vague symptom (“it’s slow”) into a specific constraint (“memory bound at batch size one”) and then choose the lever that can actually move.

Conventions used here follow the book-wide notation (for example, \(B\) is reserved for batch size and \(\text{BW}\) for bandwidth).

  • Sanity-check feasibility: Start with section 1.1 for order-of-magnitude numbers.
  • Diagnose the dominant ceiling: Use the Roofline Model in section 1.2.1 to decide whether the workload is compute bound or memory bound.
  • Reason about scaling limits: Use Amdahl’s and Gustafson’s Laws in section 1.2.3 to understand why adding accelerators may not reduce time-to-train.
  • Choose the right precision: Use section 1.4.1 to reason about FP32 vs. BF16/FP16 vs. INT8 as a systems trade-off.
  • Follow the chapter treatment: For the full narrative, use Hardware Acceleration, Model Training, and Model Serving.

Although modern hardware encompasses diverse accelerator architectures, interconnect topologies, and packaging options, the physical laws and bottleneck models detailed here govern them all.

Numbers to Know

Just as Jeff Dean’s “Latency Numbers Every Programmer Should Know”1 shaped a generation of systems engineers, these reference numbers provide the order-of-magnitude intuition essential for ML systems design. Although absolute values and ratios vary by technology and workload, the hierarchy is more durable. Memorize the relationships; use the specific numbers as sanity checks.

1 Jeff Dean: A Google Senior Fellow and one of the architects of Google’s distributed systems infrastructure, including MapReduce, BigTable, and TensorFlow. His latency numbers, originally presented with Peter Norvig around 2010, became a canonical reference for systems engineers. The numbers have been updated over the years as hardware evolved, but the hierarchy of latencies remains remarkably stable; Colin Scott’s interactive visualization shows the latency hierarchy across hardware generations (Scott 2012).

Scott, Colin. 2012. “Numbers Every Programmer Should Know by Year.”
Systems Perspective 1.1: Three numbers that matter most
  • Energy ratio: In the cited 45 nm reference, a 32-bit DRAM read uses ~581× the energy of one FP16 multiply. This motivates arithmetic intensity.
  • Training-state footprint: Model weights (2 bytes FP16) + gradients (2 bytes FP16) + master weights (4 bytes FP32) + optimizer states for Adaptive Moment Estimation (Adam) at 8 bytes. That totals 16 bytes per parameter, so a 7B model needs 112 GB just to start training.
  • Fiber propagation limit: Light travels about 200 km/ms in fiber. The table’s ~40 ms cross-country round trip includes reducible overhead, but propagation cannot be optimized away.

Reference relationships

These relationships mix physical or arithmetic bounds with technology-specific measurements; only the bounds are invariant.

Speed of light tax

Table 1 shows approximate round-trip baselines. Their propagation component is irreducible.

Table 1: Speed of Light Reference: Light in fiber travels ~200 km/ms. These approximate round trips include reducible overhead; only propagation is irreducible.
Distance Round-Trip Latency Implication
Same data center ~1 ms Distributed training feasible
Cross-country (US) ~40 ms Reduces the remaining request budget
Cross-Atlantic ~60 ms Further reduces the request budget
Cross-Pacific ~100 ms Data locality is critical

Energy hierarchy

Table 2 quantifies the energy cost of data movement vs. computation—the fundamental reason why arithmetic intensity dominates ML performance optimization.2

2 Energy hierarchy sources: The DRAM and arithmetic values come from Horowitz (2014) (45 nm). The L1 SRAM/register ratio is an illustrative hierarchy anchor, not a Horowitz measurement. The headline ratio compares a 32-bit DRAM access with a 16-bit floating-point multiply. Both the absolute values and the ratio vary with process, circuit, memory technology, and operand width; wire distance helps explain why movement remains expensive.

Horowitz, Mark. 2014. “1.1 Computing’s Energy Problem (and What We Can Do about It).” 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), 10–14. https://doi.org/10.1109/isscc.2014.6757323.
Table 2: Energy Reference Ratios: The first three ratios use Horowitz’s 45 nm measurements; the L1/register ratio is an illustrative on-chip hierarchy anchor.
Relationship Reference Ratio Source and Interpretation
DRAM access vs. FP16 multiply ~581× Horowitz 45 nm; wire capacitance scales with distance
FP32 vs. INT8 energy ~18× Horowitz 45 nm; bit width determines switching energy
FP32 vs. FP16 energy ~3.4× Horowitz 45 nm; narrower arithmetic reduces switching and datapath energy
L1 SRAM vs. register ~5× Illustrative on-chip hierarchy anchor; distance to the arithmetic unit

Memory hierarchy

Table 3 shows why the memory hierarchy is uneven: nearby on-package hops can differ by only a few times, while off-chip, storage, and network tiers introduce orders-of-magnitude jumps.

Table 3: The Latency Hierarchy: Selected latency relationships in the memory hierarchy. The largest gaps come from crossing physical boundaries: off-chip memory, peripheral links, storage, and network fabric.
Relationship Ratio Why It Persists
Accelerator high-bandwidth memory vs. register ~1000× slower On-chip vs. off-chip
SSD vs. register ~300,000× slower Electrical vs. mechanical/flash
Network vs. local memory ~16× slower Speed of light + switching
Accelerator memory BW vs. CPU\(\leftrightarrow\)Accelerator link ~52× faster Architectural investment priority

Scaling laws

Table 4 collects the arithmetic relationships that govern memory and compute requirements for training and inference.3

3 Training memory (Adam): The 16 bytes/parameter rule assumes mixed-precision training with Adam. ZeRO optimization can reduce per-accelerator memory by sharding optimizer states across accelerators, but the total memory across all accelerators remains ~16\(\times\) parameters.

Table 4: Scaling Rules: These are arithmetic, not hardware-specific. Training memory includes FP16 weights (2 bytes), FP16 gradients (2 bytes), FP32 master weights (4 bytes), and Adam optimizer states (8 bytes for momentum + variance).
Rule Formula Example
Inference memory (FP16) 2 bytes\(\times\) parameters 7B params → 14 GB
Inference memory (INT8) 1 byte\(\times\) parameters 7B params → 7 GB
Training memory (Adam) 16 bytes\(\times\) parameters 7B params → 112 GB
Inference FLOPs (transformer) ~2\(\times\) parameters per token 7B model → ~14 GFLOPs/token
Training FLOPs ~6\(\times\) parameters\(\times\) tokens 7B on 1T tokens → \(4 \times 10^{22}\) FLOPs

Illustrative latency budgets

Latency budgets depend on application-specific safety requirements, user expectations, and service-level objectives. Table 5 lists illustrative targets rather than universal limits.

Table 5: Illustrative Latency Targets: Production requirements depend on the application and its service-level objectives.
Application Budget Constraint
Autonomous braking <10 ms At 100 km/h, 10 ms = 28 cm of travel
Voice assistant <100 ms Human perception of “instant”
Web search <200 ms User patience threshold
Video streaming <1 s Buffer tolerance
Batch training hours–days Throughput dominates latency

Current hardware reference (c. 2024)

These numbers reflect the current generation. Use them for back-of-envelope calculations and recheck them against current hardware.

Memory latency and bandwidth

Table 6 provides representative latency and bandwidth anchors across the memory hierarchy.

Table 6: Memory Hierarchy (c. 2024): Representative latency and bandwidth anchors across registers, caches, memory, interconnects, networks, and storage. Together they show why locality and data movement shape performance.
Level Latency Bandwidth
Register ~0.3 ns —
L1 Cache ~1 ns —
L2 Cache ~4 ns —
GPU HBM3 ~300 ns 3.4 TB/s
PCIe Gen5 (CPU\(\leftrightarrow\)GPU) ~1000 ns 64 GB/s
CPU DRAM ~100 ns 50 GB/s
InfiniBand (network) ~5000 ns 50 GB/s
NVMe SSD ~100000 ns 7 GB/s

Compute throughput

Table 7 compares representative peak throughput and power for data-center GPUs and a mobile NPU.

Table 7: Compute Reference (c. 2024): Representative peak throughput and power for data-center GPUs and a mobile NPU. Because the rows use different precision modes and operation conventions, compare them only within a matched workload and precision context.
Platform FP16/BF16 INT8/FP8-class Power
Data center GPU (H100) 989 TFLOP/s 1979 TFLOP/s (FP8 peak) 700 W
Data center GPU (A100) 312 TFLOP/s 624 TOPS 400 W
Mobile NPU — 35 TOPS 3–5 W

Roofline ridge points

Table 8 defines the arithmetic intensity thresholds that determine whether a workload is memory bound or compute bound.

Table 8: Arithmetic Intensity Thresholds (c. 2024): Batch-1 weight-streaming inference is often memory bound; larger batches may cross the ridge.
Accelerator Ridge Point Implication
A100 (FP16) 153 FLOP/byte Below → memory-bound
H100 (FP16) 295 FLOP/byte Higher bar for compute-bound

Systems Perspective 1.2: A note on terminology: GPUs and accelerators
Throughout this book, “accelerator” is the general term for hardware acceleration. The principles—roofline analysis, memory hierarchies, numerical precision, and performance modeling—apply to GPUs, Tensor Processing Units (TPUs), NPUs, application-specific integrated circuits, and other specialized AI accelerators. Vendor-specific features such as CUDA and NVLink are identified explicitly.

Knowing the numbers is only the first step. The real power comes from compact models that reveal which number matters for the bottleneck at hand. The Roofline Model begins that diagnosis by translating raw hardware specs into actionable performance ceilings.

Physics of Computing

Raw hardware specs—TFLOP/s, TB/s, watt budgets—are necessary but insufficient for performance reasoning. Without compact analytical models, an engineer cannot distinguish a compute-bound workload from a memory-bound one, or predict whether doubling GPUs will halve training time. The models in this section turn those distinctions into quantitative checks.

The Roofline model

The Roofline Model (Williams et al. 2009) bounds how fast a workload can run on a given hardware target. The answer depends on whether the workload runs out of compute or memory bandwidth first.

Williams, Samuel, Andrew Waterman, and David Patterson. 2009. “Roofline: An Insightful Visual Performance Model for Multicore Architectures.” Communications of the ACM 52 (4): 65–76. https://doi.org/10.1145/1498765.1498785.

Every operation has an arithmetic intensity: the ratio of computations performed to bytes moved from memory. Matrix multiplication has high arithmetic intensity because each loaded element is reused many times. Element-wise operations like rectified linear unit (ReLU) have low intensity because each operation loads a number, performs one computation, and writes it back. As figure 1 illustrates, each workload is bounded by either memory bandwidth or compute throughput, and its arithmetic intensity determines which ceiling it hits first.

\begin{tikzpicture}[font=\small\sffamily, scale=1.1]
  \tikzset{
    Axis/.style={line width=1.0pt, draw=GrayLine, ->, >=Latex},
    Guide/.style={dashed, draw=GrayLine!60, line width=0.6pt},
    Label/.style={text=TextBlack, align=center, font=\footnotesize\sffamily},
    Dot/.style={circle, fill=#1, draw=white, line width=0.5pt, minimum size=5pt, inner sep=0pt}
  }
  \draw[step=0.5, gray!15, very thin] (0,0) grid (6,4);
  \draw[Axis] (0,0) -- (6,0) node[right,align=left,text=TextBlack,font=\fontsize{8pt}{10}\sffamily] {Arithmetic Intensity\\ (FLOP/byte)};
  \draw[Axis] (0,0) -- (0,4.2) node[above, text=black,font=\fontsize{8pt}{9}\sffamily] {Performance (FLOP/s)};
  \draw[BlueLine, line width=2pt] (0,0) -- (3,3);
  \draw[OrangeLine, line width=2pt] (3,3) -- (5.8,3);
  \node[Label, text=BlueLine, rotate=45, anchor=south, yshift=2pt] at (1.5, 1.5) {\textbf{Memory Bound}};
  \node[Label, text=OrangeLine, anchor=south, yshift=2pt] at (4.4, 3) {\textbf{Compute Bound}};
  \draw[Guide] (3,0) -- (3,3);
  \node[Dot=TextBlack] at (3,3) {};
  \node[below, font=\scriptsize\sffamily, text=TextBlack] at (3,0) {Ridge Point};
\end{tikzpicture}
Figure 1: The Roofline Model: The sloped memory-bandwidth ceiling and horizontal peak-compute ceiling meet at the ridge point. Plotting a workload by arithmetic intensity identifies which roofline ceiling applies.

The ridge point determines the hardware’s balance. If a workload’s intensity falls below this point, it is memory-bound (sloped region). If above, it is compute-bound (flat region). \[\begin{gather*} \text{Arithmetic Intensity} = \frac{\text{FLOPs}}{\text{bytes accessed}} \qquad\qquad \text{Ridge Point} = \frac{\text{Peak FLOP/s}}{\text{Memory Bandwidth}} \end{gather*}\]

Systems Perspective 1.3: Batch size controls arithmetic intensity
For matrix multiplications, arithmetic intensity scales with the batch dimension. For \(\mathbf{Y} = \mathbf{X}\mathbf{W}\), where \(\mathbf{X}\) is \((B\times d_{\text{in}})\) and \(\mathbf{W}\) is \((d_{\text{in}}\times d_{\text{out}})\):

  • FLOPs: \(2 \times B \times d_{\text{in}} \times d_{\text{out}}\) (multiply-adds)
  • Bytes: Moving \(X\), \(W\), and \(Y\) costs approximately \(s_{\text{elem}}(B d_{\text{in}} + d_{\text{in}}d_{\text{out}} + B d_{\text{out}})\) bytes.

Doubling \(B\) doubles FLOPs while weight traffic stays nearly constant when weights dominate, increasing arithmetic intensity. This is why batching often helps inference serving, although activation traffic, cache behavior, shape, precision, and the ridge point determine whether it reaches the compute ceiling.

A concrete example: The A100 analysis

Consider an NVIDIA A100 GPU with FP16 Tensor Core performance of 312 TFLOP/s and HBM2e bandwidth of 2.04 TB/s. The ridge point is 312 TFLOP/s/2.04 TB/s = 153 FLOP/byte (the Tera prefixes cancel, yielding FLOP/byte).

Two common operations fall on opposite sides of that ridge. General matrix multiply (GEMM) on two square matrices of size 4096 by 4096 has arithmetic intensity of approximately 1365 FLOP/byte, so 1365 FLOP/byte > 153 FLOP/byte and the operation is compute bound. An element-wise ReLU on an FP16 tensor performs 1 comparison while reading 2 bytes and writing 2 bytes (4 bytes total), yielding an arithmetic intensity of only \(1/4 =\) 0.25 FLOP/byte. Because 0.25 FLOP/byte \(\ll\) 153 FLOP/byte, the operation is severely memory bound, achieving only about 0.16 percent of peak TFLOP/s. This contrast explains why modern frameworks fuse operations. Combining ReLU directly with the preceding MatMul kernel avoids writing intermediate results to memory, effectively preserving memory bandwidth.

Dimensional analysis

The Roofline Model helps diagnose where a bottleneck lies. Before applying any performance equation, however, it must be verified as physically meaningful. Dimensional analysis provides this sanity check: any valid equation must be dimensionally homogeneous—every term must resolve to the same units. If they do not, the equation contains an error.

For serialized phases, consider the iron law of ML systems (principle 3) introduced in Iron Law of ML Systems: \[ T = \frac{D_{\text{vol}}}{\text{BW}} + \frac{O}{R_{\text{peak}} \cdot \eta_{\text{hw}}} + L_{\text{lat}} \]

Correctness is verified by confirming that every term resolves to time (seconds): \[ T [s] = \underbrace{ \frac{D_{\text{vol}} [\text{bytes}]}{\text{BW} [\text{bytes/s}]} }_{\text{seconds}} + \underbrace{ \frac{O [\text{FLOPs}]}{R_{\text{peak}} [\text{FLOP/s}] \cdot \eta_{\text{hw}} [1]} }_{\text{seconds}} + \underbrace{ L_{\text{lat}} [s] }_{\text{seconds}} \]

  • Data term: \(\frac{\text{bytes}}{\text{bytes/s}} = \text{bytes} \times \frac{\text{s}}{\text{bytes}} = \mathbf{s}\)
  • Compute term: \(\frac{\text{FLOPs}}{\text{FLOP/s}} = \text{FLOPs} \times \frac{\text{s}}{\text{FLOP}} = \mathbf{s}\)
  • Latency term: Already in seconds.

The equation is physically consistent. Apply this technique to any systems equation: if the dimensions do not match, the formula is wrong. “FLOPs” and “Bandwidth” cannot be traded directly because they have different units. Any such trade-off must convert through Time, which is precisely what the iron law quantifies.

Once an equation is dimensionally sound, the next question is how its time terms scale across devices. Section 1.2.3 answers that question with two complementary bounds.

Amdahl’s Law and Gustafson’s Law

Parallelization is the primary tool for scaling ML, but its limits depend on how the workload scales. These two laws frame the fundamental tension in parallel computing. Amdahl’s law is the pessimist’s view, governing how much faster a fixed task can run (optimizing latency). Gustafson’s law is the optimist’s view, governing how much more work can be done in the same time (optimizing throughput).

Strong scaling (Amdahl’s Law)

Strong scaling measures how much faster a fixed-size problem runs as processors are added.

Amdahl’s Law (Amdahl 1967) states that the speedup is limited by the serial portion of the task.4 If a fraction \(s\) of the task is serial (cannot be parallelized) and \(p = 1-s\) is parallelizable, the maximum speedup with \(n\) processors is: \[ \text{Speedup}(n) = \frac{1}{s + \frac{1-s}{n}} \]

Amdahl, Gene M. 1967. “Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities.” Proceedings of the April 18-20, 1967, Spring Joint Computer Conference on - AFIPS ’67 (Spring), AFIPS ’67 (spring), 483–85. https://doi.org/10.1145/1465482.1465560.

4 Gene Amdahl (1922–2015): A legendary computer architect at IBM, where he was the chief architect of the System/360. He later founded Amdahl Corporation to compete with IBM in the mainframe market.

As \(n \to \infty\), the term \(\frac{1-s}{n} \to 0\), and the speedup converges to \(1/s\).

To see Amdahl’s Law in action, suppose 5 percent of a training step is serial overhead (for example, Python global interpreter lock (GIL), kernel launch latency) and 95 percent is parallelizable matrix math:

  • With \(n=1\), speedup is 1.
  • With \(n=\) 8, speedup is 1/(0.05 + 0.95/8) ≈ 5.9×.
  • With \(n \to \infty\), speedup is capped at 1/0.05 = 20×.

No matter how many accelerators are added, this fixed workload cannot run faster than 20×.

Weak scaling (Gustafson’s Law)

Weak scaling measures how much larger a problem can become while holding runtime fixed as processors are added.5

5 John Gustafson: A computer scientist known for his work in parallel computing and for introducing the Unum (universal number) format. His law was a direct response to the perceived “limits” of Amdahl’s Law when applied to massive scale.

This is the reality of Large Language Models. Rather than using 1,000 accelerators to train a model on a small dataset in milliseconds, they are used to train on a dataset 1,000\(\times\) larger in reasonable time.

For \(n\) processors, let \(s\) be the serial fraction of execution time measured on those processors for the scaled workload. Gustafson’s Law (Gustafson 1988) models this “scaled speedup”: \[ \text{Scaled Speedup}(n) = n - s(n - 1) \]

Gustafson, John L. 1988. “Reevaluating Amdahl’s Law.” Communications of the ACM 31 (5): 532–33. https://doi.org/10.1145/42411.42415.

Because the parallel part grows linearly with \(n\) while \(s\) remains fixed, weak scaling can keep efficiency high as the problem grows.

Using the same 5 percent serial overhead (\(s\) = 0.05), Gustafson’s Law tells a very different story:

  • With \(n=1\), speedup is 1.
  • With \(n=\) 8, Scaled Speedup is 8 − 0.05 \(\times\) (7) = 8 − 0.35 = 7.65×.
  • With \(n=\) 1000, Scaled Speedup is 1000 − 0.05 \(\times\) (999) ≈ 950×.

In weak scaling, efficiency remains high because the useful work (training the model) scales up to dwarf the fixed overheads.

The same scaling lens turns model size, data size, hardware count, and utilization into a single training-time estimate.

Napkin Math 1.1: The training time equation
Problem: How long does dense-transformer training take for a stated model size, token count, accelerator count, peak rate, and utilization?

Math: \[ T \approx \frac{6 \cdot P \cdot D}{N_{\text{GPU}} \cdot R_{\text{peak}} \cdot \eta_{\text{hw}}} \]

Variables:

  • Training FLOP factor: The factor 6 derives from \(2P\text{ FLOP/token}\) for the forward pass and \(4P\text{ FLOP/token}\) for the backward pass (\(2P\) for activation gradients and \(2P\) for weight gradients), yielding \(6PD\text{ FLOP}\) total across \(D\) training tokens.
  • \(P\): Number of model parameters.
  • \(D\): Number of training tokens.
  • \(N_{\text{GPU}}\): Number of GPUs.
  • \(R_{\text{peak}}\): Peak FLOP/s of one accelerator.
  • \(\eta_{\text{hw}}\): Hardware utilization, typically 30 percent–50 percent in this training estimate.

Example: Training a 1B parameter model on 20B tokens using 1 A100 (312 TFLOP/s) at 40 percent utilization. \(\text{Total FLOPs} = 6 \times 1 \times 10^{9} \times 2 \times 10^{10} = 1.2 \times 10^{20} \text{ FLOPs}\). \(\text{Throughput} = 1 \times (3.12 \times 10^{14}) \times 0.40 \approx 1.248 \times 10^{14} \text{ FLOP/s}\). \(T = \frac{1.2 \times 10^{20}}{1.248 \times 10^{14}} \approx 961,538.5 \text{ seconds} \approx 16,025.6 \text{ minutes}\).

Systems insight: Under these assumptions, training takes 961,538.5 seconds (≈ 16025.6 minutes, or about 11.1 days); the estimate makes model scale, data scale, effective accelerator throughput, and utilization explicit levers.

Little’s Law

For capacity planning in stable inference systems, Little’s law (Little 1961) relates long-run mean concurrency (\(Q_{\text{req}}\)), mean arrival rate (\(\lambda_{\text{arr}}\)), and mean time in system (\(T_{\text{lat}}\)):6 \[ Q_{\text{req}} = \lambda_{\text{arr}} \times T_{\text{lat}} \]

Little, John D. C. 1961. “A Proof for the Queuing Formula: \(L = \lambda W\).” Operations Research 9 (3): 383–87. https://doi.org/10.1287/opre.9.3.383.

6 John Little: An Institute Professor at MIT and a pioneer in the field of operations research. His law, proved in 1961, is fundamental to queuing theory and is used across fields from manufacturing to computer network analysis.

To see this in practice, consider sustaining 1,000 QPS with 50 ms average latency. The law dictates that the system must support 1000 \(\times\) 0.05 s = 50 concurrent requests.

This sizes worker pools, but memory needs another assumption. If every in-system request simultaneously holds 1 GB of device state, the average population implies 50 GB; queued requests may instead stay in host memory or share batched state. If 24 device slots each remain occupied for 50 ms, their throughput proxy is \(N_{\text{max}}/T_{\text{lat}} = 24/0.05 = 480 \text{ QPS}\); Little’s Law alone sets no physical maximum.

These physics-based models—Roofline, Amdahl, Gustafson, and Little—diagnose where bottlenecks lie. Translating those diagnoses into actionable optimizations, however, requires understanding the concrete hardware structures that impose them: caches, memory buses, and interconnects.

Computer Architecture Essentials

A GPU advertises 1,000 TFLOP/s, yet a kernel achieves only 30 TFLOP/s. The gap may reflect data movement, insufficient parallelism, synchronization, launch overhead, or software inefficiency. While physics sets theoretical performance bounds, computer architecture defines the machinery that determines how close a real workload can get. The following discussion covers the latency, bandwidth, and energy trade-offs that shape system design.

Latencies every programmer should know

The first step in systems intuition is understanding the cost of distance. Table 9 quantifies how long the processor waits for data from different levels of the memory hierarchy. If accessing a register is like picking up a nearby pencil, fetching from high-bandwidth memory (HBM) is walking across the office, and fetching from disk is flying to the moon.

Table 9: The Latency Hierarchy: Representative access times for modern AI hardware; the approximate cycle counts use a 3 GHz reference clock. The jump from SRAM cache to HBM makes cache locality a major performance factor.
Component Latency (ns) Cycles (Approx) Relative “Distance”
Register ~0.3 ns 1 cycle 10 seconds
L1 Cache ~1 ns 3–4 cycles 33.3 seconds
L2 Cache ~4 ns 12 cycles 2.2 minutes
HBM3 (GPU Memory) ~300 ns 1,000 cycles 2.8 hours
NVLink (GPU-GPU) ~500 ns 1,500 cycles 4.6 hours
PCIe (CPU-GPU) ~1000 ns 3,000 cycles 9.3 hours
InfiniBand (Network) ~5000 ns 15,000 cycles 1.9 days
SSD (NVMe) ~100000 ns 300,000 cycles 38.6 days

The AI hardware cheat sheet (modern reference)

Latency measures the wait for the first byte; bandwidth measures how many bytes follow. Table 10 provides the constants for back-of-the-envelope “Roofline” calculations. These represent the “standard units of compute” for the current era of machine learning.

Table 10: Reference Specs: Key constants for quantitative analysis. Always check specific datasheets; HBM capacity follows vendor units, and interconnect bandwidth is aggregate bidirectional, so topology and direction matter.
Spec NVIDIA H100 (SXM) Google TPU v5p System Impact
BF16 Peak 989 TFLOP/s 459 TFLOP/s The “Speed Limit” (\(R_{\text{peak}}\))
Memory Bandwidth 3.35 TB/s 2.76 TB/s The “Width of the Pipe” (\(\text{BW}\))
HBM Capacity 80 GB 95 GiB Max Model Size (\(P\))/Batch Size (\(B\))
On-chip SRAM 50 MB L2 128 MiB VMEM/chip Critical for Operator Fusion
Interconnect 900 GB/s (NVLink) 1200 GB/s (Inter-Chip Interconnect, ICI) Determines Model Parallelism Scaling

The memory hierarchy

Computer systems use a hierarchy because no single technology provides both high capacity and low latency. Figure 2 shows this trade-off: keeping data higher in the pyramid (registers/cache) reduces access latency when data movement is on the critical path.

\scalebox{0.8}{%
\begin{tikzpicture}[line cap=round, line join=round, font=\sffamily\small]
%—parameters ---
\def\H{6.2}   % triangle height
\def\W{4.6}   % half triangle width

%—levele (down to up) ---
\def\yone{1.5}
\def\ytwo{3}
\def\ythree{4.5}
%—macro: calculate the half-width at the height y and write it in the macro \tmp
\newcommand{\halfwidthat}[2]{%
  \pgfmathsetmacro#2{\W*(1-#1/\H)}%
}
% ---calculate the required widths ---
\halfwidthat{0}{\wzero}
\halfwidthat{\yone}{\wone}
\halfwidthat{\ytwo}{\wtwo}
\halfwidthat{\ythree}{\wthree}
%—centri po visini (x=0 zbog simetrije) ---
\pgfmathsetmacro{\ycA}{0.5*(0+\yone)}
\pgfmathsetmacro{\ycB}{0.5*(\yone+\ytwo)}
\pgfmathsetmacro{\ycC}{0.5*(\ytwo+\ythree)}
\pgfmathsetmacro{\ycD}{0.46*(\ythree+\H)}
%up to down
 \filldraw[fill=RedFill, draw=RedLine, line width=1pt]
(-\wthree,\ythree) -- (\wthree,\ythree) -- (0,\H) -- cycle;

\filldraw[fill=YellowFill, draw=YellowLine, line width=1pt]
(-\wtwo,\ytwo) -- (\wtwo,\ytwo) -- (\wthree,\ythree) -- (-\wthree,\ythree) -- cycle;

\filldraw[fill=BlueFill, draw=BlueLine, line width=1pt]
(-\wone,\yone) -- (\wone,\yone) -- (\wtwo,\ytwo) -- (-\wtwo,\ytwo) -- cycle;

 \filldraw[fill=GreenFill, draw=GreenD, line width=1pt]
(-\wzero,0) -- (\wzero,0) -- (\wone,\yone) -- (-\wone,\yone) -- cycle;

% ---text ---
\node[font=\sffamily\bfseries\small,text=GreenD] at (0,\ycA) {Storage (SSD/Disk)};
\node[font=\sffamily\bfseries\small,text=BlueLine] at (0,\ycB) {HBM/DRAM};
\node[font=\sffamily\bfseries\small,text=YellowLine] at (0,\ycC) {L1/L2/L3 Cache};
\node[font=\sffamily\bfseries\small,text=RedLine] at (0,\ycD) {Registers};
%
\coordinate(D)at($(\W,0)+(0.65,0)$);
\coordinate(L)at($(-\W,0)+(-0.65,0)$);
\coordinate(V)at($(0,\H)+(0,0)$);
\path[green](D)|-coordinate(D1)(V);
\path[green](L)|-coordinate(L1)(V);
%
\draw[->,>=Latex,line width=1pt,draw=black!40](D)--
node[align=center,right]{Faster Speed\\ Lower Latency}(D1);
\draw[->,>=Latex,line width=1pt,draw=black!40](L1)--
node[align=center,left]{Larger Capacity\\ Lower Cost}(L);
%—outline ---
%\draw[thick] (-\W,0) -- (0,\H) -- (\W,0) -- cycle;
\end{tikzpicture}}
Figure 2: The Memory Hierarchy: Performance depends on data proximity. Accessing HBM is roughly 1,000\(\times\) slower than registers; accessing SSD is roughly 300,000\(\times\) slower.

The memory hierarchy is the fundamental physical constraint of machine learning systems. Table 11 consolidates the physical properties—latency, bandwidth, and energy—across the entire stack.

Table 11: Physical Properties of the Memory Hierarchy (c. 2024): Representative latency and bandwidth anchors across the memory hierarchy. The hierarchy spans roughly five orders of magnitude in latency. NVLink is aggregate bidirectional; only the cited 45 nm DRAM energy is shown because other tiers are implementation-specific. Keeping data higher in the hierarchy can therefore deliver large performance gains.
Layer Technology Latency Bandwidth Energy Reference (per 32b)
Registers Flip-Flops ~0.3 ns — —
L1 Cache SRAM ~1 ns — —
L2 Cache SRAM ~4 ns — —
Memory (Local) HBM3 ~300 ns 3350 GB/s 640 pJ
Interconnect NVLink 4.0 ~500 ns 900 GB/s —
Host Link PCIe Gen5 ~1000 ns 64 GB/s —
System RAM DDR5 ~100 ns 50 GB/s —
Network (Fabric) InfiniBand NDR ~5000 ns 50 GB/s —
Storage (Local) NVMe SSD ~100000 ns 7 GB/s —

The hierarchy’s energy costs reveal why data movement dominates modern system design.

Systems Perspective 1.4: The high cost of data movement
In the cited 45 nm reference, fetching a 32-bit value from DRAM costs roughly 581× the energy of one FP16 multiply (~640 pJ vs. ~1.1 pJ). The ratio is platform-specific and does not alone determine workload energy, which also depends on access counts and reuse. It nevertheless explains why arithmetic intensity matters.

Bandwidth vs. latency

Bandwidth (throughput) and latency (delay) are distinct constraints. For data volume \(D_{\text{vol}}\) sent over effective bandwidth \(\text{BW}\) with fixed path latency \(L_{\text{lat}}\), and assuming the latency and serialization costs do not overlap, total transfer time is: \[ T = L_{\text{lat}} + \frac{D_{\text{vol}}}{\text{BW}} \]

The crossover point is the transfer size where the two terms are equal: \[ D_{\text{vol,cross}} = L_{\text{lat}} \times \text{BW} \]

For transfers below \(D_{\text{vol,cross}}\) (for example, small request or control messages), latency dominates. For transfers above it (for example, loading weights), bandwidth dominates.

Consider sending data over a 10 Gb/s link with 10 ms base request/response latency. The crossover size is 12.5 MB, so the dominant term depends on which side of that threshold the transfer falls.

  • Latency-bound packet (1 KB):
    • Transmission: 1 KB \(\times\) 8/10 Gb/s ≈ 0.8 μs.
    • Total Time ≈ 10 ms + 0.8 μs ≈ 10 ms.
    • Result: Fixed path and protocol latency dominates; propagation is one component.
  • Bandwidth-bound checkpoint (1 GB):
    • Transmission: \(1\text{ GB} \times 8/10\text{ Gbps} \approx 800\text{ ms}\).
    • Total Time \(\approx 10\text{ ms} + 800\text{ ms} = 810\text{ ms}\).
    • Result: Base latency is negligible; bandwidth is the bottleneck.

Architecture determines how fast data can move, but numerical precision directly controls how much data must move. Halving precision from FP32 to FP16 halves bytes per parameter and can nearly double useful bandwidth when the hardware, kernels, and model accuracy support it. Understanding these trade-offs requires a closer look at how numbers are represented in hardware.

Numerical Representations

Statistics characterizes data distributions; numerical representations determine how the system stores those values. In ML systems, the choice of precision (FP32 vs. BF16 vs. INT8) is a direct trade-off between statistical fidelity and hardware throughput.

Floating-point format comparison

IEEE 754 formats such as FP32 and FP16, together with AI-specific formats such as BF16 and FP8, define different trade-offs between dynamic range (the span of representable values) and precision (the granularity of values within that range). Table 12 summarizes the key formats and their use cases, while figure 3 visualizes the bit allocations.

Table 12: Numerical Format Comparison: Each format trades off precision, dynamic range, memory footprint, and compute throughput. BF16 shares FP32’s normal exponent range while using half the storage.
Format Bits Exponent Mantissa Dynamic Range Typical Use Case
FP32 32 8 23 \(\sim 10^{-38}\) to \(10^{38}\) Training (full precision), reference inference
FP16 16 5 10 Normal: \(6.10 \times 10^{-5}\)–\(6.55 \times 10^{4}\) Training often with loss scaling, inference
BF16 16 8 7 FP32 normal exponent range Training, usually without loss scaling
FP8 8 4 or 5 3 or 2 Varies Training/inference on supported hardware
INT8 8 N/A N/A -128 to 127 Inference after quantization 
\begin{tikzpicture}[font=\small\sffamily, scale=0.9, every node/.append style={scale=0.9}]
  \tikzset{
    BitBox/.style={draw=white, line width=0.8pt, minimum height=0.6cm, align=center, font=\scriptsize\bfseries\sffamily, text=white},
    Label/.style={text=TextBlack, font=\small\bfseries\sffamily, anchor=east}
  }

  % Colors
  \definecolor{SignColor}{HTML}{D9534F}      % Red
  \definecolor{ExpColor}{HTML}{5BC0DE}       % Blue
  \definecolor{MantColor}{HTML}{F0AD4E}      % Orange
  \definecolor{IntColor}{HTML}{5CB85C}       % Green

  % FP32
  \node[Label] at (-0.2, 3) {FP32 (32-bit)};
  \node[BitBox, fill=SignColor, minimum width=0.3cm] (fp32_s) at (0.15, 3) {S};
  \node[BitBox, fill=ExpColor, minimum width=2.4cm, right=0pt of fp32_s] (fp32_e) {Exponent (8)};
  \node[BitBox, fill=MantColor, minimum width=6.9cm, right=0pt of fp32_e] (fp32_m) {Mantissa (23)};

  % BF16
  \node[Label] at (-0.2, 2) {BF16 (16-bit)};
  \node[BitBox, fill=SignColor, minimum width=0.3cm] (bf16_s) at (0.15, 2) {S};
  \node[BitBox, fill=ExpColor, minimum width=2.4cm, right=0pt of bf16_s] (bf16_e) {Exponent (8)};
  \node[BitBox, fill=MantColor, minimum width=2.1cm, right=0pt of bf16_e] (bf16_m) {Mant (7)};
  \node[right=0.2cm of bf16_m, font=\scriptsize\sffamily, text=gray] {Matches FP32 Range};

  % FP16
  \node[Label] at (-0.2, 1) {FP16 (16-bit)};
  \node[BitBox, fill=SignColor, minimum width=0.3cm] (fp16_s) at (0.15, 1) {S};
  \node[BitBox, fill=ExpColor, minimum width=1.5cm, right=0pt of fp16_s] (fp16_e) {Exp (5)};
  \node[BitBox, fill=MantColor, minimum width=3.0cm, right=0pt of fp16_e] (fp16_m) {Mantissa (10)};

  % INT8
  \node[Label] at (-0.2, 0) {INT8 (8-bit)};
  \node[BitBox, fill=IntColor, minimum width=2.4cm] (int8) at (1.2, 0) {Integer (8)};

  % Grid/Scale markers (approximate)
  \draw[gray!30, dashed] (0, -0.5) -- (0, 3.5);
  \draw[gray!30, dashed] (9.6, -0.5) -- (9.6, 3.5);
  \node[below, font=\scriptsize\sffamily, text=gray] at (0, -0.5) {Bit 31/15/7};
  \node[below, font=\scriptsize\sffamily, text=gray] at (9.6, -0.5) {Bit 0};

\end{tikzpicture}
Figure 3: Numerical Format Bit Layouts: A visual comparison of bit allocations. BF16 preserves FP32’s 8-bit exponent and therefore its normal exponent range. FP16 trades range for precision and often uses loss scaling to reduce underflow.

Beyond bit width, the allocation of bits between exponent and mantissa determines what range of values each format can represent.

Among these formats, BF167 deserves special attention (Wang and Kanwar 2019; Kalamkar et al. 2019). It occupies the same 16-bit storage class as FP16 but behaves differently in training, so nominal width alone does not predict the operational burden. The following systems perspective connects bit allocation to memory traffic, numerical stability, and deployment efficiency.

7 BF16 (brain floating point 16): Originally introduced with Google TPUv2 and later adopted by Intel, Arm, and NVIDIA, BF16 became a common training format across accelerator ecosystems. Its adoption made format support in hardware, kernels, and distributed-training libraries a portability concern rather than a vendor-specific feature.

Wang, Shibo, and Pankaj Kanwar. 2019. BFloat16: The Secret to High Performance on Cloud TPUs.
Systems Perspective 1.5: The dynamic range wall

The choice of numerical format is a direct application of the iron law of ML systems (principle 3). Reducing precision from FP32 to BF16 or FP16 halves the data volume term, potentially doubling throughput on memory-bound workloads. However, the type of 16-bit format determines the engineering complexity:

  • Dynamic range (the exponent): BF16 preserves the eight-bit exponent of FP32 and therefore the same normal exponent range (Kalamkar et al. 2019).
  • Precision (the mantissa): FP16 has a larger 10-bit mantissa than BF16 (7 bits), offering higher precision for values within its range. Its five-bit exponent, however, raises underflow risk. FP16 training therefore often uses loss scaling, an operational step that multiplies gradients by a large constant to keep more values representable (Micikevicius et al. 2017).
  • Energy efficiency: INT8 operations can be substantially more efficient than floating-point equivalents because they move less data and use simpler integer arithmetic paths. Moving to INT8 for inference is a primary lever for deploying neural networks under tight memory, latency, or power budgets (Jacob et al. 2018; Krishnamoorthi 2018).
Krishnamoorthi, Raghuraman. 2018. “Quantizing Deep Convolutional Networks for Efficient Inference: A Whitepaper.” arXiv Preprint arXiv:1806.08342 abs/1806.08342.
Jacob, Benoit, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. 2018. “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2704–13. https://doi.org/10.1109/cvpr.2018.00286.
Micikevicius, Paulius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, et al. 2017. “Mixed Precision Training.” arXiv Preprint arXiv:1710.03740.
Kalamkar, Dhiraj, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, et al. 2019. A Study of BFLOAT16 for Deep Learning Training.

Integer quantization

Quantization maps continuous floating-point values to discrete integers, typically INT8. The key challenge is choosing how to map the floating-point range to integers. Two approaches dominate.

For symmetric quantization, let \(s_{\text{quant}} = \max |x|\) over the calibrated range and assume \(x\) is clipped to \([-s_{\text{quant}}, s_{\text{quant}}]\). The signed INT8 code is: \[ x_{\text{int}} = \text{round}\left(\frac{x}{s_{\text{quant}}} \times 127\right) \] This mapping works well for weight distributions centered around zero.

Asymmetric quantization handles distributions that are not centered (common after ReLU, which produces only nonnegative values) by shifting the range before scaling. Let \(x_{\min}\) and \(x_{\max}\) bound the calibrated range, set the normalization span \(s_{\text{quant}} = x_{\max} - x_{\min}\), and clip \(x\) to \([x_{\min}, x_{\max}]\). A common unsigned 8-bit mapping is: \[ x_{\text{uint8}} = \text{round}\left(\frac{x - x_{\min}}{s_{\text{quant}}} \times 255\right) \]

The choice between symmetric and asymmetric quantization depends on the tensor’s distribution and has measurable accuracy implications.

Summary

Machine-side diagnosis begins with units, ceilings, and working sets. Reference numbers establish orders of magnitude; the Roofline Model and the iron law identify whether operations, bytes, or latency bind; Amdahl’s Law, Gustafson’s Law, and Little’s Law test speedup, scaling, and concurrency claims; and dimensional analysis catches impossible results. The memory hierarchy and interconnect show how those limits appear as latency and bandwidth constraints, while numerical representations show how dynamic range, precision, and quantization change storage, energy, throughput, and accuracy.

Treat peak specifications as ceilings, not predictions. Compare a measurement with the relevant ceiling, then trace the remaining gap to utilization, data movement, runtime overhead, architecture, or software. A back-of-envelope estimate is useful when it states its assumptions and units and makes the next measurement obvious.

Back to top