Optimization Principles

A working model is rarely an efficient one. Part II established how to construct ML systems that respect physical constraints; Part III addresses how to meet real-world demands on time, memory, and energy. Every optimization involves navigating a frontier: improving one metric (accuracy, latency, energy) while managing the cost to others. Every optimization in Part III enacts algorithm-machine co-design by trading mathematical structure for physical feasibility. The principles here define the physics of efficiency—the laws that determine why some models are fast and affordable while others are slow and prohibitively expensive.

Principle 5: The Pareto Frontier

Invariant: After objective directions are normalized, a feasible point lies on the Pareto frontier exactly when no other point is at least as good on every objective and strictly better on one.

• Quantization trades numerical precision for reduced memory footprint.
• Pruning trades model capacity for smaller representations and can improve speed when the resulting sparsity or removed structures are supported by the target hardware.
• Distillation trades training compute for inference efficiency.

Implication: Systems engineers navigate this frontier to find the operating point appropriate to a specific deployment environment. There is no universal optimum.

Navigating the Pareto frontier requires knowing which resource to optimize. Before selecting a technique, engineers must diagnose whether the workload is limited by computation, memory movement, or dispatch latency. Arithmetic intensity supplies the first test.

Principle 6: Arithmetic Intensity Law

Invariant: Attainable throughput (\(R\)) is no greater than the minimum of peak compute (\(R_{\text{peak}}\)) and DRAM bandwidth (\(\text{BW}\)) times operational intensity at the cache–DRAM boundary (\(I\)) (Williams et al. 2009): \[ R \leq \min(R_{\text{peak}}, I \times \text{BW}) \]

Operational intensity and DRAM bandwidth must both be measured at the cache–DRAM boundary; this ideal bound excludes launch, synchronization, and other fixed overheads.

Implication: Adding peak compute does not raise the ideal roofline bound for a bandwidth-bound kernel. Engineers must identify whether the bottleneck is compute, bandwidth, or another system term before selecting an optimization.

Williams, Samuel, Andrew Waterman, and David Patterson. 2009. “Roofline: An Insightful Visual Performance Model for Multicore Architectures.” Communications of the ACM 52 (4): 65–76. https://doi.org/10.1145/1498765.1498785.

Wulf, Wm. A., and Sally A. McKee. 1995. “Hitting the Memory Wall: Implications of the Obvious.” ACM SIGARCH Computer Architecture News 23 (1): 20–24. https://doi.org/10.1145/216585.216588.

Gholami, Amir, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, and Kurt Keutzer. 2024. “AI and Memory Wall.” IEEE Micro 44 (3): 33–39. https://doi.org/10.1109/mm.2024.3373763.

Many important ML kernels fall on the memory-bound side of the ridge point, especially low-reuse operations such as embedding lookup, normalization, softmax, depthwise convolution, and small-batch inference paths. Dense matrix multiplications and convolutions can instead be compute-bound when batching and hardware utilization are high. This split is a consequence of the memory wall: processor speed has historically outpaced memory bandwidth, and the cumulative gap has widened over three decades (Wulf and McKee 1995; Gholami et al. 2024). Neural networks, with their massive weight tensors and uneven temporal locality, are especially vulnerable. The arithmetic intensity law diagnoses where a workload sits relative to this wall. The cost of moving data explains why the wall is so punishing. Table 1 maps each bottleneck type to representative optimizations that address it and to changes that leave the dominant term untouched.

Table 1: The Bottleneck Diagnostic: Before optimizing, identify which iron law term dominates. Optimizing the wrong term leaves the dominant bottleneck unchanged and usually yields little or no end-to-end improvement.

If the workload is…	Dominant Term	Optimization That Works	Optimization That is Wasted
Memory-Bound	\(D_{\text{vol}}/\text{BW}\)	Quantization, pruning, batching	Faster accelerator (more FLOP/s)
Compute-Bound	\(O/(R_{\text{peak}} \cdot \eta_{\text{hw}})\)	Better kernels, Tensor Cores, faster accelerator	More memory bandwidth
Latency-Bound	\(L_{\text{lat}}\)	Kernel fusion, async dispatch, bounded microbatching under the latency SLO	More FLOP/s or bandwidth alone

Performance is only one cost of memory movement; energy exposes a second constraint.

Principle 7: The Energy-Movement Invariant

Invariant: Total workload energy depends jointly on per-event energy and event counts. In the cited 45 nm reference, moving a 32-bit value from DRAM can cost roughly 100–1,000\(\times\) more energy per event than a 32-bit floating-point operation, depending on operation type (Horowitz 2014). Here, \(e_{\text{DRAM,32-bit}}\) denotes the energy of one 32-bit DRAM access, and \(e_{\text{arithmetic,32-bit}}\) denotes the energy of one 32-bit floating-point arithmetic event in the cited 45 nm technology. \[ e_{\text{DRAM,32-bit}} \gg e_{\text{arithmetic,32-bit}} \quad \text{for the cited reference} \]

Implication: When memory movement dominates workload energy, optimization strategies should prioritize kernel fusion (avoiding intermediate off-chip traffic) and quantization (reducing data size); when computation dominates, reducing operation counts and improving arithmetic kernels remain central.

Horowitz, Mark. 2014. “1.1 Computing’s Energy Problem (and What We Can Do about It).” 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), 10–14. https://doi.org/10.1109/isscc.2014.6757323.

Even with perfect data locality and optimal bottleneck targeting, a final constraint limits how much speedup any optimization can deliver.

Principle 8: Amdahl's Law

Invariant: The maximum speedup of a system is limited by the fraction of the workload that cannot be accelerated (Amdahl 1967). Here, \(f_{\text{parallel}}\) is the fraction of baseline execution time accelerated and \(S_{\text{parallel}}\) is that fraction’s speedup; the fixed-work model assumes the remaining fraction is unchanged and the acceleration introduces no additional overhead. \[ \text{Speedup} = \frac{1}{(1-f_{\text{parallel}}) + \frac{f_{\text{parallel}}}{S_{\text{parallel}}}} \]

Implication: If 95 percent of a model runs 100\(\times\) faster on a GPU, the total system speedup is capped at ~16.8\(\times\). This explains why data loading and preprocessing often become the ultimate bottlenecks in highly optimized systems.

Amdahl, Gene M. 1967. “Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities.” Proceedings of the April 18-20, 1967, Spring Joint Computer Conference on - AFIPS ’67 (Spring), AFIPS ’67 (spring), 483–85. https://doi.org/10.1145/1465482.1465560.

Part III applies these principles systematically through the D·A·M taxonomy—Data, Algorithm, Machine—asking first whether the work is necessary, then whether it can be simplified, and finally how to do it faster (see The D·A·M Taxonomy for the full diagnostic framework). Benchmarking closes the loop by testing whether those theoretical savings survive end-to-end execution without violating quality constraints.