Acceleration Fundamentals
Hardware Acceleration
Purpose
Why does moving data cost more than computing it?
Arithmetic is inexpensive relative to many memory accesses. In the time it takes to fetch a value from main memory, a processor can perform many calculations. This imbalance, the “memory wall,” reflects the latency, bandwidth, and energy costs of moving data across a system. It explains why specialized accelerators are not merely faster at math but are architected to hide, amortize, and reduce the cost of moving data through deep memory hierarchies, massive parallelism, and specialized data paths. Hardware acceleration allows many large AI workloads to grow when general-purpose processor scaling alone is insufficient. It also explains why some optimizations that reduce theoretical computation fail to improve runtime. If an operation is memory bound, computing less may not help because computation is not the bottleneck. Hardware selection therefore cannot be reduced to comparing peak FLOP/s. What matters is whether a workload’s data movement patterns align with what the hardware was designed to accelerate. A model with large embedding tables and irregular lookups needs a different accelerator than one performing dense matrix multiplications over compact weight tensors. Software must expose and schedule the available reuse; otherwise specialized units wait while data moves and headline throughput remains theoretical. Matching workload and hardware determines whether execution remains at a fraction of theoretical peak or approaches the hardware’s practical ceiling. In D·A·M terms, the accelerator makes the machine constraint concrete and shapes which algorithms remain practical in production.
Learning Objectives
- Explain hardware acceleration as machine-axis specialization for tensor workloads, data reuse, and performance per watt
- Calculate arithmetic intensity and roofline ceilings to classify kernels as compute bound or memory bound
- Diagnose memory-wall bottlenecks using bandwidth, cache hierarchy, host-device transfer, and energy-movement costs
- Compare Tensor Cores, systolic arrays, SIMD/SIMT units, and sparse execution for ML compute primitives
- Select dataflow, tiling, and mapping strategies that maximize reuse under memory-capacity constraints
- Analyze compiler and runtime optimizations that fuse kernels, plan memory, and schedule accelerators
- Evaluate accelerator choices across throughput, latency, power, cost, and deployment-context constraints
Hardware acceleration turns on the machine axis.
Reducing parameters, precision, or operations only matters when the machine can execute the resulting representation efficiently. Data selection reduced the data term, and compression reduced the algorithm’s work; hardware acceleration asks what the machine can actually deliver. The answer starts with the memory wall: arithmetic is often cheap relative to data movement. An off-chip memory access can take many processor cycles and consume orders of magnitude more energy than a low-precision arithmetic operation, with the exact ratio depending on the device and memory level. Specialized hardware matters because it raises compute throughput while organizing memory, dataflow, and parallelism so those arithmetic units stay fed.
Definition 1.1: Hardware acceleration
Hardware acceleration is the practice of mapping work from general-purpose execution onto specialized processors or functional units optimized for a narrower operation class. For ML workloads, that specialization commonly trades some programmability for the compute density \((R_{\text{peak}})\) and performance-per-watt gains that regular tensor operations can exploit.
- Significance: An A100 GPU delivers 312 TFLOP/s for FP16/BF16 matrix multiplication; against this chapter’s illustrative 1–2 TFLOP/s reference-CPU baseline, this is a 156–312× gap. Its 54.2 billion transistors devote far more silicon to parallel arithmetic than to branch prediction, out-of-order scheduling, and large caches (NVIDIA Corporation 2020a; Choquette et al. 2021).
- Distinction: A general-purpose CPU is designed for flexible execution, low-latency control, and varied workloads. An accelerator instead concentrates resources on a narrower operation class, so it achieves its gains only when the workload exposes the supported operations and enough parallel work to keep its arithmetic units busy.
- Common pitfall: A frequent misconception is that an accelerator’s advertised peak throughput is the throughput a workload receives. Delivered performance is the lower of the compute ceiling and what memory bandwidth can feed, the roofline constraint: a low-arithmetic-intensity kernel can sit at a small fraction of peak FLOP/s no matter how fast the silicon is rated, because it starves for data rather than for arithmetic.
This definition frames the chapter’s central engineering trade-off. General-purpose processors support broad control flow, memory, and execution patterns. An accelerator can devote more of its design budget to the arithmetic, storage, and data paths of a narrower workload class. The result can be order-of-magnitude improvements in throughput per watt when the workload and software path match that specialization.
Hardware alone, however, cannot achieve these gains. The algorithms must be designed to exploit what the hardware offers, and the hardware must be built to accelerate the operations algorithms actually use. This symbiosis motivates a complementary principle: hardware-software co-design.
Definition 1.2: Hardware-software co-design
Hardware-Software Co-design is an ML accelerator development method that crosses conventional abstraction boundaries, allowing algorithm constraints to inform silicon design and hardware capabilities to shape algorithm formulation.
- Significance: On a supported tensor-core path, 8-bit integer (INT8) quantization can deliver multi-fold throughput gains because hardware executes lower-precision operations more densely and moves fewer bytes than an FP32 path; the exact gain depends on the architecture and workload (NVIDIA Corporation 2020a; Dally et al. 2021; Dally 2023).
- Distinction: Unlike layered abstraction (where software calls a hardware API without knowing the silicon details), co-design exposes hardware constraints directly to algorithm and compiler authors: data alignment requirements, precision formats, and memory access patterns all become visible inputs to global cross-layer optimization.
- Common pitfall: A frequent misconception is that co-design is a one-time hardware design choice. In practice, the supported features continue to evolve: NVIDIA Tensor Cores began with FP16 matrix multiplication, later generations added TF32 and broader integer support, and Ampere added acceleration for 2:4 structured sparsity (NVIDIA 2017; NVIDIA Corporation 2020a).
Co-design explains why the compression techniques introduced in Model Compression deliver real speedups. The quantization techniques in Quantization and Precision show why converting FP32 to INT8 can accelerate a supported hardware path: fewer bytes reduce traffic, and the accelerator may execute more low-precision operations per cycle than FP32 operations (NVIDIA Corporation 2020a). Structured pruning can improve performance when it produces smaller dense shapes or a sparsity pattern supported by the hardware, whereas arbitrary unstructured sparsity needs matching sparse kernels and metadata handling. Evaluating accelerator performance requires tracing the path from workload to silicon: compute primitives, memory systems, roofline diagnosis, mapping and dataflow, then compiler and runtime support. The recurring question is why some promising algorithmic optimizations survive contact with hardware while others remain paper savings.
Theorem 1.1: The fundamental limit of acceleration (Amdahl's Law)
Hardware acceleration does not speed up the entire system; it speeds only the baseline-time fraction using the accelerated path. For that fraction \(p\) and path gain \(G_{\text{accel}}\), Amdahl’s law for AI (Amdahl 1967) gives the bound in equation 1. \[ \text{Speedup} = \frac{1}{(1 - p) + \frac{p}{G_{\text{accel}}}} \tag{1}\]
- Accelerated fraction (\(p\)): Share of baseline time receiving the stated gain, often in matrix multiplication or another supported kernel.
- Accelerator gain (\(G_{\text{accel}}\)): The raw speed advantage of the GPU or Tensor Processing Unit (TPU) over the CPU for the accelerated portion of the workload.
- Unaccelerated fraction (\(1-p\)): Work outside the chosen path, such as data loading, host overhead, unsupported operators, and kernel launch latency.
Pitfall: Unaccelerated work caps total speedup. If 10 percent of baseline time is unaffected (\(p=0.9\)), even infinite path gain (\(G_{\text{accel}}=\infty\)) yields at most 10\(\times\) overall. Once the accelerated component is fast enough, unaffected work dominates.
Amdahl, Gene M. 1967. “Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities.” Proceedings of the April 18-20, 1967, Spring Joint Computer Conference on - AFIPS ’67 (Spring), AFIPS ’67 (spring), 483–85. https://doi.org/10.1145/1465482.1465560.
1 Amdahl’s law: Maps onto the iron law only after the proposed change is specified. If a faster tensor unit reduces the computation term \((O/(R_{\text{peak}} \cdot \eta_{\text{hw}}))\) but leaves data transfer and fixed latency unchanged, those unaffected terms bound total speedup. A different accelerator or memory system may improve bandwidth as well. The unaccelerated fraction is therefore defined by the intervention, not by a permanent classification of each term as serial.
Hardware acceleration targets specific terms in the iron law of ML systems (Iron Law of ML Systems), which decomposes end-to-end time into data volume \((D_{\text{vol}}/\text{BW})\), computation \((O/(R_{\text{peak}} \cdot \eta_{\text{hw}}))\), and fixed latency \((L_{\text{lat}})\). While data selection reduced the total data and model compression reduced \(O\) per sample, hardware acceleration increases the rate at which those operations execute by improving \(R_{\text{peak}}\), \(\eta_{\text{hw}}\), and \(\text{BW}\). Physics of Computing supplies the analytical performance models that diagnose which of these terms dominates a given workload, including the dimensional analysis that confirms each iron law term resolves to seconds. Yet acceleration has a hard ceiling, established by Amdahl’s Law.1
Amdahl’s Law also explains why many GPU upgrades disappoint. Figure 1 visualizes the acceleration wall: when meaningful baseline time remains unaffected, the ceiling \(1/(1-p)\) makes further gain progressively less valuable unless \(p\) approaches one. The contours are illustrative.
The key intuition to carry into specific hardware architectures is that raw speedups matter only after the unaccelerated fraction has been reduced. The accelerated fraction \(p\) can differ substantially between workload archetypes running on the same hardware, and at fleet scale these differences determine whether an accelerator investment pays off or stalls at the remaining bottleneck.
Checkpoint 1.1: The parallelism gate
These checks turn Amdahl’s bound into a hardware-selection test.
Amdahl’s reality
The serial bottleneck becomes concrete on real hardware, where the same accelerator that nears its parallel ceiling on one workload can stall on another. Numbers to Know collects the reference \(R_{\text{peak}}\) figures across accelerator generations and the latency hierarchy that ground these hardware comparisons in order-of-magnitude terms.
Lighthouse 1.1: An illustrative Amdahl budget on H100
Illustrative ResNet-50 inference on NVIDIA H100:
- Assume H100 provides \(G_{\text{accel}}\) = 247× on matrix multiplication relative to a reference CPU without matrix extensions (1979 TOPS INT8 vs. ~8 TOPS) (Choquette 2023).
- Assume \(p\) = 0.95: 95 percent of baseline time receives that gain, while 5 percent remains in data loading, preprocessing, postprocessing, and other work. \[ \text{$\text{Speedup} = \frac{1}{(1-0.95) + \frac{0.95}{247}} = \frac{1}{0.05 + 0.0038} \approx 18.6\times$} \] The 247× path gain produces only 18.6× end-to-end speedup because the 5 percent unaffected fraction sets the ceiling.
Contrast with GPT-2 (autoregressive):
- For GPT-2 token generation, assume \(p\) = 0.80; the remaining 20 percent includes host overhead, sampling, and decoding work outside the accelerated matrix path. \[ \text{$\text{Speedup} = \frac{1}{(1-0.80) + \frac{0.80}{247}} = \frac{1}{0.20 + 0.0032} \approx 4.9\times$} \] Less token-generation time receives the assumed gain, so even an infinite path gain yields at most \(1/(1-p)\) = 5×. Serving optimizations address this limit differently: batching improves cross-request utilization, while speculative decoding uses a draft model to reduce sequential target-model steps.
2 Arithmetic intensity: The ratio of compute operations to bytes transferred across the memory interface being modeled (FLOP/byte). Workloads above that hardware level’s ridge point, such as well-tiled matrix multiplications with substantial reuse, are compute bound and can benefit from more TFLOP/s. Low-intensity operations, including some low-batch attention and decoding kernels, are bandwidth bound at that level and need more bandwidth or reuse rather than only more compute throughput.
These examples reveal that hardware optimization turns on whether a workload is limited by compute rate or data movement. That distinction determines which accelerator to choose, which optimizations matter, and whether a 10\(\times\) more powerful chip will help. The Roofline Model provides the analytical framework for making this diagnosis (Williams et al. 2009); it is introduced formally in The Roofline model and section 1.5 applies it to AI workloads. It plots an operation’s arithmetic intensity,2 defined as the ratio of floating-point operations to bytes of memory traffic (FLOP/byte), against hardware capabilities, revealing whether performance is capped by compute or bandwidth. A dense matrix multiplication with high arithmetic intensity benefits from more TFLOP/s; a LayerNorm with low arithmetic intensity benefits from more memory bandwidth. High-reuse ResNet-50 convolutions can cross into the compute-bound regime, while low-batch autoregressive attention can be memory bound. This distinction is precisely why these workloads require different optimization strategies.
The chapter develops these ideas from hardware history through computational primitives, memory hierarchies, roofline analysis, mapping, dataflow, and runtime support. The core analysis stays with single-accelerator and single-node systems; the closing material uses multi-device examples only to show how the same bottleneck diagnoses scale. The history of specialized hardware comes first because it reveals the recurring design patterns behind modern accelerators.
Hardware Specialization
The TPUv1/K80 efficiency shock is a modern instance of a recurring hardware pattern: when a workload becomes important and regular enough, specialized hardware can outperform a more general platform. Machine learning acceleration follows a trajectory also seen in floating-point arithmetic, graphics processing, and digital signal processing. Each era specialized the operation or data path that constrained its workload; in modern ML, the cost of feeding dense arithmetic makes data movement a central design target.
Modern ML accelerators (DianNao-class neural-network accelerators (Chen et al. 2014), GPUs with Tensor Cores, Google’s TPUs,3 Apple’s Neural Engine) emerged from these established architectural principles. The evolution spans four phases: specialized computing origins, parallel graphics processing, domain-specific architectures, and the emergence of ML-specific hardware. Each phase reveals design principles that remain relevant for understanding and optimizing contemporary AI systems.
Chen, Tianshi, Zidong Du, Ninghui Sun, Jia Wang, Chengyong Wu, Yunji Chen, and Olivier Temam. 2014. “DianNao: A Small-Footprint High-Throughput Accelerator for Ubiquitous Machine-Learning.” Proceedings of the 19th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS ’14), 269–84. https://doi.org/10.1145/2541940.2541967.
3 TPU (tensor processing unit): The first TPU made a deliberately narrow bet, centering the design on a single \(256{\times}256\) systolic array for 8-bit matrix multiplication and avoiding much of the control machinery required by a general-purpose core (Jouppi et al. 2017). That trade buys high compute density on dense matrix multiplication at the cost of flexibility. Irregular or branch-heavy code remains a poor fit, and compiler tiling, padding, and layer dimensions determine how fully the array is used.
Example 1.1: The TPUv1 vs. K80 efficiency shock
Scenario: In 2015, Google deployed its first Tensor Processing Unit (TPUv1) into production data centers, comparing its throughput and energy efficiency against general-purpose NVIDIA K80 GPUs (Jouppi et al. 2017).
Diagnosis: The K80 was a programmable GPU designed to support a broader mix of graphics and compute workloads. The TPUv1 domain-specific architecture (DSA) made a narrower inference-focused bet, pairing a large 8-bit systolic matrix unit with software-managed memory and simpler control.
Systems lesson: On the six Google inference workloads and contemporary CPU/GPU baselines evaluated in the TPU paper, tailoring silicon to the dominant matrix operations delivered 15–30\(\times\) higher throughput and 30–80\(\times\) better performance per watt (Jouppi et al. 2017). Those ratios describe the evaluated systems, not every workload.
Hardware specialization improves performance by implementing frequent patterns in dedicated circuits, but introduces trade-offs in flexibility, silicon area, and programming complexity. The principles that shaped early floating-point and graphics accelerators now inform AI hardware design.
Specialized computing
Hardware specialization emerges when specific computational patterns become the primary system bottleneck, preventing general-purpose processors from scaling efficiently. This history highlights three recurring limits: slow implementation of scalar floating-point, insufficient throughput for parallel graphics, and poor integration between memory and parallel compute.
The first phase, the precision bottleneck, occurred when scientific and engineering applications required floating-point arithmetic that general-purpose CPUs performed poorly. In the late 1970s, CPUs typically emulated floating-point operations in software, requiring many instructions for a single multiplication. This scalar inefficiency led to the first major instance of hardware specialization: the mathematics coprocessor.
The Intel 8087 (1980)4 addressed this bottleneck by offloading arithmetic-intensive tasks to a dedicated unit. Implementing floating-point logic in hardware greatly accelerated operations that otherwise required software emulation (Palmer 1980). This established a core principle: moving a dominant operation to specialized silicon can provide a substantial speedup.
4 Intel 8087: The coprocessor implemented floating-point logic directly in silicon, avoiding the CPU’s slow, multi-instruction software emulation for each calculation. Its benefit depended on how much time an application spent in supported floating-point operations, illustrating why specialization helps most when the accelerated work dominates execution.
As specialized functions like floating-point math proved their value, they followed a recurring pattern of integration. The Intel 486DX (1989) moved the FPU directly onto the CPU die, eliminating the off-chip communication latency and making high-precision math a standard feature rather than an optional accelerator (Hennessy and Patterson 2017). This cycle, specialization to solve a bottleneck followed by integration into the general-purpose stack, has recurred across several eras of hardware evolution.
The progression from specialization to integration has shaped modern computing. Each domain (graphics, signal processing, machine learning) introduced specialized architectures that were later absorbed into general-purpose platforms.
Figure 2 traces this recurring cycle of specialization and integration across five eras, each addressing the dominant computational bottleneck of its period: the 1980s floating-point and signal-processing units (Intel 8087, TI TMS32010 DSP), 1990s 3D graphics (NVIDIA GeForce 256), 2000s media and network processing (H.264 codecs, Intel IXP2800), 2010s deep-learning tensor operations (Google TPU v1, NVIDIA Tensor Cores), and 2020s application-specific accelerators (AI engines, wafer-scale ML chips). Capabilities such as real-time translation, recommendations, and on-device inference build directly on principles established in these earlier specialization waves.
\begin{tikzpicture}[font=\sffamily\small]
\tikzset{
Box/.style={inner xsep=1pt,
font=\sffamily\footnotesize,
draw=none,node distance=3mm,
fill=#1,align=flush center,
anchor=west,
text width=35mm,
minimum width=35mm, minimum height=10mm
},
Box/.default=red
}
\definecolor{col1}{RGB}{128, 179, 255}
\definecolor{col2}{RGB}{255, 255, 128}
\definecolor{col3}{RGB}{204, 255, 204}
\definecolor{col4}{RGB}{230, 179, 255}
\definecolor{col5}{RGB}{255, 153, 204}
\definecolor{col6}{RGB}{245, 82, 102}
\definecolor{col7}{RGB}{255, 102, 102}
\node[Box={col1}](B1){1980s};
\node[Box={col2},right=of B1](B2){1990s};
\node[Box={col3},right=of B2](B3){2000s};
\node[Box={col4},right=of B3](B4){2010s};
\node[Box={col5},right=of B4](B5){2020s};
\foreach \x in{1,2,...,5}
\draw[dashed,thick,-latex](B\x)--++(270:7.2);
\path[red]([yshift=-5mm]B1.south west)coordinate(P)-|coordinate(K)(B5.south east);
\draw[line width=2pt,-latex](P)--(K)--++(0:3mm);
%
\def\vi{1.1}
\node[Box={col1!50},below=\vi of B1](BB1){Floating-Point \&\\Signal Processing};
\node[Box={col1!50},below=of BB1](BB2){Intel 8087 FPU\\(1980)};
\node[Box={col1!50},below=of BB2](BB3){Texas Instruments\\TMS32010 DSP (1983)};
\node[Box={col1!50},below=of BB3](BB4){Integration of FPU\\into Intel 486DX\\(1989)};
%
\node[Box={col2!50},below=\vi of B2](2BB1){3D Graphics \&\\Multimedia};
\node[Box={col2!50},below=of 2BB1](2BB2){Introduction of\\Early GPUs};
\node[Box={col2!50},below=of 2BB2](2BB3){NVIDIA GeForce 256 --\\First GPU with\\Hardware T\&L (1999)};
\node[Box={col2!50},below=of 2BB3](2BB4){Rise of SIMD\\Processing Units};
%
\node[Box={col3!50},below=\vi of B3](3BB1){Real-time Media\\Coding \&\\Network Processing};
\node[Box={col3!50},below=of 3BB1](3BB2){Media Codecs\\(H.264, MP3)};
\node[Box={col3!50},below=of 3BB2](3BB3){Intel IXP2800\\Network Processor};
\node[Box={col3!50},below=of 3BB3](3BB4){Dedicated hardware\\for streaming\\and encoding};
%
\node[Box={col4!50},below=\vi of B4](4BB1){Deep Learning\\Tensor Operations};
\node[Box={col4!50},below=of 4BB1](4BB2){Google TPU v1 for\\ML Inference (2015)};
\node[Box={col4!50},below=of 4BB2](4BB3){NVIDIA Tensor Cores\\for DL Acceleration};
\node[Box={col4!50},below=of 4BB3](4BB4){AI-specific memory\\optimizations};
%
\node[Box={col5!50},below=\vi of B5](5BB1){Application-Specific\\Acceleration};
\node[Box={col5!50},below=of 5BB1](5BB2){AI Engines \&\\SmartNICs};
\node[Box={col5!50},below=of 5BB2](5BB3){Multi-chip and\\wafer-scale ML\\acceleration};
\node[Box={col5!50},below=of 5BB3](5BB4){ML frameworks\\optimizing for\\specialized hardware};
\end{tikzpicture}Parallel computing and graphics processing
The principles established through floating-point acceleration provided a blueprint for addressing subsequent computational challenges. As computing applications diversified, new computational patterns emerged that exceeded the capabilities of general-purpose processors, and each domain contributed unique insights to hardware acceleration strategies.
Graphics processing emerged as a primary driver of hardware specialization in the 1990s. Early graphics accelerators focused on specific operations like bitmap transfers and polygon filling. NVIDIA’s GeForce 256 in 1999 represented a milestone in specialized computing. The GeForce 256 implemented hardware-accelerated transform and lighting (T&L), moving these computations from CPU to dedicated silicon. While not yet programmable, these Graphics Processing Units (GPUs) demonstrated how fixed-function parallel architectures could efficiently handle data-parallel workloads such as texture mapping and vertex transformation. The transition to programmable shaders with the GeForce 3 (2001) and unified shader architectures with the GeForce 8 (2006) eventually enabled GPU computing for general-purpose workloads. By 2004, high-end GPUs could process over 100 million polygons per second (Owens et al. 2008).
Lyons, Richard G. 2011. Understanding Digital Signal Processing. 3rd ed. Prentice Hall.
Concurrently, Digital Signal Processing (DSP) processors established parallel data path architectures with specialized multiply-accumulate units and circular buffers optimized for filtering and transform operations. Texas Instruments’ TMS32010 (1983) demonstrated how domain-specific instruction sets could dramatically improve performance for signal processing applications (Lyons 2011).
Network processing introduced additional patterns of specialization. Network processors developed unique architectures to handle packet processing at line rate, incorporating multiple processing cores, specialized packet manipulation units, and tiered memory management systems. Intel’s IXP2800 network processor shows the consequence of one hard constraint: meeting line-rate packet deadlines leaves no slack for cache misses, so the design arranges many parallel cores around tiered on-chip memory to keep data adjacent to compute. That compute-near-memory organization, forced here by packet timing, is the same arrangement ML accelerators later adopt to keep their processing-element grids fed.
Across these domains, a common blueprint emerges: identify the dominant computational patterns, build specialized processing elements and memory hierarchies around them, create tailored programming models, and progressively evolve toward more flexible architectures. This pattern of architectural co-evolution established the foundation for contemporary AI hardware design. DSP innovations in low-power signal processing enabled real-time inference on edge devices, including voice assistants and wearables. Together, these domains informed ML hardware designs and demonstrated that accelerators could be deployed across both cloud and embedded contexts.
A single result made the GPU’s relevance to AI unmistakable. AlexNet5 (Krizhevsky et al. 2012) won the 2012 ImageNet competition by more than ten percentage points on two consumer-grade NVIDIA GTX 580 graphics cards, each with only 3 GB of VRAM. The systems lesson was that matching a workload’s data parallelism to GPU hardware could make previously impractical training runs feasible. The era of GPU-centric deep learning had begun.
5 AlexNet: Krizhevsky, Sutskever, and Hinton’s 60-million-parameter convolutional neural network (CNN) won ImageNet 2012 by more than ten percentage points on two consumer GTX 580 GPUs with only 3 GB of VRAM each. Because the model exceeded single-GPU memory, the authors partitioned the network across two cards and limited communication between them, an early form of model parallelism that foreshadowed systematic tensor and pipeline parallelism. The reported training run took five to six days (Krizhevsky et al. 2012).
Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. 2012. “ImageNet Classification with Deep Convolutional Neural Networks.” Advances in Neural Information Processing Systems (NeurIPS) 25.
Emergence of domain-specific architectures
These diverse acceleration patterns converged in a broader architectural shift. The emergence of domain-specific architectures (DSAs)6 marks a transition in computer system design, driven by the slowing benefits of traditional scaling (Esmaeilzadeh et al. 2011) and the increasing computational demands of specialized workloads. Moore’s Law7 described a long-running increase in transistor density (Moore 1998). Dennard scaling8 (Dennard et al. 1974) allowed voltage to fall as transistors shrank; its breakdown removed the accompanying path to routine frequency gains at constant power density. Together, these shifts constrained general-purpose performance and efficiency. As Hennessy and Patterson (2019) argued in the 2017 Turing Lecture, the response was a new era of domain-specific solutions optimized for important workloads.
6 DSA (domain-specific architecture): Hardware optimized for an application domain, sacrificing some general-purpose programmability for efficiency. Google’s TPUv1 achieved 15–30\(\times\) better performance and 30–80\(\times\) better performance per watt than the contemporary CPU and GPU systems evaluated on Google’s inference workloads by centering its design on a systolic array and software-managed memory (Jouppi et al. 2017). A DSA that excels at dense matrix multiplication may perform worse than a CPU on irregular workloads such as graph traversal, so the workload limits and efficiency gain must justify the software, compiler, and ecosystem cost (Hennessy and Patterson 2019, 2017).
Hennessy, John L., and David A. Patterson. 2019. “A New Golden Age for Computer Architecture.” Communications of the ACM 62 (2): 48–60. https://doi.org/10.1145/3282307.
Hennessy, John L., and David A. Patterson. 2017. Computer Architecture: A Quantitative Approach. 6th ed. Morgan Kaufmann.
7 Moore’s law: In the illustrative assumptions used by figure 3, model compute demand grows roughly 6.1× per year while accelerator peak supply improves roughly 1.7× per year, widening the modeled demand/supply gap by about 3.5× per year. Published compute trends motivate the divergence, but the plotted rates are assumptions rather than forecasts. The figure therefore shows how sensitive the systems gap is to sustained differences in growth rates, not a fixed future trajectory. Under such a gap, algorithmic efficiency techniques such as compression, quantization, and sparsity become increasingly valuable (Amodei and Hernandez 2018; Epoch AI 2024).
Moore, G. E. 1998. “Cramming More Components onto Integrated Circuits.” Proceedings of the IEEE 86 (1): 82–85. https://doi.org/10.1109/jproc.1998.658762.
8 Dennard scaling: The 1974 scaling model held that operating voltage could fall as transistor dimensions shrank, keeping power density approximately constant (Dennard et al. 1974). Its breakdown after about 2005 constrained further clock-frequency growth within a chip’s thermal design power; the resulting dark silicon literature estimated, for modeled advanced nodes and design assumptions, that thermal limits could prevent large fractions of available transistors from operating simultaneously. That fraction is not a universal constant, but the power constraint is durable: specialized units make the active silicon budget more useful by dedicating it to high-value operation classes (Esmaeilzadeh et al. 2011).
Dennard, Robert H., Frank H. Gaensslen, Hwa-Nien Yu, Victor L. Rideout, Elias Bassous, and Antoine R. LeBlanc. 1974. “Design of Ion-Implanted MOSFET’s with Very Small Physical Dimensions.” IEEE J. Solid-State Circuits 9 (5): 256–68. https://doi.org/10.1109/jssc.1974.1050511.
Esmaeilzadeh, Hadi, Emily Blem, Renee St. Amant, Karthikeyan Sankaralingam, and Doug Burger. 2011. “Dark Silicon and the End of Multicore Scaling.” Proceedings of the 38th Annual International Symposium on Computer Architecture, 365–76. https://doi.org/10.1145/2000064.2000108.
9 Huang’s law: The observation that GPU performance for AI workloads has improved through architectural innovations such as Tensor Cores as well as transistor scaling. The normalized figure uses illustrative GPU-supply and model-demand curves to show how unequal growth rates create a widening systems gap; it is not a forecast of either rate (Amodei and Hernandez 2018; Epoch AI 2024; NVIDIA Corporation 2020a; Choquette 2023).
Amodei, Dario, and Danny Hernandez. 2018. “AI and Compute.” OpenAI Blog 2.
Epoch AI. 2024. Machine Learning Trends. Epoch AI Research Database.
The scale of this challenge becomes stark in figure 3, which plots an illustrative systems gap between model demand and hardware supply, sometimes discussed under the informal label Huang’s Law.9 The normalized scenario assumes GPU supply rises about 1.7\(\times\) per year while model demand rises about 6\(\times\) per year, producing a gap that widens by roughly 3–4\(\times\) each year. Published compute trends motivate the qualitative divergence, while the plotted rates are scenario assumptions rather than a forecast (Amodei and Hernandez 2018; Epoch AI 2024).
The plot is normalized to a 2012 baseline to emphasize relative growth. Notice how the purple-shaded region between the curves keeps widening—this gap cannot be closed by waiting for faster chips; it requires architectural innovation.
The technology S-curve: Why we must shift
Many computing technologies are described by a lifecycle of ferment (initial slow progress), take-off (rapid growth), and saturation (diminishing returns near physical or economic limits). Figure 4 uses two conceptual S-curves to express the pattern: as gains from one design approach taper, domain-specific architectures can open a new efficiency curve for workloads with stable computational structure.
The routine gains once associated with shrinking transistors have slowed, while some measures of AI model demand have grown much faster than hardware supply. The response cannot be limited to waiting for the next CPU generation; it requires architectural changes that improve how important workloads use transistors, memory bandwidth, and energy. Understanding this transition requires examining the scaling laws that fueled the general-purpose era.
Historically, improvements in processor performance depended on semiconductor process scaling and increasing clock speeds. As power density limitations restricted further frequency scaling and transistor miniaturization encountered increasing physical and economic constraints, architects explored alternative approaches to sustain computational growth. The result was a shift toward domain-specific architectures, which dedicate silicon resources to optimize computation for specific application domains, trading flexibility for efficiency.
Domain-specific architectures achieve superior performance and energy efficiency when the hardware stops treating the workload as arbitrary code. The first shift is a customized data path: matrix multiplication units in AI accelerators, for example, implement systolic arrays, grid-like networks of processing elements that rhythmically compute and pass data through neighboring units. Once that data path is fixed, the memory hierarchy can be tuned around the reuse pattern the workload actually needs, with cache configurations, prefetching logic, and memory controllers designed for the expected tensor flow.
The same specialization then reduces control overhead. Domain-specific instruction sets encode common operation sequences into single instructions, minimizing decode and dispatch complexity, while fixed-function circuit blocks bypass software interpretation for operations that appear constantly. The result is not one trick but a stack of matching decisions: data movement, memory locality, instruction overhead, and circuit implementation all align around the same computational pattern.
Modern smartphones illustrate these principles compellingly. They can decode high-resolution video within tight power and thermal envelopes even though video processing requires billions of operations per second. This efficiency is achieved through dedicated hardware video codecs10 that implement industry standards such as H.264/AVC and H.265/HEVC (Sullivan et al. 2012). These specialized circuits can provide order-of-magnitude performance-per-watt gains compared with software decoding on general-purpose processors, with the exact gain depending on codec, resolution, process node, and CPU baseline.
10 Codec: A portmanteau of “coder-decoder,” reflecting the hardware’s dual function. Encoding (compression) is compute-intensive because it searches for optimal representations, while decoding (decompression) is bandwidth-intensive because it reconstructs full-resolution frames from compressed streams. Dedicated codec silicon implements both paths in fixed-function hardware, so neither path wastes transistors on unrelated general-purpose control logic.
Sullivan, Gary J., Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. 2012. “Overview of the High Efficiency Video Coding (HEVC) Standard.” IEEE Transactions on Circuits and Systems for Video Technology 22 (12): 1649–68. https://doi.org/10.1109/tcsvt.2012.2221191.
Turakhia, Yatish, Gill Bejerano, and William J. Dally. 2018. “Darwin: A Genomics Co-Processor Provides up to 15,000X Acceleration on Long Read Assembly.” Proceedings of the Twenty-Third International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 199–213. https://doi.org/10.1145/3173162.3173193.
11 ASIC (application-specific integrated circuit): These circuits achieve their extreme efficiency by implementing a single algorithm directly in silicon, often improving performance-per-watt by \(10^3\times\) to \(10^5\times\). Examples include cryptographic hashing for blockchain mining and sequence alignment for genomics. The trade-off is total inflexibility: if that core algorithm changes, the ASIC cannot be reprogrammed and becomes obsolete, locking the hardware design to the specific problem version it was built to solve.
Bedford Taylor, Michael. 2017. “The Evolution of Bitcoin Hardware.” Computer 50 (9): 58–66. https://doi.org/10.1109/mc.2017.3571056.
These later domains are not separate anecdotes; they repeat the same bottleneck response. Genomics processing benefits from custom accelerators because long-read assembly contains stable alignment kernels that dedicated hardware can accelerate (Turakhia et al. 2018). Blockchain computation produced application-specific integrated circuits (ASICs)11 for the same reason: cryptographic hashing is fixed enough to justify silicon that trades flexibility for efficiency (Bedford Taylor 2017).
The trajectory yields an engineering rule: the era of “free” performance gains from general-purpose scaling is over. For decades, software engineers could rely on Moore’s Law to accelerate existing code without architectural changes. The breakdown of Dennard scaling forced a decisive change: engineers can no longer wait for faster CPUs to solve computational bottlenecks but must instead design the hardware to fit the algorithm. This necessity of hardware-software co-design is why modern AI engineering requires deep understanding of the underlying silicon. Performance is now determined by how well the algorithm’s memory access patterns and parallelism map to the specialized physical structures of domain-specific architectures.
Machine learning hardware specialization
Many important neural-network workloads contain regular tensor operations, substantial data parallelism, and opportunities for reduced precision. Dense matrix multiplications and convolutions are prominent examples because their loop structure exposes repeated operations and predictable reuse. These characteristics have driven specialized hardware architectures that would be ineffective for arbitrary code but can provide substantial gains on matching ML kernels. Other ML operations remain irregular, bandwidth bound, sequential, or too small to fill a wide engine, so the same device need not accelerate every part of a model. The hardware built to exploit the regular portion constitutes a class of devices known as ML accelerators, and the economic trigger for specialization appears when that supported workload is important at fleet scale rather than only in a benchmark.
Example 1.2: The TPU capacity cliff
Scenario: Google projected in 2013 that user adoption of voice-search speech recognition (3 minutes/day/user) would require doubling global data center capacity to run CPU-based neural network inference (Jouppi et al. 2017).
Diagnosis: General-purpose CPU server capacity was economically unable to absorb the exponential growth in inference FLOPs, threatening a capital infrastructure bottleneck.
Systems lesson: Custom hardware acceleration becomes mandatory when workload volume crosses fleet-level economic thresholds. Aggregate arithmetic intensity and server total cost of ownership drive custom ASIC deployment decisions.
Machine learning computational requirements reveal limitations in traditional processors. In an illustrative scenario where a reference CPU sustains 5 percent–10 percent of its peak on a neural network workload, the upper endpoint delivers approximately 0 GFLOP/s (billions of floating-point operations per second). This inefficiency results from architectural mismatches: CPUs optimize for single-thread performance and irregular memory access, while neural networks require massive parallelism and predictable data streams. The memory bandwidth constraint compounds the problem: a single neural network layer may require accessing gigabytes of parameters, overwhelming CPU cache hierarchies designed for smaller working sets.
The energy economics of data movement influence accelerator design. Per event, accessing a 32-bit value from DRAM can consume on the order of \(10^2\times\) more energy than one FP32 multiply (exact values vary by technology node and design), making data movement a primary optimization target (Horowitz 2014; Sze et al. 2017). This disparity helps explain the progression from repurposed graphics processors to purpose-built neural network accelerators. TPUs and other custom accelerators can sustain high utilization on dense kernels by implementing systolic arrays and other architectures that maximize data reuse while minimizing movement.
IEEE Standards Association. 2019. IEEE 754-2019: Standard for Floating-Point Arithmetic. https://doi.org/10.1109/IEEESTD.2019.8766229.
12 Latency vs. throughput in accelerator design: Training commonly uses throughput-oriented execution and larger batches to amortize work, while latency-sensitive inference must control single-request and tail latency. This is a service objective, not a universal training/inference divide: batched inference can also be throughput oriented, and small or interactive training loops may care about step latency. Pipeline depth, batching delay, kernel setup, and queueing can make a training-oriented path less suitable for an interactive request even when its peak FLOP/s is higher, so the design must match the workload’s actual arrival pattern, batch opportunity, and objective.
Training and inference present different computational profiles that influence accelerator design. Training computes gradients and updates weights: FP32 and FP16 are standardized binary floating-point formats (IEEE Standards Association 2019), while mixed-precision training uses lower-precision tensor operations with higher-precision accumulation when accuracy permits (Micikevicius et al. 2017). Backpropagation also retains or recomputes activations (see Activation memory requirements), increasing memory demand. Inference requires only the forward path and may use INT8 or INT4 when the model and kernels support it. Interactive inference may prioritize tail latency, while offline or batched inference may prioritize throughput, cost, or energy; accelerator choice follows the actual service objective. These profiles often favor high-capacity, throughput-oriented training systems and energy-efficient inference paths, but batch size and service policy can reverse the apparent preference.12
Definition 1.3: ML accelerator
Machine Learning Accelerators are processors or specialized units designed primarily for common neural-network tensor operations and dataflows. They pursue high \(R_{\text{peak}}\) and data reuse on matching workloads by devoting more resources to arithmetic and local data movement than a general-purpose execution path can.
- Significance: An ML accelerator’s defining feature is not raw arithmetic alone but a balanced path that supplies operands and retains reusable values nearby. The A100 provides 2.04 TB/s of device-memory bandwidth, about 10.2× the illustrative 200 GB/s host-DRAM path used here (NVIDIA Corporation 2020a; Choquette et al. 2021). This is a ceiling comparison, not a guarantee: sufficient arithmetic intensity, concurrency, and reuse are still required to approach its 312 TFLOP/s FP16/BF16 peak.
- Distinction: The gains are conditional on operation support, parallel work, regular data access, and a software stack that selects the intended path. An ML accelerator can be orders of magnitude faster than a reference CPU on large dense matrix multiplication, yet lose much of that advantage on small, irregular, synchronization-heavy, or control-heavy workloads. Transfers and unsupported operators can dominate even when the accelerated kernel itself is fast.
- Common pitfall: A frequent misconception is that ML accelerators always accelerate ML. Approaching peak throughput requires a compatible precision, operation shape, layout, implementation, and enough parallel work to keep the relevant units busy. Utilization also depends on whether memory and interconnect paths can sustain that work. A batch-1 autoregressive request may therefore use only a small fraction of a large training accelerator’s arithmetic capacity even though the same device performs well on batched prefill or training.
Choquette, Jack, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. 2021. “NVIDIA A100 Tensor Core GPU: Performance and Innovation.” IEEE Micro 41 (2): 29–35. https://doi.org/10.1109/mm.2021.3061394.
A DRAM access costs over 100\(\times\) one FP32 multiply per event.
Deployment context shapes architectural choices by identifying the binding constraint. In data centers, the constraint is often time-to-result for training massive models. An NVIDIA H100 trades hundreds of watts of power for high throughput (Choquette 2023); whether that trade lowers total cost depends on utilization, rental price, workload scaling, and the electricity rate. Google’s TPUv4 makes a similar architectural trade, prioritizing throughput through systolic arrays and high-bandwidth memory (Jouppi et al. 2023).
Checkpoint 1.2: The accelerator gate
Use the energy hierarchy to decide when specialization pays.
Energy inversion
Selection logic
At the edge, the priority often shifts toward energy per inference and hard latency or power limits, although throughput remains important for continuous camera, audio, or sensor streams. A smartphone camera or always-on audio path operating inside a few-watt budget cannot simply adopt a data center accelerator’s high-power memory system. Edge architectures instead reduce movement through local scratchpads, tightly integrated accelerators, dynamic voltage and frequency scaling, and event-driven processing when the workload permits. The same memory-wall principle applies in both settings: data center chips invest in high-bandwidth memory (HBM) capacity and bandwidth, while edge chips depend heavily on proximity and reuse.
No single architecture dominates every ML workload. Edge devices favor energy efficiency and bounded latency, while cloud-scale training values throughput, capacity, and interconnect performance. Cloud inference can emphasize cost per request or tail latency, and an edge device may still need sustained throughput for a real-time stream. Specialized architectures therefore reflect their deployment context, yet all remain subject to the energy and latency costs of moving data.
Table 1 summarizes these milestones in hardware specialization. Floating-point coprocessors accelerated arithmetic previously implemented in software, early GPUs increased graphics throughput, and media engines showed how stable pipelines could justify fixed-function blocks. AI accelerators combine dense tensor units with memory hierarchies and software support, making integration between data movement and parallel execution a central challenge.
Modern AI acceleration therefore extends beyond the chip. Useful accelerators need framework, compiler, library, driver, and runtime support for graph transformations, kernel selection, fusion, memory scheduling, and deployment across environments from data centers to edge devices. Unsupported operations or surrounding transfers can erase a kernel-level gain.
| Era | Computational pattern | Architecture examples | Characteristics |
|---|---|---|---|
| 1980s | Floating-Point & Signal Processing | FPU, DSP | • Single-purpose engines • Focused instruction sets • Coprocessor interfaces |
| 1990s | 3D Graphics & Multimedia | GPU, SIMD Units | • Many identical compute units • Regular data patterns • Wide memory interfaces |
| 2000s | Real-time Media Coding | Media Codecs, Network Processors | • Fixed-function pipelines • High throughput processing • Power-performance optimization |
| 2010s | Deep Learning Tensor Operations | TPU, GPU Tensor Cores | • Matrix multiplication units • Massive parallelism • Memory bandwidth optimization |
| 2020s | Application-Specific Acceleration | ML Engines, Smart NICs, Domain Accelerators | • Workload-specific datapaths • Customized memory hierarchies • Application-optimized designs |
The integration bottleneck
For many accelerator workloads, adding arithmetic units is easier than feeding them efficiently. Early coprocessors accelerated floating-point arithmetic implemented in software, and GPUs increased throughput for graphics and later general compute. Modern AI systems must also solve an integration bottleneck: moving operands and results through a memory hierarchy quickly and efficiently enough to sustain many parallel compute units.
Three properties of many neural-network kernels make this integration tractable. Large matrix multiplications and convolutions expose data parallelism that dense processing-element arrays can exploit. Static or compiled tensor graphs often have analyzable dataflow, allowing compilers to tile transfers into local scratchpads13 instead of relying only on hardware-managed caches. This gives software explicit control over selected, regular working sets without eliminating caches or irregular access. Selected operations may also tolerate reduced precision, allowing supported 8-bit or 4-bit paths to increase compute density and reduce bytes per value (Dally et al. 2021; Dally 2023). Dynamic control flow, embeddings, small tensors, variable shapes, and accuracy-sensitive operators fit these assumptions less well and can remain bottlenecks.
13 Scratchpad memory: When a compiler can determine a tensor kernel’s access pattern, it can explicitly stage tiles in fast, software-controlled local memory. This avoids some tag, replacement, and coherence machinery, but transfers, synchronization, banking, and capacity must be scheduled correctly. Scratchpads complement rather than universally replace caches because irregular or shared data may still benefit from hardware management. Google’s TPU v1, for example, used a 24 MB software-managed Unified Buffer for intermediate activations, while weights and instructions used other paths (Jouppi et al. 2017).
Jouppi, Norman P., Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, et al. 2017. “In-Datacenter Performance Analysis of a Tensor Processing Unit.” Proceedings of the 44th Annual International Symposium on Computer Architecture, ISCA ’17, 1–12. https://doi.org/10.1145/3079856.3080246.
14 HBM: The A100 and H100 generations provide 2.0–3.4 TB/s of device-memory bandwidth through stacked memory and wide interfaces, compared with 760 GB/s for the GDDR6X reference used here (NVIDIA Corporation 2020a; Choquette 2023). Raising the bandwidth ceiling can move some kernels toward the compute-bound side of the roofline, but only if their access pattern and arithmetic intensity can use it. HBM also increases packaging and system cost, so its value depends on the workload’s bandwidth demand.
Once arithmetic capacity is abundant, the engineering challenge becomes keeping data close enough to use it. A DRAM access can consume more than 100\(\times\) the energy of a low-precision arithmetic operation on the technology assumptions summarized by Horowitz (2014). This hierarchy helps explain why accelerator designs invest in HBM, on-chip SRAM, and software-managed scratchpads rather than only adding compute units.14
Hardware acceleration addresses the memory bottleneck through specialized spatial layout and multi-tiered storage. Examine the architectural blueprint in figure 5, noting how high-bandwidth memory feeds local scratchpads surrounding the processing element grid.
\begin{tikzpicture}[line cap=round,line join=round,font=\sffamily\small]
\tikzset{
Box/.style={align=center,outer sep=0pt,
inner xsep=2pt,
node distance=0.45,
draw=GreenLine,
line width=0.75pt,
fill=GreenL!60,
% text width=32mm,
minimum width=77mm, minimum height=11mm
},
Box2/.style={Box, minimum width=10mm, minimum height=6mm,fill=BrownL!60,draw=BrownLine},
Box3/.style={Box,text width=20mm, minimum width=20mm, minimum height=9mm,fill=RedL!60,draw=RedLine},
Box4/.style={Box3, fill=BlueLine!20,draw=BlueLine},
Box5/.style={Box3, fill=OrangeLine!20,draw=OrangeLine},
Box6/.style={Box3, text width=27mm, minimum width=27mm, minimum height=13mm,fill=OrangeLine!20,draw=OrangeLine},
Line/.style={violet!50, line width=1.1pt,shorten <=1pt,shorten >=2pt},
LineA/.style={violet!50,line width=0.8pt,{-{Triangle[width=1.0*4pt,length=1.0*6pt]}},shorten <=1pt,shorten >=1pt},
ALine/.style={black!50, line width=1.1pt,{{Triangle[width=0.9*6pt,length=1.2*6pt]}-}},
Larrow/.style={fill=violet!50, double arrow, inner sep=2pt, double arrow head extend=3pt,
single arrow head indent=0pt,minimum height=17mm, minimum width=3pt}
}
\tikzset{
pics/dram/.style = {
code = {
\pgfkeys{/channel/.cd, #1}
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
\node[draw=\drawcolor,fill=\filllcolor!70,line width=1.5*\Linewidth,inner sep=0pt,outer sep=0pt,
minimum width=56mm,minimum height=14mm](DRAM\picname)at(0,0){};
\node[draw=\drawcolor,fill=\filllcolor!30,line width=1.5*\Linewidth,inner sep=0pt,outer sep=0pt,anchor=north,
minimum width=52mm,minimum height=6mm](MDRAM\picname)at(DRAM\picname.south){};
%
\pgfmathsetmacro{\spacing}{56/(6+1)}
\foreach \i in {1,...,6} {
\pgfmathsetmacro{\x}{\i * \spacing}
\node[draw=\drawcolor,fill=\filllcolor!20,line width=\Linewidth, inner sep=0pt, outer sep=0pt,
minimum width=6mm, minimum height=8mm]
at ([xshift=\x mm]DRAM\picname.west) {};
}
%
\foreach \i in {1,...,19} {
\pgfmathsetmacro{\x}{\i*(52/20)}
\draw[draw=\drawcolor, line width=3*\Linewidth]
([xshift=\x mm,yshift=1pt]MDRAM\picname.south west) -- ++(0,2mm);
}
\end{scope}
}
}
}
%CPU style
\tikzset{
pics/cpu/.style = {
code = {
\pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box = CPU,scale=0.6, every node/.append style={transform shape}]
\node[fill=\filllcolor,minimum width=66, minimum height=66,
rounded corners=2,outer sep=2pt] (C1) {};
\node[fill=white,minimum width=54, minimum height=54] (C2) {};
\node[fill=\filllcolor!50,minimum width=44, minimum height=44] (C3) {\large CPU};
\foreach \x/\y in {0.11/1,0.26/2,0.41/3,0.56/4,0.71/5,0.85/6}{
\node[fill=\filllcolor,minimum width=4, minimum height=15,
inner sep=0pt,anchor=south](GO\y)at($(C1.north west)!\x!(C1.north east)$){};
}
\foreach \x/\y in {0.11/1,0.26/2,0.41/3,0.56/4,0.71/5,0.85/6}{
\node[fill=\filllcolor,minimum width=4, minimum height=15,
inner sep=0pt,anchor=north](DO\y)at($(C1.south west)!\x!(C1.south east)$){};
}
\foreach \x/\y in {0.11/1,0.26/2,0.41/3,0.56/4,0.71/5,0.85/6}{
\node[fill=\filllcolor,minimum width=15, minimum height=4,
inner sep=0pt,anchor=east](LE\y)at($(C1.north west)!\x!(C1.south west)$){};
}
\foreach \x/\y in {0.11/1,0.26/2,0.41/3,0.56/4,0.71/5,0.85/6}{
\node[fill=\filllcolor,minimum width=15, minimum height=4,
inner sep=0pt,anchor=west](DE\y)at($(C1.north east)!\x!(C1.south east)$){};
}
\end{scope}
} }}
\pgfkeys{
/channel/.cd,
Depth/.store in=\Depth,
Height/.store in=\Height,
Width/.store in=\Width,
filllcirclecolor/.store in=\filllcirclecolor,
filllcolor/.store in=\filllcolor,
drawcolor/.store in=\drawcolor,
drawcircle/.store in=\drawcircle,
scalefac/.store in=\scalefac,
Linewidth/.store in=\Linewidth,
picname/.store in=\picname,
filllcolor=BrownLine,
filllcirclecolor=BlueFill,
drawcolor=black,
drawcircle=violet,
scalefac=1,
Linewidth=0.5pt,
Depth=1.3,
Height=0.8,
Width=1.1,
picname=C
}
\node[Box](B1){L2 Cache (Shared)};
\coordinate(PO1)at($(B1.north west)+(0.5,0.65)$);
\pgfmathsetmacro{\spacing}{47/(2+1)}
\foreach \i [count=\j] in {0,...,2} {
\pgfmathsetmacro{\x}{\i * \spacing}
\node[Box2,anchor=south west](GPE\j)at([xshift=\x mm]PO1){PE};
}
\node[Box2,right=1.5 of GPE3](GPE4){PE};
\node[font=\tiny\sffamily]at($(GPE3)!0.5!(GPE4)$){$\bullet$ $\bullet$ $\bullet$};
%
\coordinate(PO2)at($(B1.south west)+(0.5,-0.65)$);
\pgfmathsetmacro{\spacing}{47/(2+1)}
\foreach \i [count=\j]in {0,...,2} {
\pgfmathsetmacro{\x}{\i * \spacing}
\node[Box2,anchor=north west](DPE\j)at([xshift=\x mm]PO2){PE};
}
\node[Box2,right=1.5 of DPE3](DPE4){PE};
\node[font=\tiny\sffamily]at($(DPE3)!0.5!(DPE4)$){$\bullet$ $\bullet$ $\bullet$};
%arrows
\foreach \i in {1,...,4} {
\draw[LineA](B1.south)--++(0,-0.25)
-|(DPE\i.north);
}
\foreach \i in {1,...,4} {
\draw[LineA](B1.north)--++(0,0.25)-|(GPE\i.south);
}
\begin{scope}[shift={($(GPE1)+(0.3,2.8)$)}]
\node[Box3](L1){L1 Cache / Scratchpad};
\node[Box4,above right=-0.10 and 0.3 of L1](TC){Tensor Core};
\node[Box4,below right=0.1 and 0.3 of L1](VU){Vector Unit};
\node[Box5,below right=0 and 0.3 of TC](SFU){SFU};
\draw[LineA](L1)|-(TC);
\draw[LineA](L1)|-(VU);
\draw[LineA](TC)-|(SFU);
\draw[LineA](VU)-|(SFU);
%%fitting
\scoped[on background layer]
\node[draw=BackLine,fill=BackColor!20, inner ysep=4mm, inner xsep=2mm,yshift=2mm,
fit=(L1)(TC)(SFU)(VU),yshift=0mm](BB1){};
\node[below left=0 and 0 of BB1.north east]{Processing Element};
\scoped[on background layer]
\fill[BrownLine!10](GPE3.north west)--(BB1.south west)--(BB1.south east)--(GPE3.north east)--cycle;
\draw[BrownLine](GPE3.north west)--(BB1.south west) (BB1.south east)--(GPE3.north east);
\end{scope}
%%fitting
\node[draw=red,dashed,fill=none, inner ysep=4mm, inner xsep=3mm,yshift=2mm,
fit=(BB1)(DPE1)(DPE4)(B1),yshift=0mm](BB2){};
\node[below =0pt of BB2.north]{AI Accelerator Chip};
%CPU
\begin{scope}[local bounding box=CPU1,shift={($(B1)+(-8.1,0)$)}]
\pic[shift={(0,0)}] at (0,0) {cpu={scalefac=1,picname=1,drawcolor=BlueLine,filllcolor=BlueLine!80!,Linewidth=0.5pt}};
\end{scope}
\node[above=6pt of CPU1]{Host CPU};
%%%%
\begin{scope}[local bounding box=DRAM1,shift={($(CPU1)+(0,-1.9)$)},scale=1, every node/.append style={transform shape}]
\pic[shift={(0,0)}] at (0,0){dram={scalefac=0.45,picname=1,drawcolor=black,filllcolor=OrangeLine!50!,Linewidth=0.5pt}};
\end{scope}
\node[below=9pt of DRAM1]{Host DRAM};
\node[Larrow](AR1)at($(CPU1.east)!0.45!(B1.west)$){};
\node[align=center,above=2pt of AR1,font=\sffamily\footnotesize]{Host Interface\\ (PCIe/NVLink)};
\draw[LineA,dashed](DRAM1)--(CPU1);
%%
\node[Box6,right=2.5 of B1](B6){High-Bandwidth Memory (HBM)};
\node[Larrow](AR2)at($(B1.east)!0.55!(B6.west)$){};
\node[align=center,above=2pt of AR2,font=\sffamily\footnotesize]{Memory\\ Interface};
\end{tikzpicture}The evolution from the Intel 8087 to the Google TPU reveals a consistent pattern: hardware evolves to fit the algorithm’s dominant bottleneck. Where the 8087 addressed floating-point operations that dominated many scientific workloads, modern AI accelerators address dense matrix and convolution operations that dominate much of neural-network training and inference (Palmer 1980; Goodfellow et al. 2016; Sze et al. 2017; Jouppi et al. 2017). This concentration of demand explains why specialized AI silicon can deliver large performance-per-watt improvements over general-purpose processors on matching workloads.
These three opportunities—parallel work, analyzable dataflow, and supported lower precision—shape many accelerator decisions. Their value depends on the accelerator’s physical organization and on how well software maps the workload onto it.
In the generic design in figure 5, an array of processing elements contains units for different operation classes: matrix units execute matrix multiplication, vector units perform element-wise work, and special function units approximate functions such as exponentials. The number, organization, and names of these units vary by architecture, but all seek to exploit data-level parallelism across tensor tiles. Some designs expose an explicit systolic grid; others distribute matrix units among wider parallel cores. The schematic therefore represents functional roles, not a universal physical topology.
The memory hierarchy is equally important. High-bandwidth memory supplies off-chip capacity and aggregate throughput, while shared on-chip storage and per-element caches or scratchpads reduce traffic and energy. The exact hierarchy differs across products: some designs rely more on hardware-managed caches, others expose software-managed SRAM, and many combine both. Moving a tensor tile downward through this hierarchy costs time and energy, so compilers and kernels coordinate capacity, reuse, prefetching, and double buffering. More HBM bandwidth raises the off-chip ceiling but does not remove the need for local reuse. The common purpose is to keep reusable weights, activations, and partial results near the units that consume them. The machine foundations appendix collects reference specifications for modern accelerators, including H100 and TPU v5, and summarizes the latency hierarchy.
The host interface connects the accelerator to the broader system. A CPU commonly coordinates program execution, I/O, and unsupported operations, while the accelerator executes the kernels assigned to it. This partition is not absolute: accelerators can schedule work locally, and CPUs can execute tensor operations. What matters is where each operator runs and which transfers, launches, and synchronization points surround it. Direct-memory-access engines and device-side work queues can overlap some of that overhead with computation, but unsupported operators may still force costly handoffs. The path from host through device memory and on-chip storage to the processing elements therefore defines the useful boundary of acceleration. A fast kernel does not yield a fast application when communication across that boundary dominates end-to-end time.
With the accelerator’s physical architecture established, the next step is to explain why these specific components dominate. Tensor Cores, vector units, and hierarchical memory do not exist by accident; they exist because neural network computations repeatedly invoke a small set of operations. Understanding these patterns is essential because they explain which algorithmic changes translate to real speedups (those that align with hardware primitives) and which remain purely theoretical.
Self-Check: Question
What primary physical limitation brought about the end of Dennard scaling in the mid-2000s, necessitating the transition from increasing CPU clock frequencies to domain-specific hardware accelerators?
- Lithography light diffraction preventing further reduction of transistor gate length below \(1\,\mu\text{m}\)
- Inability to lower operating voltage proportionally with transistor size, leading to unsustainable power density and heat dissipation limits
- Quantum tunneling in copper interconnect lines preventing data transmission between arithmetic units
- Depletion of global silicon substrate supplies requiring migration to gallium nitride semiconductors
Contrast the architectural trade-offs of software-managed scratchpad memory (such as Google TPUv1’s Unified Buffer) with hardware-managed cache hierarchies when executing large-scale tensor workloads.
Place the following computing milestones in chronological order (from earliest to most recent) as hardware evolved toward modern AI accelerators:
- Introduction of dedicated Tensor Cores and TPUs for deep learning matrix operations
- Emergence of fixed-function media codecs and network processors for video/packet streaming
- Integration of floating-point units (FPUs) and digital signal processors (DSPs) as discrete coprocessors
- General-purpose programmable GPUs and SIMD instruction set extensions for 3D graphics and multimedia
True or False: Google’s TPUv1 achieved substantial performance-per-watt improvements over contemporary general-purpose CPUs primarily by operating at significantly higher clock frequencies.
In the context of hardware scaling, what phenomenon does the ‘Systems Gap’ describe since the 2012 deep learning breakthrough?
- The difference in memory bandwidth between high-end datacenter GPUs and consumer-grade mobile SoCs
- The latency discrepancy between on-chip SRAM access times and host DRAM access times over PCIe
- The exponential divergence between model compute demand (growing \(\approx 6\times\)/year) and single-device hardware supply (growing \(\approx 1.7\times\)/year)
- The mismatch between Python framework dispatch overhead and raw GPU kernel execution duration
AI Compute Primitives
Across fully connected, convolutional, and attention layers, learned linear transformations reduce to repeated multiply-accumulate (MAC) operations. A large layer can issue millions or billions of MACs, and its regular tensor structure exposes parallel work and reuse, making it a natural specialization target. MACs do not dominate every model or phase—normalization, routing, indexing, communication, and element-wise operations can become limiting—but they account for much of the peak arithmetic capacity in modern ML accelerators. Whether they dominate elapsed time still depends on tensor shape, data movement, and the surrounding operators.
The hardware units that exploit these patterns are AI compute primitives: specialized functional blocks, each optimized for a particular class of operation. Three primitives are especially common in accelerators, each targeting a distinct computational pattern found in neural networks.
Listing 1 demonstrates how a dense layer decomposes at the framework level, encapsulating thousands of multiply-accumulate operations in a single high-level call.
# Framework abstracts compute-intensive operations
dense = Dense(512)(input_tensor) # 256x512 MACs per sampleThis single line of code conceals the computational complexity that accelerators must handle. Listing 2 reveals how the framework expands this high-level call into mathematical operations.
# Linear transformation work scales with input_dim x output_dim x
# batch.
output = (
matmul(input, weights) + bias
) # Matrix multiply dominates cost
output = activation(
output
) # Element-wise: proportional to output_dim x batchThe matrix multiplication dominates computation time, but this abstraction still hides the underlying loop structure. At the processor level, listing 3 reveals how nested loops multiply inputs and weights, sum the results, and apply a nonlinear function, exposing the \(\mathcal{O}(B \times d_{\text{in}} \times d_{\text{out}})\) complexity that accelerators must handle efficiently.
# Total operations: batch_size × output_size × input_size MACs
for n in range(batch_size): # Batch dimension: parallelizable
for m in range(output_size): # Output neurons: parallelizable
sum = bias[m] # Initialize accumulator
for k in range(input_size): # Reduction dimension: sequential
sum += input[n, k] * weights[k, m] # MAC operation
output[n, m] = activation(sum) # Nonlinear transformation
# Example work scales as batch_size × output_size ×
# input_size multiply-accumulate operationsThis loop structure reveals three computational patterns that recur across many neural-network architectures: element-wise operations along vectors, matrix-level reductions, and nonlinear transformations. Their frequency justifies specialized execution paths, although the gain depends on tensor shape, data movement, and operator support. These patterns become hardware blocks: vector units handle independent elements and reductions, matrix engines organize multiply-accumulate work across tiles, and special-function paths approximate nonlinear math. The boundaries are architectural rather than mathematical; one fused kernel may use several paths without exposing the transitions to the framework.
Vector operations
Vector operations provide one level of acceleration by processing multiple data elements per instruction. Recall the nested-loop structure exposed in listing 3: a batch of 32 samples through a 256-to-512 dense layer requires 4.2M MACs. A scalar formulation loads one input-weight pair, multiplies it, and updates one accumulator at a time. Vector execution performs that work across several lanes, amortizing instruction decoding, loop control, and address generation. Its advantage depends on vector width, alignment, tail handling, and enough memory bandwidth to supply the lanes.
fmv.w.x fa0, zero # Scalar dot-product accumulator
fmv.w.x ft1, zero # Zero seed for vector reduction
loop_feature:
vsetvli t0, feature_cnt, e32, m1, ta, ma
vle32.v v1, (in_ptr)
vle32.v v2, (wt_ptr)
vfmul.vv v3, v1, v2
vfmv.v.f v0, ft1
vfredusum.vs v4, v3, v0
vfmv.f.s ft0, v4
fadd.s fa0, fa0, ft0
slli t1, t0, 2 # Four bytes per FP32 element
add in_ptr, in_ptr, t1
add wt_ptr, wt_ptr, t1
sub feature_cnt, feature_cnt, t0
bnez feature_cnt, loop_featureVector processing units solve this by operating on multiple data elements simultaneously. RISC-V,15 an open instruction-set architecture (ISA) with a standardized vector extension (Waterman et al. 2013), provides a useful setting for illustrating this idea. Listing 4 uses vector-style assembly code in which a single instruction processes several data elements. The loop has five hardware-visible stages:
15 RISC-V (reduced instruction set computer V): The open ISA allows hardware teams to add custom ML instructions, including vector dot products, activation functions, and sparse tensor operations, without changing the base ISA. The trade-off is software ecosystem maturity: custom extensions require corresponding compiler, library, and runtime support, which can limit portability and increase integration work.
Waterman, Andrew, Yunsup Lee, Rimas Avizienis, Henry Cook, David Patterson, and Krste Asanovic. 2013. “The RISC-V Instruction Set.” 2013 IEEE Hot Chips 25 Symposium (HCS), 1–1. https://doi.org/10.1109/hotchips.2013.7478332.
- Vector length configuration: Configures the vector units to process 32-bit elements, automatically determining how many operations happen in parallel based on hardware width (VLEN).
- Vector initialization: Clears the scalar accumulator that will hold the completed dot product.
- Vector loads: Loads contiguous 32-bit values into
v1andv2, using one vector-load instruction for each source. - Vector multiply and reduction: Multiplies corresponding elements and reduces the products to a partial scalar sum.
- Pointer arithmetic: Advances the pointers and decrements the remaining element count by the run-time vector length.
In this assembly sequence, the vector multiply instruction processes several element pairs at once, while the reduction completes each partial dot product and the vector loads amortize instruction overhead across multiple data elements. The run-time vector length determines how many values each iteration transfers and processes, so the same loop works across implementations with different vector widths. The illustrative workload is divided into roughly 524,288 vector chunks, each executed by several vector and scalar instructions.
Key vector operations map directly to common deep learning patterns. Table 2 enumerates how operations such as reduction, gather, scatter, and masked operations appear frequently in pooling, embedding lookups, and attention mechanisms, clarifying the direct mapping between low-level vector hardware and high-level machine learning workloads.
| Vector operation | Description | Neural network application |
|---|---|---|
| Reduction | Combines elements across a vector (for example, sum, max) | Pooling layers, attention score computation |
| Gather | Loads multiple nonconsecutive memory elements | Embedding lookups, sparse operations |
| Scatter | Writes to multiple nonconsecutive memory locations | Gradient updates for embeddings |
| Masked operations | Selectively operates on vector elements | Attention masks, padding handling |
| Vector-scalar broadcast | Applies scalar to all vector elements | Bias addition, scaling operations |
The benefit extends beyond instruction count. Contiguous vector loads can use memory interfaces efficiently, and control overhead is amortized across several data elements. Gather, scatter, masking, and short tails may use fewer lanes or require extra memory transactions, so vector width alone does not guarantee proportional speedup. The architectural pattern is not new. The Cray-116 used vector registers and pipelined functional units for scientific computing in the 1970s (Jordan 1982); modern ML processors apply related principles at much larger commercial scale.
16 Cray-1 vector legacy: The Cray-1 (1975) used 64-element vector registers with pipelined functional units, allowing a stream of elements to advance through arithmetic operations without issuing one scalar instruction per element. Modern accelerators extend the same principles of wide registers, pipelined execution, and data reuse to vector and matrix tiles.
Jordan, T. L. 1982. “A Guide to Parallel Computation and Some Cray-1 Experiences.” In Parallel Computations. Elsevier. https://doi.org/10.1016/b978-0-12-592101-5.50006-3.
Vector operations excel at element-wise transformations like activation functions, where each output depends only on its corresponding input. Neural networks, however, also require structured computations where each output depends on all inputs—the weighted sums that define layer transformations. These many-to-many operations naturally express themselves as matrix multiplications, our second compute primitive.
Matrix operations
Matrix multiplication accounts for much of the arithmetic in many dense neural networks, transforming high-dimensional data through structured patterns of weights, activations, and gradients (Goodfellow et al. 2016). While vector operations process elements independently or reduce them, matrix operations organize work across multiple dimensions. Hardware and libraries tile those dimensions so blocks of weights and activations can be reused while partial sums accumulate locally. This regular structure drives important hardware strategies, but small or irregular matrices may leave tile capacity unused.
Matrix operations in neural networks
Neural network computations decompose into hierarchical matrix operations. Listing 5 captures this hierarchy through a linear layer that transforms input features into output neurons over a batch.
layer = nn.Linear(256, 512) # Layer transforms 256 inputs to 512 outputs
output = layer(input_batch) # Process a batch of 32 samples
# Framework Internal: Core operations (column-batch convention)
Z = matmul(weights, input) # Matrix: transforms [256×32]
# input to [512×32] output
Z = Z + bias # Vector: adds bias to each
# output independently
output = relu(Z) # Vector: applies activation to
# each element independentlyThis computation demonstrates the scale of matrix operations in neural networks. Each output neuron (512 total) must process all input features (256 total) for every sample in the batch (32 samples). The weight matrix alone contains 256 \(\times\) 512 = 131,072 parameters that define these transformations, illustrating why efficient matrix multiplication dominates performance considerations.
Neural networks employ matrix operations across architectural patterns beyond simple linear layers. Convolution operations transform into matrix multiplications through the im2col technique,17 enabling efficient execution on matrix-optimized hardware. Listing 6 illustrates these applications.
17 Im2col (image-to-column): Transforms convolution into a matrix multiplication by arranging receptive-field values as columns. An explicit materialization can duplicate overlapping values and expand storage by up to roughly the kernel area before edge, stride, and channel effects are considered. Production libraries often use implicit-GEMM or direct-convolution kernels to obtain matrix-friendly execution without materializing the full expanded matrix.
hidden = matmul(weights, inputs)
# weights: [out_dim x in_dim], inputs: [in_dim x batch]
# Result combines all inputs for each output
# Attention Mechanisms - Multiple matrix operations
Q = matmul(Wq, inputs)
# Project inputs to query space [query_dim x batch]
K = matmul(Wk, inputs)
# Project inputs to key space [key_dim x batch]
attention = matmul(Q.T, K)
# Compare all query tokens with all key tokens [batch x batch]
# Convolutions - Matrix multiply after reshaping
patches = im2col(input)
# Convert [H x W x C] image to matrix of patches
output = matmul(kernel, patches)
# Apply kernels to all patches simultaneouslyThese examples differ in tensor shape and reuse pattern, which determines how efficiently each multiplication occupies a matrix unit. Linear layers reuse one weight matrix across a batch; attention forms projections and then a token-to-token score matrix; convolution reuses each kernel across many image patches. Expressing all three as matrix multiplication exposes a common interface to hardware, but it does not make their execution costs identical. Batch size, sequence length, channel count, and tiling determine whether operands are reused locally or repeatedly fetched from memory.
Matrix operations hardware acceleration
This pervasive pattern of matrix multiplication has direct implications for hardware design: accelerators need specialized units that can handle these computations at scale. Listing 7 demonstrates a representative dedicated matrix unit that processes an entire \(16{\times}16\) block at once, illustrating why matrix instructions and Tensor Cores can deliver much higher throughput than scalar or vector-only execution paths (NVIDIA 2017; Intel Corporation 2021a).
mload mr1, (weight_ptr) # Load e.g., 16x16 block of
# weight matrix
mload mr2, (input_ptr) # Load corresponding input block
matmul.mm mr3, mr1, mr2 # Multiply and accumulate entire
# blocks at once
mstore (output_ptr), mr3 # Store computed output blockThe illustrative unit operates on matrix tiles rather than individual elements. A full \(16{\times}16{\times}16\) tile product contains \(16^3=4{,}096\) multiply-accumulate operations, but the cycles required depend on the actual instruction and architecture. Sustained throughput also depends on the number of matrix units, operand precision, issue rate, pipeline behavior, and whether memory can keep the units fed. Matrix units complement vector execution by accelerating structured many-to-many transformations.
Like vector processing, matrix acceleration has deep historical roots—DSPs and GPUs optimized for matrix computations in the 1980s-1990s for image processing, scientific computing, and 3D rendering (Owens et al. 2008; Hwu 2011). Neural networks have made matrix multiplication commercially dominant, driving the development of dedicated Tensor Cores and TPUs that process these operations at unprecedented scale.
Owens, John D., Mike Houston, David Luebke, Simon Green, John E. Stone, and James C. Phillips. 2008. “GPU Computing.” Proceedings of the IEEE 96 (5): 879–99. https://doi.org/10.1109/jproc.2008.917757.
Hwu, Wen-mei W. 2011. “Introduction.” In GPU Computing Gems Emerald Edition. Elsevier. https://doi.org/10.1016/b978-0-12-384988-5.00064-4.
Matrix and vector operations together handle the linear algebra of neural networks. Between every linear transformation, however, sits a nonlinear activation function—and these transcendental computations (exponentials, square roots, trigonometric functions) cannot be efficiently expressed through multiply-accumulate alone. Table 3 summarizes the three execution roles, clarifying which neural network operations map to each.
| Operation type | Best For | Examples | Key characteristic |
|---|---|---|---|
| Matrix Operations | Many-to-many transforms | Layer transformations, attention, convolutions | Each output depends on multiple inputs |
| Vector Operations | Vector and reduction work | Activation functions, layer normalization, element-wise gradients | Uses element-wise operations and vector reductions |
| Special-Function Paths | Transcendental arithmetic | Exponential, logarithm, reciprocal square root | Uses architecture-specific approximation or dedicated logic |
Special function units
Special Function Units (SFUs) or specialized instruction paths accelerate nonlinear functions and related arithmetic, completing the chapter’s trio of processing primitives. The need is not new: floating-point coprocessors addressed scalar arithmetic bottlenecks (Palmer 1980), and digital signal processors added specialized arithmetic for signal-processing workloads (Smith 1997). An implementation may use a standalone unit, vector approximation instructions, or a library sequence, trading accuracy, latency, and throughput. In neural networks, activation, normalization, and softmax operations can become important between matrix kernels, particularly when repeated memory passes or transcendental functions limit throughput.
Palmer, John F. 1980. “The Intel 8087 Numeric Data Processor.” Proceedings of the 1980 National Computer Conference (AFIPS), 887–93. https://doi.org/10.1145/1500518.1500674.
Smith, Steven W. 1997. “Digital Signal Processing.” In Digital Signal Processing Demystified. Elsevier. https://doi.org/10.1016/b978-187870716-1/50004-4.
Nonlinear functions
To see why dedicated hardware matters, consider a typical layer sequence (Goodfellow et al. 2016). Listing 8 combines linear transformations with nonlinear activations—operations that appear simple in Python but reveal substantial computational complexity at the hardware level.
Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press.
layer = nn.Sequential(
nn.Linear(256, 512), nn.ReLU(), nn.BatchNorm1d(512)
)
output = layer(input_tensor)This sequence introduces multiple nonlinear transformations that extend beyond simple matrix operations. Listing 9 breaks down these operations into their mathematical components, exposing the computational complexity that hardware must address.
Z = matmul(weights, input) + bias # Linear transformation
H = max(0, Z) # ReLU activation
mean = reduce_mean(H, axis=0) # BatchNorm statistics
var = reduce_mean((H - mean) ** 2) # Variance computation
output = gamma * (H - mean) / sqrt(var + eps) + beta # NormalizationHardware implementation of nonlinear functions
The computational complexity of these operations becomes apparent when examining their implementation on traditional processors. These seemingly simple mathematical operations translate into complex sequences of instructions. Consider batch normalization (Ioffe and Szegedy 2015): computing the normalization requires reductions, variance calculation, and a square root, while operations like softmax introduce exponentials whose cost depends on the processor implementation. A rectified linear unit (ReLU) is mathematically simple, but a naive scalar implementation still performs a comparison and selection for every element; optimized ML kernels usually make that step branchless. Listing 10 therefore uses ReLU and batch normalization to show two different sources of overhead: element-wise passes through memory and multi-pass normalization work.
Ioffe, Sergey, and Christian Szegedy. 2015. “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.” Proceedings of the 32nd International Conference on Machine Learning (ICML) 37: 448–56.
The label nonlinear hides three different execution contracts. Pointwise functions such as ReLU leave elements independent and therefore map naturally to vector lanes or parallel threads. Reductions such as the mean and variance in batch normalization introduce cross-element dependencies: partial results must be combined before later stages can proceed. Transcendental functions such as exponential, logarithm, and reciprocal square root add a numerical contract because their implementations trade latency, precision, and valid input range. The dependency pattern, not the mathematical label alone, determines whether vector arithmetic, a reduction tree, or a specialized approximation path is appropriate. A faster special-function path accelerates only the operation it implements; it does not remove surrounding tensor traffic or synchronization.
Listing 10 is intentionally dependency-explicit rather than a model of an optimized kernel. The loops expose three state lifetimes: per-element activations, per-feature accumulators, and the final normalized outputs. The normalization parameters cannot be applied until the required statistics are available, so implementations must either retain intermediate activations or recompute them. Training derives those statistics from the current batch, whereas inference can use stored statistics and reduce normalization to an element-wise affine transform. Compilers may fuse legal stages, but the data dependencies determine which transformations preserve the computation. Reading the listing as a dependency graph therefore reveals the durable hardware questions: how often each tensor crosses a memory interface, which intermediates remain local, and where reductions impose synchronization.
for batch in range(32):
for feature in range(512):
# ReLU: Naive scalar compare/select; optimized kernels
# usually implement this branchlessly.
z = matmul_output[batch, feature]
h = max(0.0, z) # Conditional operation
# BatchNorm: Multiple passes over data
mean_sum[feature] += h # First pass for mean
var_sum[feature] += h * h # Additional pass for variance
temp[batch, feature] = h # Extra memory storage needed
# Normalization requires complex arithmetic
for feature in range(512):
mean = mean_sum[feature] / batch_size
var = (var_sum[feature] / batch_size) - mean * mean
# Square root computation: Multiple iterations
scale = gamma[feature] / sqrt(var + eps) # Iterative
# approximation
shift = beta[feature] - mean * scale
# Additional pass over data for final computation
for batch in range(32):
output[batch, feature] = temp[batch, feature] * scale + shiftEach non-linear operation introduces distinct hardware challenges across deep network layers. Batch normalization requires multiple passes through data: one for mean computation, another for variance, and a final pass for output transformation. Each pass loads and stores data through the memory hierarchy. Operations that appear simple in mathematical notation often expand into many instructions, especially for square roots and exponentials on processors without specialized hardware paths. ReLU generally maps to a compare-and-select or maximum operation, so its standalone cost is dominated less by arithmetic than by the additional read and write if it is not fused with neighboring work. The implementation needs temporary storage for intermediate values, increasing memory usage and bandwidth consumption. While vector units excel at regular computations, functions like exponentials and square roots often require specialized implementations that may not fully use vector processing capabilities.
SFU hardware implementation
SFUs address these inefficiencies through dedicated hardware implementation. Modern ML accelerators include specialized circuits that transform these complex operations into low-latency, fixed-function computations. Listing 11 illustrates the mapping: after one vector load, architecture-dependent ReLU, sigmoid, tanh, and reciprocal-square-root paths consume the same input vector.
vld.v v1, (input_ptr) # Load vector of values (pseudocode)
vrelu.v v2, v1 # Vector ReLU path
vsigm.v v3, v1 # Vector sigmoid path
vtanh.v v4, v1 # Vector tanh path
vrsqrt.v v5, v1 # Vector reciprocal-square-root pathSpecialized function paths implement nonlinear and reduction primitives through architecture-specific circuitry. A ReLU can use compare-and-select logic; square root, exponential, and logarithmic functions may use iterative approximations, lookup tables, or interpolation. Table 4 summarizes representative mechanisms and their qualitative latency behavior.
| Function unit | Operation | Implementation strategy | Illustrative latency |
|---|---|---|---|
| Activation unit | ReLU | Compare-and-select or maximum | Low; architecture-dependent |
| Statistics unit | Mean, variance | Parallel reduction trees | Grows with reduction depth |
| Exponential unit | exp, log, sigmoid, tanh | Approximation, table lookup, and interpolation | Architecture-dependent |
| Root/power unit | sqrt, rsqrt | Iterative or approximation hardware | Architecture-dependent |
Vector operations, matrix operations, and special-function paths cover three important computational patterns, but primitives alone do not determine throughput. The primitives tell us what an accelerator can execute efficiently; the execution model and memory system determine how the work maps onto parallel hardware. The same matrix multiplication can therefore achieve very different fractions of peak performance as tensor shape, tile alignment, thread organization, occupancy, memory access, and synchronization change. Peak ratings describe a supported operation under favorable conditions; execution models explain how much of that capacity a real workload can sustain.
Self-Check: Question
When executing a Transformer layer containing attention projection (\(Q = X W_q\)), Softmax (\(\text{Softmax}(S)\)), Layer Normalization (\(\text{LayerNorm}(X)\)), and GeLU activation (\(\text{GeLU}(Z)\)), which execution unit is specifically responsible for computing transcendental functions (exponential and error function approximations)?
- Systolic 2D Matrix Multiply Units
- Dense Tensor Cores
- Vector Load-Store Memory Controllers
- Special Function Units (SFUs)
Explain how the \(\text{im2col}\) (image-to-column) transformation allows standard 2D convolution operations to execute on high-throughput matrix multiplication hardware (GEMM engines), and describe the primary memory overhead associated with this approach.
True or False: In modern AI accelerators, element-wise vector operations such as residual additions (\(Y = X_1 + X_2\)) achieve higher arithmetic intensity than large matrix-matrix multiplications (\(C = A \cdot B\)).
The hardware transformation technique that enables convolutional layers to execute as matrix multiplications without physically duplicating overlapping patch data in memory is known as ____ GEMM.
Which of the following operations in a modern deep learning architecture exhibits the highest operational arithmetic reuse, making it most suitable for dense 2D systolic arrays and Tensor Cores?
- Batched linear layer matrix multiplication (\(Y = X W\))
- Element-wise ReLU activation (\(\max(0, x)\))
- Channel-wise Batch Normalization mean computation
- Token-wise embedding table lookup
Compute Units and Execution Models
Applying ReLU to a 512-element vector shows why execution models matter: the operation is simple, but throughput depends on whether the hardware treats those 512 comparisons as scalar instructions, SIMD lanes, GPU threads, or tensor-program fragments. Modern AI processors package the three compute primitives into distinct execution units: single instruction, multiple data (SIMD) units, Tensor Cores, and processing elements that define how computations are structured and exposed to programmers. Understanding this organization reveals both the theoretical capabilities and practical performance characteristics that determine real-world throughput.
Mapping primitives to execution units
The progression from computational primitives to execution units follows a structured hierarchy that reflects the increasing complexity and specialization of AI accelerators:
- Vector operations → SIMD and single instruction, multiple threads (SIMT) units that enable parallel processing of independent data elements
- Matrix operations → Tensor Cores18 and systolic arrays that provide structured matrix multiplication
- Special functions → Dedicated hardware units integrated within processing elements
18 Reduced-precision ML: Halving operand width halves the bytes per stored element and can support denser arithmetic, although realized throughput depends on the available datapath and kernel (Dally et al. 2021; Dally 2023). NVIDIA’s transition from Pascal P100 vector arithmetic to mixed-precision Tensor Cores in Volta V100 increased advertised peak FP16 throughput from about 21.2 to 125 TFLOP/s, roughly 6\(\times\) across those products (NVIDIA 2017). Precision selection must preserve numerical behavior and use an efficient supported path; the smallest format is not automatically the best one.
Dally, William J., Stephen W. Keckler, and David B. Kirk. 2021. “Evolution of the Graphics Processing Unit (GPU).” IEEE Micro 41 (6): 42–51. https://doi.org/10.1109/mm.2021.3113475.
Dally, Bill. 2023. “Hardware for Deep Learning.” 2023 IEEE Hot Chips 35 Symposium (HCS), 1–58. https://doi.org/10.1109/hcs59251.2023.10254716.
Each execution unit combines these computational primitives with specialized memory and control mechanisms, optimizing both performance and energy efficiency. This structured packaging allows hardware vendors to expose standardized programming interfaces while implementing diverse underlying architectures tailored to specific workload requirements. The choice of execution unit significantly influences overall system efficiency by determining data locality, compute density, synchronization overhead, and how much of the theoretical peak the workload can actually use.
Evolution from SIMD to SIMT architectures
Imagine applying ReLU to a 512-element vector. A scalar formulation contains 512 comparison-and-select operations. A SIMD instruction might cover 8 or 16 elements at a time, while a SIMT GPU can express one lightweight thread per element and schedule those threads in warps or waves. Actual instruction counts and throughput depend on the processor, compiler, memory path, and surrounding fusion. The progression illustrates two related ideas: Flynn’s SIMD taxonomy formalized data-parallel execution (Flynn 1966), and GPU SIMT architectures expose many logical threads that execute in hardware groups (Lindholm et al. 2008; Nickolls et al. 2008).
Flynn, M. J. 1966. “Very High-Speed Computing Systems.” Proceedings of the IEEE 54 (12): 1901–9. https://doi.org/10.1109/proc.1966.5273.
SIMD execution applies identical operations to multiple data elements in parallel, minimizing instruction overhead while maximizing data throughput. This execution model is widely used to accelerate workloads with regular, independent data parallelism, such as neural network computations. The Arm Scalable Vector Extension (SVE) provides a representative example of how modern architectures implement scalable SIMD operations efficiently (Stephens et al. 2017). Listing 12 demonstrates this approach.
ptrue p0.s # Create predicate for vector length
ld1w z0.s, p0/z, [x0] # Load vector of inputs
fmul z1.s, z0.s, z0.s # Multiply elements
fadd z2.s, z1.s, z0.s # Add elements
st1w z2.s, p0, [x1] # Store resultsThe ptrue instruction activates all available predicate lanes without encoding a fixed vector length. This vector-length-agnostic programming model allows the same SVE binary to run on implementations from 128 to 2048 bits without recompilation (Stephens et al. 2017). Intel’s Advanced Matrix Extensions (AMX) are a different kind of specialization: tile registers and matrix instructions expose two-dimensional matrix operations directly to software rather than merely widening a vector lane (Intel Corporation 2021a). Together, SVE and AMX show two hardware-facing paths for ML kernels: vector-length-portable SIMD and fixed tile-based matrix acceleration.
Stephens, Nigel, Stuart Biles, Matthias Boettcher, Jacob Eapen, Mbou Eyole, Giacomo Gabrielli, Matt Horsnell, et al. 2017. “The ARM Scalable Vector Extension.” IEEE Micro 37 (2): 26–39. https://doi.org/10.1109/mm.2017.35.
Lindholm, Erik, John Nickolls, Stuart Oberman, and John Montrym. 2008. “NVIDIA Tesla: A Unified Graphics and Computing Architecture.” IEEE Micro 28 (2): 39–55. https://doi.org/10.1109/mm.2008.31.
Nickolls, John, Ian Buck, Michael Garland, and Kevin Skadron. 2008. “Scalable Parallel Programming with CUDA: Is CUDA the Parallel Programming Model That Application Developers Have Been Waiting For?” Queue 6 (2): 40–53. https://doi.org/10.1145/1365490.1365500.
19 SM (streaming multiprocessor): The physical hardware engine that implements the SIMT model by using warp schedulers to coordinate many parallel threads. Maintaining enough active warps can help hide instruction and memory latency, but occupancy alone does not identify the bottleneck. Low occupancy may result from register use, shared-memory use, block dimensions, or insufficient parallel work, while a memory-bound kernel can still have high occupancy.
20 Warp: On NVIDIA GPUs, a warp is a hardware scheduling group of 32 threads that execute a common instruction when their control paths agree. If threads take different paths, the hardware executes the required paths under different masks, reducing useful lane utilization. The penalty depends on path length and how many lanes take each path, which is why ML kernels often prefer uniform or predicated control flow.
To address these limitations, SIMT extends SIMD principles by enabling parallel execution across multiple independent threads, each maintaining its own program counter and architectural state (Lindholm et al. 2008; Nickolls et al. 2008). This model maps naturally to matrix computations, where each thread processes different portions of a workload while still benefiting from shared instruction execution. In NVIDIA’s GPU architectures, each Streaming Multiprocessor (SM)19 coordinates thousands of threads executing in parallel, allowing for efficient scaling of neural network computations. Threads are organized into warps,20 which are the basic execution units that enable SIMT efficiency. Listing 13 shows this parallel processing model in action.
__global__ void matrix_multiply(float* C, float* A, float*
B, int N) { // CUDA kernel
// Each thread processes one output element
int row = blockIdx.y * blockDim.y + threadIdx.y;
int col = blockIdx.x * blockDim.x + threadIdx.x;
float sum = 0.0f;
for (int k = 0; k < N; k++) {
// Threads in a warp execute in parallel
sum += A[row * N + k] * B[k * N + col];
}
C[row * N + col] = sum;
}The CUDA21 kernel assigns one output element to each thread, exposing independent work that the GPU can schedule across many warps. Similar execution models appear in AMD’s RDNA and Intel’s Xe architectures, reinforcing SIMT as a core mechanism for AI acceleration.
21 CUDA (compute unified device architecture): Released by NVIDIA in 2006, CUDA eliminated the need to disguise general-purpose computations as graphics operations, opening GPUs to scientific and ML workloads through a C-like programming model. The ecosystem it created—cuBLAS, cuDNN, TensorRT—constitutes a software moat that can lock the ML training stack to NVIDIA hardware: migrating away requires rewriting or replacing thousands of GPU-optimized kernels, a cost that often exceeds the hardware savings of competing platforms. This software lock-in, not raw silicon performance alone, helps explain why many large ML training stacks remain CUDA-centered.
Tensor Cores
Consider a single transformer attention head computing the \(\mathbf{Q} \times \mathbf{K}^T\) product for a 2,048-token sequence with 64-dimensional embeddings. This operation requires multiplying a \(2048{\times}64\) matrix by a \(64{\times}2048\) matrix: roughly 268.4M MACs, or about 536.9 MFLOP when the multiply and add are counted separately. On a scalar processor executing one FLOP per cycle at 2 GHz, this single attention head would take about 268.4 ms. GPU SIMT execution can distribute this work across many threads, but Tensor Cores go further by processing entire matrix tiles per instruction; under the illustrative assumption used here, the tiled path completes the same operation in 0.5 milliseconds, a roughly 536.9× improvement over scalar execution. This dramatic speedup arises not from faster clock speeds but from a fundamentally different approach to organizing computation around matrix blocks rather than individual elements.
While SIMD and SIMT units execute vector work efficiently, large matrix computations benefit from units organized around multidimensional tiles. Dedicated matrix engines amortize operand loads across many multiply-accumulate operations by reusing tiles on chip. NVIDIA GPUs expose Tensor Cores, whereas Google TPUs use matrix units built around systolic arrays. Both execute tiled matrix multiplication and accumulation.22 The benefit depends on tile reuse, shape, precision, and memory behavior.
22 Tensor Core dimension alignment: NVIDIA Tensor Cores are most efficient when matrix dimensions are aligned to precision- and architecture-specific multiples, such as multiples of 8 or 16 for common FP16/BF16/INT8 paths. Modern cuBLAS and cuDNN can still use Tensor Cores for many nonaligned dimensions, but poorly aligned shapes may trigger less efficient kernels, require padding, or reduce effective throughput (NVIDIA 2024a; NVIDIA Corporation 2021). This is why model architects often choose embedding and channel dimensions such as 512 rather than 500 and why batch-size-1 inference may fail to reach peak Tensor Core utilization: alignment and arithmetic intensity jointly determine whether the hardware’s matrix engines stay full.
NVIDIA Corporation. 2021. NVIDIA cuDNN Developer Guide.
23 Tensor Core: A single Tensor Core instruction executes a complete matrix-multiply-accumulate operation on a small tile of data using a dedicated hardware block (NVIDIA 2017; NVIDIA Corporation 2020a). This approach bypasses the overhead of fetching and scheduling dozens of individual arithmetic instructions on general-purpose CUDA cores. Because these blocks constitute a large fraction of a modern accelerator’s advertised tensor throughput, failing to use them can leave most of the chip’s theoretical peak unavailable to the workload.
NVIDIA. 2017. Training with Mixed Precision.
Tensor Cores23 provide an example of this approach. Listing 14 exposes matrix computation capabilities through specialized instructions that use dedicated hardware blocks.
Tensor Core Operation (example GPU PTX):
mma.sync.aligned.m16n8k16.row.col.f32.f16.f16.f32
{d0,d1,d2,d3}, // Destination registers
{a0,a1,a2,a3}, // Source matrix A
{b0,b1}, // Source matrix B
{c0,c1,c2,c3}; // AccumulatorA single Tensor Core instruction processes an entire matrix block while maintaining intermediate results in local registers, improving computational efficiency compared to implementations based on scalar or vector operations. This structured approach enables hardware to achieve high throughput while reducing the burden of explicit loop unrolling and data management at the software level.
Design priorities determine how matrix engines appear in different processor families. GPU Tensor Cores preserve programmability while accelerating general-purpose deep learning kernels. TPU-style designs use large-scale matrix units arranged in systolic arrays to maximize sustained training throughput on dense tensor kernels. Mobile NPUs24 shrink the same idea into low-power inference blocks, while server CPUs add matrix instruction extensions (AMX-class tiles) for inference and mixed workloads. Each version changes the same contract: how much flexibility the hardware keeps while reducing movement around dense matrix operations.
24 NPU (neural processing unit): Mobile NPUs achieve low-power inference by implementing common tensor operations in fixed-function or narrowly programmable hardware rather than as fully general GPU kernels. This architectural commitment can deliver large energy-efficiency gains for supported kernels, but it makes deployment dependent on operator coverage: unsupported functions must fall back to a CPU or GPU path that may be far less efficient for that workload (Sze et al. 2017).
Sze, Vivienne, Yu-Hsin Chen, Tien-Ju Yang, and Joel S. Emer. 2017. “Efficient Processing of Deep Neural Networks: A Tutorial and Survey.” Proceedings of the IEEE 105 (12): 2295–329. https://doi.org/10.1109/jproc.2017.2761740.
Figure 6 shows how specialization, lower precision, and sparsity support changed advertised peak capability. From K20X FP32 to H100 FP8 with structured sparsity, the displayed values rise by roughly three orders of magnitude (NVIDIA Corporation 2017, 2020a, 2024; Choquette 2023). The points do not measure one consistent quantity: they mix precision formats and combine FLOP/s with INT8 operations per second. The curve therefore compares each generation’s most prominent advertised path, not same-precision application performance. Master Accelerator Specification Matrix provides the corresponding precision-specific specifications, bandwidth, and power limits.
\begin{tikzpicture}[font=\sffamily\small]
\pgfplotsset{
layers/axis lines on top/.define layer set={
axis background,
axis grid,
axis ticks,
axis tick labels,
pre main,
main,
axis lines,
axis descriptions,
axis foreground,
}{/pgfplots/layers/standard},
}
\begin{axis}[
/pgf/number format/.cd,
tick label style={/pgf/number format/assume math mode=true},
ticklabel style={font=\footnotesize\sffamily},
1000 sep={}, % uklanja zareze
yticklabel style={
/pgf/number format/.cd,
sci,
sci generic={mantissa e exponent},
precision=1
},
width=15cm,
height=85mm,
xmin=2010, xmax=2026,
ymin=1e0, ymax=1e4,
ymode=log,
log basis y=10,
xtick={2012,2014,2016,2017,2020,2022,2024},
%ytick={1e9,1e10,1e11,1e12,1e13,1e14,1e15},
yticklabels={1,10,100,{1,000},{10,000}},
xlabel={Year},
ylabel={Peak throughput (as labeled)},
title style={align=center},
title={NVIDIA GPU Inference Performance\\
{\color{black!60}\footnotesize\sffamily\itshape TFLOP/s for floating point; TOPS for INT8, logarithmic scale}},
axis lines*=left,
axis line style={gray!70,line width=1pt},
% axis on top=true,
set layers=axis lines on top,
tick style={black!90},
tick label style={font=\sffamily\small},
xlabel style={font=\sffamily\large},
ylabel style={font=\sffamily\large},
ylabel style={font=\small\sffamily,align=center,yshift=-1.2mm},
xlabel style={font=\small\sffamily},
grid=major,
major grid style={black!22},
%minor grid style={gray!6},
clip=false,
tick align=outside,
major tick length=1.5mm,
]
%—shaded public text stock band ---
\addplot[
draw=none,
fill=magenta!03,opacity=0.7,
] coordinates {
(2010,1e0)
(2010,1e4)
(2016,1e4)
(2016,1e0)
}; %\closedcycle
\addplot[
draw=none,
fill=magenta!08,opacity=0.7,
] coordinates {
(2016,1e0)
(2016,1e4)
(2026,1e4)
(2026,1e0)
}; %\closedcycle
%—data points ---
\addplot[
%only marks,
mark=*,
mark options={draw=white,fill=crimson, line width=0.95pt, mark size=3.25,
},
crimson,line width=1.75pt
] coordinates {
(2012,5e0) % K20X
(2014,7e0) % M40
(2016,0.44e2) % P40
(2017,1.3e2) % V100
(2020,1.1e3) % A100
(2022,0.4e4) % H100
(2024,0.9e4) % B200 (FP8 sparse ~9,000 TFLOP/s; mlsysim Hardware.Cloud.B200)
};
%—point labels ---
\node[anchor=north, align=center,text=black, font=\sffamily\fontsize{7pt}{8}\selectfont]
at (axis cs:2012,4.4e0) {\textbf{K20X} \\ {\color{black!60}3.9 TFLOP/s FP32}};
\node[anchor=south, align=center,text=black, font=\sffamily\fontsize{7pt}{8}\selectfont]
at (axis cs:2014,9.5e0) {\textbf{M40} \\ {\color{black!60}6.8 TFLOP/s FP32}};
\node[anchor=east, align=center,text=black, font=\sffamily\fontsize{7pt}{8}\selectfont]
at (axis cs:2015.75,0.62e2) {\textbf{P40} \\ {\color{black!60}47 TOPS INT8}};
\node[anchor=west, align=center,text=black, font=\sffamily\fontsize{7pt}{8}\selectfont]
at (axis cs:2017.25,1.05e2) {\textbf{V100} \\ {\color{black!60}125 TFLOP/s FP16}};
\node[anchor=north, align=center,text=black, font=\sffamily\fontsize{7pt}{8}\selectfont]
at (axis cs:2020.4,1.0e3) {\textbf{A100} \\ {\color{black!60}1,248 TOPS INT8}};
\node[anchor=south east, align=center,text=black, font=\sffamily\fontsize{7pt}{8}\selectfont]
at (axis cs:2021.8,0.45e4) {\textbf{H100} \\ {\color{black!60}4,000 TFLOP/s FP8}};
\node[anchor=west, align=center,text=black, font=\sffamily\fontsize{7pt}{8}\selectfont]
at (axis cs:2024.3,0.9e4) {\textbf{B200} \\ {\color{black!60}9,000 TFLOP/s FP8}};
%
\coordinate(A)at(axis cs:2022.5,0.4e4);
\coordinate(B)at(axis cs:2012,5e0);
\coordinate(C)at(axis cs:2016,1e0);
\coordinate(D)at(axis cs:2016,1e4);
\end{axis}
%\scoped[on background layer]
\path[](A)|-coordinate(SR)(B);
\draw[myred,dashed,thick](C)--(D);
\draw[myred,dashed,thick](A)--node[right,font=\sffamily\fontsize{9pt}{9}\selectfont]{1,000$\times$}(SR);
\end{tikzpicture}Processing elements
The highest level of execution unit organization integrates multiple Tensor Cores with local memory into processing elements (PEs). A processing element serves as the primary building block in many AI accelerators, combining different computational units to efficiently execute neural network operations. Each PE typically includes vector units for element-wise operations, Tensor Cores for matrix computation, special function units for nonlinear transformations, and dedicated memory resources to optimize data locality and minimize data movement overhead.
Processing element design varies because each architecture chooses a different balance between compute density, local memory, and interconnect distance. Graphcore’s Intelligence Processing Unit (IPU) distributes computation across 1,472 tiles, each containing independent processing elements optimized for fine-grained parallelism (Graphcore 2020). Cerebras extends the same local-compute principle in the CS-2 system, integrating roughly 850,000 AI-optimized cores across a wafer-scale device for deep learning acceleration (Systems 2021). Tesla’s D1 processor emphasizes substantial local memory inside its processing elements, optimizing throughput and latency for real-time autonomous vehicle workloads (Tesla, Inc. 2021).
Graphcore. 2020. The Colossus MK2 IPU Processor.
Tesla, Inc. 2021. Tesla Dojo Technology: A Guide to Tesla’s Configurable Floating Point Formats & Arithmetic. Tesla AI Day, Whitepaper.
Across these designs, the binding trade-off is the one the roster illustrates: compute density versus data locality. Packing more cores raises peak throughput only if each one can be kept fed, so a processing element’s delivered efficiency depends as much on interconnect strategy and memory locality as on raw arithmetic capability.
That same dependence on locality governs which algorithmic optimizations the hardware can actually exploit. A regular grid of processing elements accelerates sparsity only when the surviving nonzero values preserve the predictable access patterns the grid depends on, which is precisely the constraint that N:M structured sparsity is designed to satisfy.
N:M structured sparsity mechanics
While unstructured pruning reduces model size, it rarely translates to hardware speedup because memory access becomes irregular. Hardware accelerators solve this with N:M structured sparsity,25 a pattern-based approach that enforces regularity. The notation “\(N{:}M\)” specifies that exactly \(N\) values must be nonzero within every contiguous block of \(M\) values, creating a predictable pattern that hardware can exploit.
25 N:M structured sparsity: The 2:4 ratio (50 percent density) used by NVIDIA’s Ampere Sparse Tensor Cores is a hardware-friendly compromise: every contiguous four-value group retains two nonzero values, preserving regular indexing while halving the dense value payload (NVIDIA Corporation 2020a). At 2:4, the metadata overhead is compact enough to store alongside the weights without overwhelming the memory-traffic savings, which is the constraint that makes the advertised 2\(\times\) tensor-math throughput path plausible when kernels and model weights satisfy the pattern.
NVIDIA Corporation. 2020a. NVIDIA A100 Tensor Core GPU Architecture. NVIDIA Whitepaper, V1.0.
NVIDIA. 2020a. Accelerating Matrix Multiplication with Block Sparse Format and NVIDIA Tensor Cores.
NVIDIA’s Sparse Tensor Cores implement a concrete instance of this pattern: the 2:4 constraint, which requires that exactly two of every contiguous block of four values be nonzero (equivalently, two must be zero) (NVIDIA Corporation 2020a; NVIDIA 2020a). This constraint allows the hardware to compress the matrix by 50 percent in memory plus metadata. The execution proceeds in three stages: first, the hardware stores only the two nonzero values and compact metadata for every four-element block (compression); second, during matrix multiplication, the Sparse Tensor Core reads the metadata to select the corresponding activations and performs math only on the nonzero weights (compute); third, this increases the effective FLOP/byte ratio, providing an up-to-2\(\times\) tensor-math throughput path over dense matrix multiplication when the model is fine-tuned to respect the 2:4 constraint.
To understand why “Structured” patterns are required for hardware speedup, consider how sparse matrices are actually stored in memory. CSR and block sparse storage are treated formally in Sparse matrix formats. Figure 7 compares their occupancy patterns and makes the block-index overhead visible. If the sparsity is random, the index overhead and irregular access kill performance. Structured sparsity, whether at the large block scale or the fine-grained N:M scale, makes this indexing predictable and compact, allowing hardware to fetch data efficiently.
\begin{tikzpicture}[ x=5mm,y=5mm,line join=round,font=\sffamily\small]
\tikzset{%
cell/.style={draw=white, line width=0.6pt},
gridline/.style={draw=black!70, line width=1.5pt},
Arr/.style={->,>=Latex,line width=0.75pt},
}
% dimensions
\def\Rows{3}
\def\Cols{9}
\definecolor{blue1}{RGB}{23,68,150}
\definecolor{blue2}{RGB}{84,131,217}
\definecolor{blue3}{RGB}{145,177,237}
%—2) Macro: paint cell (r,c) with color ---
\newcommand{\ColorCell}[3]{% #1=row, #2=col, #3=color
\pgfmathtruncatemacro{\yy}{\Rows-#1}
\fill[#3] (#2-1,\yy) rectangle ++(1,1);
\draw[cell] (#2-1,\yy) rectangle ++(1,1);
}
%Grid 9x3
\newcommand{\Gridd}{%
\foreach \r in {1,...,\Rows}{
\foreach \c in {1,...,\Cols}{
\pgfmathtruncatemacro{\yy}{\Rows-\r}
% draw cell
\fill[gray!25] (\c-1,\yy) rectangle ++(1,1);
\draw[cell] (\c-1,\yy) rectangle ++(1,1);
% corners
\coordinate (\br-cell-\r-\c-sw) at (\c-1,\yy);
\coordinate (\br-cell-\r-\c-se) at (\c,\yy);
\coordinate (\br-cell-\r-\c-nw) at (\c-1,\yy+1);
\coordinate (\br-cell-\r-\c-ne) at (\c,\yy+1);
\coordinate (\br-cell-\r-\c-n) at (\c-0.5,\yy+1);
% center
\coordinate (cell-\r-\c) at (\c-0.5,\yy+0.5);
}
}
}
%%%%%%%%%%%%%
%Dense Matrix
%%%%%%%%%%%%%
\begin{scope}[local bounding box=BLOCK-A,shift={(0.0,0.0)}]
%—1) Base + all cell coordinates ---
\def\br{A}
\Gridd
\ColorCell{1}{1}{blue2}
\ColorCell{1}{2}{blue3}
\ColorCell{1}{3}{blue2}
\ColorCell{2}{1}{blue2}
\ColorCell{2}{2}{blue1}
\ColorCell{2}{3}{blue2}
\ColorCell{3}{1}{blue2}
\ColorCell{3}{2}{blue2}
\ColorCell{3}{3}{blue2}
%Center
\ColorCell{1}{4}{blue1}
\ColorCell{1}{5}{blue2}
\ColorCell{1}{6}{blue2}
\ColorCell{2}{4}{blue3}
\ColorCell{2}{5}{blue2}
\ColorCell{2}{6}{blue1}
\ColorCell{3}{4}{blue2}
\ColorCell{3}{5}{blue1}
\ColorCell{3}{6}{blue2}
%—4) Thick frame around the entire 9x3 grid ---
\draw[gridline] (0,0) rectangle (\Cols,\Rows);
% (optional) thick vertical division after 3 and after 6 columns (as in the picture)
\draw[gridline] (3,0) -- (3,\Rows);
\draw[gridline] (6,0) -- (6,\Rows);
\end{scope}
%%%%%%%%%%%%%
%Sparse Matrix (CSR)
%%%%%%%%%%%%%
\begin{scope}[local bounding box=BLOCK-B,shift={(0.0,-6.7)}]
%—1) Base + all cell coordinates ---
\def\br{B}
\Gridd
\ColorCell{1}{7}{blue1}
\ColorCell{1}{8}{blue2}
\ColorCell{1}{9}{blue2}
\ColorCell{2}{7}{blue3}
\ColorCell{2}{8}{blue2}
\ColorCell{2}{9}{blue1}
\ColorCell{3}{7}{blue2}
\ColorCell{3}{8}{blue1}
\ColorCell{3}{9}{blue2}
%—4) Thick frame around the entire 9x3 grid ---
\draw[gridline] (0,0) rectangle (\Cols,\Rows);
% (optional) thick vertical division after 3 and after 6 columns (as in the picture)
\draw[gridline] (3,0) -- (3,\Rows);
\draw[gridline] (6,0) -- (6,\Rows);
\end{scope}
%%%%%%%%%%%%%
%Block Sparse Matrix
%%%%%%%%%%%%%
\begin{scope}[local bounding box=BLOCK-C,shift={(12,0,0)}]
%—1) Base + all cell coordinates ---
\def\br{C}
\Gridd
%LEFT
\ColorCell{1}{1}{blue2}
\ColorCell{1}{2}{blue3}
\ColorCell{1}{3}{blue2}
\ColorCell{2}{1}{blue2}
\ColorCell{2}{2}{blue1}
\ColorCell{2}{3}{blue2}
\ColorCell{3}{1}{blue2}
\ColorCell{3}{2}{blue2}
\ColorCell{3}{3}{blue2}
%RIGHT
\ColorCell{1}{7}{blue1}
\ColorCell{1}{8}{blue2}
\ColorCell{1}{9}{blue2}
\ColorCell{2}{7}{blue3}
\ColorCell{2}{8}{blue2}
\ColorCell{2}{9}{blue1}
\ColorCell{3}{7}{blue2}
\ColorCell{3}{8}{blue1}
\ColorCell{3}{9}{blue2}
%
\draw[gridline] (0,0) rectangle (\Cols,\Rows);
\draw[gridline] (3,0) -- (3,\Rows);
\draw[gridline] (6,0) -- (6,\Rows);
\end{scope}
%%%%%%%%%%%%%
%Block Sparse (BSR)
%%%%%%%%%%%%%
\begin{scope}[local bounding box=BLOCK-D,shift={(12,-6.7)}]
%—1) Base + all cell coordinates ---
\def\br{D}
\Gridd
\ColorCell{1}{4}{blue1}
\ColorCell{1}{5}{blue2}
\ColorCell{1}{6}{blue2}
\ColorCell{2}{4}{blue3}
\ColorCell{2}{5}{blue2}
\ColorCell{2}{6}{blue1}
\ColorCell{3}{4}{blue2}
\ColorCell{3}{5}{blue1}
\ColorCell{3}{6}{blue2}
\draw[gridline] (0,0) rectangle (\Cols,\Rows);
\draw[gridline] (3,0) -- (3,\Rows);
\draw[gridline] (6,0) -- (6,\Rows);
\end{scope}
% =========================
% MINI GRID: 1-by-6
% =========================
\begin{scope}[local bounding box=MINI-G,shift={(27,2.1)},node distance=-0.95pt,
mCell/.style={draw=black!60,rectangle,minimum width=7.5mm,
minimum height=7.5mm,line width=0.75pt, fill=orange!15}]
\node[mCell](R1){1};
\node[mCell,below =of R1](R2){2};
\node[mCell,below =of R2](R3){4};
\node[mCell,below =of R3](R4){5};
\node[mCell,below =of R4](R5){7};
\node[mCell,below =of R5](R6){9};
\draw[black!70, line width=1pt] (R1.north west) rectangle (R6.south east);
\end{scope}
%%%%%%
\node[below=1pt of BLOCK-A]{Dense Matrix};
\node[below=1pt of BLOCK-B]{Sparse Matrix (CSR)};
\node[below=1pt of BLOCK-C]{Block Sparse Matrix};
\node[below=1pt of BLOCK-D](BSS){Block Sparse (BSR)};
\path[red](BSS)-|coordinate(SR1)(MINI-G);
\node[align=center]at(SR1){Non-zero Block Indices};
%
%—Above (R1,R2,R3 -> BLOCK-C) ---
\coordinate (busC) at ([xshift=-7mm]R1.west);
%
\coordinate (tapC1) at (busC |- R1.west);
\coordinate (tapC2) at (busC |- R2.west);
\coordinate (tapC3) at (busC |- R3.west);
% Arrows:
\draw[] (R1.west) -- (tapC1) ;
\draw[] (R2.west) -- (tapC2);
\draw[] (R3.west) -- (tapC3);
%
\draw[Arr] (tapC2)--++(-1.2,0)--++(0,3.5)-| (C-cell-1-1-n);
\draw[Arr] (tapC2)--++(-1.2,0)--++(0,3.5)-| (C-cell-1-3-n);
\draw[Arr] (tapC2)--++(-1.2,0)--++(0,3.5)-| (C-cell-1-9-n);
\draw[black!60,line width=4.0pt]
([yshift= 4mm]busC |- R1.west) -- ([yshift=-3mm]busC |- R3.west);
\fill[black!70] ($(tapC2)+(-3pt,0)$) circle (1.75pt);
% ---Below (R4,R5,R6 -> BLOCK-D) ---
\coordinate (busD) at ([xshift=-7mm]R4.west);
\coordinate (tapD4) at (busD |- R4.west);
\coordinate (tapD5) at (busD |- R5.west);
\coordinate (tapD6) at (busD |- R6.west);
\draw[] (R4.west) -- (tapD4);
\draw[] (R5.west) -- (tapD5);
\draw[] (R6.west) -- (tapD6);
%
\draw[Arr] (tapD5)--++(-1.2,0)--++(0,1.2)-| (D-cell-1-4-n);
\draw[Arr] (tapD5)--++(-1.2,0)--++(0,1.2)-| (D-cell-1-7-n);
\draw[Arr] (tapD5)--++(-1.2,0)--++(0,1.2)-| (D-cell-1-9-n);
\draw[black!60,line width=4.0pt]
([yshift= 2mm]busD |- R4.west) -- ([yshift=-3mm]busD |- R6.west);
\fill[black!70] ($(tapD5)+(-3pt,0)$) circle (1.75pt);
\end{tikzpicture}The 2:4 pattern illustrates a broader principle: hardware achieves efficiency not by computing zeros faster, but by never loading them in the first place. This insight connects sparsity to the memory wall, since structured patterns reduce memory traffic, which is where the real cost lies.
Beyond structured sparsity optimizations, different hardware architectures implement matrix operations through distinct computational structures. Systolic arrays represent one such approach that has proven particularly effective for AI workloads.
Systolic arrays
While Tensor Cores package matrix operations into small, specialized instruction tiles, systolic arrays structure processing elements into large two-dimensional grids optimized for continuous data flow and operand reuse. By pulsing inputs horizontally and partial sums vertically through the array, systolic architectures minimize expensive DRAM fetches, enabling low-precision, quantized weights to achieve maximum compute density. The core motivation for systolic architectures stems from the same energy constraint that drives accelerator design: minimizing memory access penalties through physical operand reuse. A simple energy comparison through the array reveals why this architecture has become central to modern AI accelerators.
Napkin Math 1.1: The energy advantage of pulsing data
Scenario: The systolic architecture improves energy efficiency by keeping operands local as work pulses through the array; the “Systolic” (heartbeat) metaphor reflects this benefit of reusing operands locally. This worked model compares systolic dataflow with a naive implementation that streams every operand through DRAM:
- DRAM-streaming baseline: Loads \(\mathbf{A}\), loads \(\mathbf{B}\), computes \(\mathbf{A} \times \mathbf{B} + \mathbf{C}\), and writes \(\mathbf{C}\) without cache or register reuse.
- Data movement: 3 loads + 1 write = 4 DRAM accesses (per operation).
- Energy: ≈ 4 \(\times\) 640 pJ + 1 pJ (compute) = 2561 pJ/op.
- Systolic Array (128 \(\times\) 128 size): Loads A and B once at the edges. Data “pulses” through 128 processing elements.
- Data movement: 2 loads per 128 operations = 0.016 DRAM accesses (per operation).
- Energy: ≈ 0.016 \(\times\) 640 pJ + 1 pJ (compute) ≈ 11 pJ/op.
Systems insight: Under this deliberately naive DRAM-streaming baseline, the worked model gives the systolic array a 232.8× energy advantage. The ratio measures locality, not an inherent gap between vector and systolic arithmetic.
- Concretely, a \(128{\times}128\) array can sustain 16,384 MACs/cycle with a large energy dividend by pulsing data through processing elements instead of repeatedly loading it from DRAM (Horowitz 2014; Jouppi et al. 2023).
- This locality lets dense arrays sustain many simultaneous MAC operations without paying a DRAM access for every operation.
- Limitation: The comparison assumes no effective reuse in the baseline and full reuse across the array. Caches, register blocking, array underutilization, and other dataflows change the ratio substantially.
A systolic array arranges processing elements in a grid pattern, where data flows rhythmically between neighboring units in a synchronized manner, enabling each operand to participate in multiple computations as it propagates through the array. This structured movement minimizes external memory accesses by maximizing local data reuse. A single weight value can contribute to dozens of operations as it moves through the processing elements, transforming the energy profile from memory-bound to compute-efficient execution.
Kung and Leiserson26 (Kung and Leiserson 1979) first introduced systolic arrays, formalizing their use in parallel computing architectures for efficient matrix operations (Kung 1982). Unlike general-purpose execution units, systolic arrays exploit spatial and temporal locality by reusing operands as they propagate through the grid. Google’s TPU exemplifies this architectural approach: in the TPUv4, a \(128{\times}128\) systolic array of multiply-accumulate units processes matrix operations by streaming data through the array in a pipelined manner (Jouppi et al. 2023). Figure 8 follows these data paths: a control unit feeds input buffers that stream data horizontally into the array, while the partial sums each cell produces flow vertically down to the accumulator chain at the bottom, which collects the finished results. Each processing element performs one multiply-accumulate per cycle and passes its operands to its neighbors, so a value loaded once is reused across an entire row or column rather than refetched from memory.
26 Systolic array: From Greek sustole (“contraction”), borrowed from cardiology to evoke rhythmic pumping. Kung and Leiserson used the term for arrays in which data advances through neighboring processing elements on a regular schedule (Kung and Leiserson 1979). Pipeline fill permits concurrent wavefronts and local reuse. The benefit depends on the stationary operand, tile, buffers, and utilization; it does not guarantee fixed DRAM savings. Irregular workloads and poorly aligned shapes remain a weaker fit because fill, drain, and idle elements consume capacity.
Kung, Hsiang Tsung, and Charles E Leiserson. 1979. “Systolic Arrays (for VLSI).” Sparse Matrix Proceedings 1978 1: 256–82.
Kung, H. T. 1982. “Why Systolic Architectures?” Computer 15 (1): 37–46. https://doi.org/10.1109/mc.1982.1653825.
\resizebox{0.75\textwidth}{!}{%
\begin{tikzpicture}[font=\sffamily]
%
\tikzset{%
Line/.style={line width=1.3pt,black!70,rounded corners}
}
\node[line width=0.75pt, draw=VioletLine,fill=VioletL!30, rectangle,
minimum width=200,minimum height=200](B){};
\foreach \x/\y in{0.08/1,0.33/2,0.58/3,0.95/4}
\draw[Line,line cap=round]($(B.south west)!\x!(B.south east)$)coordinate(G\y)
--++(270:0.7)coordinate(D\y);
%
\foreach \a in{1,2,3,4}{
\begin{scope}[shift={(D\a)}, yshift=-33]
\node[line width=1.25pt, draw,fill=GreenL!30,
minimum width=22, minimum height=32](MB\a){};
\foreach \x in{0.2,0.4,0.6,0.8}
\draw[line width=1.25pt]($(MB\a.north west)!\x!(MB\a.south west)$)--
($(MB\a.north east)!\x!(MB\a.south east)$);
\node[circle,line width=1.25pt,draw,minimum width=19,
above=0.22 of MB\a,fill=white](C\a){};
\node[font=\bfseries\sffamily\bfseries]at(C\a){+};
\draw[Line](C\a)--(MB\a);
\draw[Line](MB\a.south)--++(270:0.3)--++(180:0.9)|-(C\a.west)coordinate(T\a);
\end{scope}
}
\draw[Line,-latex](MB1)--(MB2);
\draw[Line,-latex](MB2)--(MB3);
\node[font=\Huge](DL)at($(MB3.east)!0.44!(MB4.west)$){...};
\draw[Line,-latex](MB3)--(DL);
\draw[Line,-latex](DL)--(MB4);
\draw[Line,-latex](MB4)--++(0:1)node[right]{Done};
\foreach \x/\y in{0.08/1,0.31/2,0.55/3,0.95/4}
\draw[Line,line cap=round]($(B.north west)!\x!(B.south west)$)coordinate(GG\y)
--++(180:0.7)coordinate(DD\y);
\foreach \a in{1,2,3,4}{
\begin{scope}[shift={(DD\a)}, xshift=-12,line cap=round]
\node[line width=1.25pt, draw=none,fill=GreenL!80,
minimum width=32, minimum height=20](2MB\a){};
\foreach \x in{0,0.25,0.5,0.75}
\draw[line width=1.25pt]($(2MB\a.north west)!\x!(2MB\a.north east)$)--
($(2MB\a.south west)!\x!(2MB\a.south east)$);
\draw[line width=1.25pt,line cap=round,red](2MB\a.north west)
--++(180:2mm)coordinate(Z);
\draw[line width=1.25pt,line cap=round,red](2MB\a.south west)
--++(180:2mm)coordinate(DZ);
\draw[line width=1.25pt,line cap=round](Z)--(2MB\a.north east)|-(DZ);
\end{scope}
}
\draw[Line,-latex](2MB1)--(2MB2);
\draw[Line,-latex](2MB2)--(2MB3);
\node[font=\Huge,rotate=90](2DL)at($(2MB3.south)!0.52!(2MB4.north)$){...};
\draw[Line,-latex](2MB3)--(2DL);
\draw[Line,-latex](2DL)--(2MB4);
\draw[Line,-latex](2MB4)|-(MB1);
%
\node[line width=1.25pt, draw,fill=BlueL,
% minimum width=22mm, minimum height=10mm,
inner ysep=8,inner xsep=10,
above left=0.25 and 1.2 of 2MB1](CO){Control};
\draw[Line,-latex](CO.350)-|(2MB1);
\draw[Line,-latex](CO.10)-|(B.north west);
%%
\def\di{0.5}
\def\du{1.0}
\draw[Line,-latex](GG1)++(\di,0)--++(0:\du)coordinate(H);
\draw[Line,-latex](H)++(\di,0)--++(0:\du)coordinate(H1);
\draw[Line,-latex](H1)++(\di,0)--++(0:\du)coordinate(H2)
node[right]{Data};
\draw[Line,-latex](GG2)++(\di,0)--++(0:\du)coordinate(2H);
\draw[Line,-latex](2H)++(\di,0)--++(0:\du)coordinate(2H1);
\draw[Line,-latex](GG3)++(\di,0)--++(0:\du)coordinate(3H);
%
\path[](H)-|coordinate(V1)(G4);
\draw[Line,-latex](V1)++(0,-5mm)--++(270:\du)coordinate(V2);
\draw[Line,-latex](V2)++(0,-5mm)--++(270:\du)coordinate(V3);
\draw[Line,-latex](V3)++(0,-5mm)--++(270:\du)coordinate(V4);
%
\path[](2H)-|coordinate(2V1)(G3);
\draw[Line,-latex](2V1)++(0,-0.8*\di)--++(270:0.8*\du)coordinate(2V2);
\draw[Line,-latex](2V2)++(0,-0.8*\di)--++(270:0.8*\du)coordinate(2V3);
\draw[Line,-latex](2V3)++(0,-0.8*\di)--++(270:0.8*\du)node[below]{Partial Sums};
%
\path[](3H)-|coordinate(3V1)(G2);
\draw[Line,-latex](3V1)--++(270:0.8*\du)coordinate(3V2);
\draw[Line,-latex](3V2)++(0,-0.6*\di)--++(270:0.8*\du)coordinate(3V3);
\draw[Line,-latex](3V3)++(0,-0.6*\di)--++(270:0.8*\du)coordinate(3V4);
\end{tikzpicture}}The tiling principle: Bridging graph and silicon
A fundamental mismatch exists between the computational graph (which sees a single 4,096 \(\times\) 4,096 matrix multiplication) and the physical silicon (which possesses a fixed 128 \(\times\) 128 systolic array). Bridging this gap requires tiling: the process of partitioning large tensor operations into “tiles” that fit exactly into the hardware’s fast local memory (SRAM or Scratchpad).
To process our 4,096-wide worked-example layer on a 128-wide systolic array, the compiler decomposes the result into 1,024 output tiles. Each output tile accumulates across 32 reduction-dimension tiles, so the complete matrix multiplication executes 32,768 \(\mathbf{A}\)-tile-by-\(\mathbf{B}\)-tile products. This is not merely a software convenience; it is a physical requirement. Operand tiles are fetched from slow HBM, staged in fast SRAM, and pulsed through the systolic array. Algorithm 1 states the loop nest a compiler emits for this decomposition: stream tiles of \(\mathbf{A}\) and \(\mathbf{B}\) on chip and accumulate their product into a tile of \(\mathbf{C}\) before writing it back.
One extra dimension past the tile width tips utilization off a cliff.
The tile sizes are the lever. A larger tile reuses each loaded byte across more multiply-accumulate operations, raising the kernel’s arithmetic intensity and pushing it toward the compute-bound side of the roofline; the ceiling is how much of \(\mathbf{A}\), \(\mathbf{B}\), and \(\mathbf{C}\) fits in fast on-chip memory at once. This tiling pattern is the central mechanism behind high-performance ML systems. It allows the hardware to maintain high system efficiency \((\eta_{\text{hw}})\) by ensuring that for every byte loaded from main memory, the data is reused 128× within the systolic grid. An engineer who understands tiling understands the “silicon contract”: if a layer’s dimensions are not multiples of the tile size (for example, a width of 129 on a 128 array), the system pays a fringe tax in underutilized silicon, where 127 units sit idle while one unit finishes the “remainder” tile.
\begin{algorithm} \caption{Tiled (blocked) matrix multiply} \begin{algorithmic} \Require $\mathbf{A} \in \mathbb{R}^{M\times K}$, $\mathbf{B} \in \mathbb{R}^{K\times N}$; tile sizes $T_M, T_N, T_K$ \Ensure $\mathbf{C} = \mathbf{A}\mathbf{B}$ \For{each row tile $i_0$ (size $T_M$)} \For{each column tile $j_0$ (size $T_N$)} \State initialize the $\mathbf{C}$-tile accumulator to zero on chip \For{each $k_0$ over $K$ in steps of $T_K$} \State load $\mathbf{A}$-tile $[i_0,k_0]$ and $\mathbf{B}$-tile $[k_0,j_0]$ on chip \State accumulate the tile product on chip \Comment{reuse $T_N$/$T_M$ per byte} \EndFor \State write the $\mathbf{C}$-tile back to memory \EndFor \EndFor \end{algorithmic} \end{algorithm}
The systolic array architecture achieves computational efficiency through synchronized data movement across a structured grid of processing elements. Systolic arrays organize computation around four components:
- Control unit: Coordinates timing and data distribution across the array, maintaining synchronized operation throughout the computational grid.
- Data streams: Input matrices propagate through coordinated pathways where matrix A elements traverse horizontally while matrix B elements flow vertically through the processing grid.
- Processing element grid: Individual processing elements execute multiply-accumulate operations on streaming data, generating partial results that accumulate toward the final computation.
- Output collection: Results aggregate at designated output boundaries where accumulated partial sums form complete matrix elements.
The synchronized data flow ensures that matrix element \(A_{ik}\) encounters corresponding \(B_{kj}\) elements at precise temporal intervals, executing the multiply-accumulate operations required for matrix multiplication \(C_{ij} = \sum_k A_{ik}\times B_{kj}\). This systematic reuse of operands across multiple processing elements substantially reduces memory bandwidth requirements by eliminating redundant data fetches from external memory subsystems.
For a \(2{\times}2\) product, each output \(C_{ij}=A_{i0}B_{0j}+A_{i1}B_{1j}\) requires two products and one accumulation. A weight-stationary mapping holds selected elements of \(\mathbf{B}\) in local PE storage, streams elements of \(\mathbf{A}\) across the array, and routes or retains partial sums until both \(k\) terms arrive. Thus \(C_{00}\) combines \(A_{00}B_{00}\) with \(A_{01}B_{10}\), while \(C_{01}\) combines \(A_{00}B_{01}\) with \(A_{01}B_{11}\). Values are injected with a time skew so operands belonging to the same product meet at the intended PE. The exact cycle path depends on array geometry, operand placement, and whether weights, inputs, or outputs are stationary. The lesson is local reuse: a loaded tile should serve several MACs before replacement, while fill and drain cycles remain part of the cost.
Each processing element in a 2D array performs a multiply-accumulate operation in every cycle. In a representative weight-stationary configuration:
- Holds a stationary weight parameter (\(B_{k,j}\)) within its local register
- Receives a streaming input activation (\(A_{i,k}\)) from its left neighbor
- Computes the partial product \(A_{i,k} \times B_{k,j}\) and adds it to the local accumulator
- Passes the activation value rightward to the adjacent processing element for the next cycle
This structured computation model minimizes data movement between global memory and processing elements, improving both efficiency and scalability. 2D matrix multiplication operations map naturally onto grid-structured systolic processing arrays: follow the synchronized dataflow in figure 8, tracing how input activations stream horizontally while stationary weights accumulate partial products vertically. As systolic arrays operate in a streaming fashion, they are particularly effective for high-throughput workloads such as deep learning training and inference. However, as detailed in section 1.4.1, practical effectiveness is ultimately constrained by memory bandwidth bottlenecks.
A 128 \(\times\) 128 systolic array capable of 16,384 operations per cycle requires a continuous operand stream to maintain utilization. On-chip buffers feed activations and weights to the array edges and are replenished from off-chip memory as needed; reuse within the array avoids fetching every operand from HBM each cycle. The TPU v4’s 1,200 GB/s HBM2 bandwidth helps sustain this stream, but off-chip bandwidth can still limit models whose working sets and reuse patterns exceed on-chip capacity.
The quantization techniques in Quantization and Precision reduce model memory footprint by converting FP32 weights to INT8 representations. This optimization directly addresses the memory bandwidth constraints identified here. Converting 32-bit floating-point weights to 8-bit integers can reduce weight traffic by 4\(\times\); whether that changes a kernel from bandwidth bound to compute bound depends on the operation’s original arithmetic intensity, the accelerator’s INT8 ridge point (the intensity threshold at which its INT8 compute saturates), and the overhead of quantization and dequantization. Similarly, structured pruning removes entire rows or columns of weight matrices, reducing both the data volume that must traverse memory hierarchies and the computation required. These algorithmic optimizations prove valuable precisely because they target the memory bottleneck that limits accelerator performance in practice.
Systems Perspective 1.1: Matching architecture to workload
Challenge: Systolic arrays must choose which data to keep stationary (in registers) to minimize movement. This choice hard-codes the hardware’s preference for certain model types. Table 5 previews the three stationary-operand strategies and the workloads each favors; section 1.7 develops each one in full as a general mapping decision.
Table 5: Systolic-Array Dataflow Strategies: Three stationary-operand choices for systolic arrays, the reuse pattern each maximizes, and the workload class that benefits, showing how a fixed dataflow choice hard-codes an accelerator’s affinity for specific models.
| Strategy | Stationary item | Optimized for | Example Workload |
|---|---|---|---|
| Weight-stationary | Weights (\(\mathbf{W}\)) | High Reuse of Weights | CNNs (Conv2D): Filters are small and reused across the entire image. |
| Output-stationary | Partial Sums (\(\mathbf{C}\)) | High Reuse of Accumulators | Large Batch MatMul: Accumulating results for many inputs against a large weight matrix. |
| Input-stationary | Inputs (\(\mathbf{A}\)) | High Reuse of Activations | Transformers: The same activations feed many weight matrices across attention heads. |
There is no “perfect” accelerator. A chip optimized for Weight-Stationary flow (like early TPUs) excels at CNNs where filters are small and heavily reused, but faces challenges with large language model (LLM) inference at small batch sizes, where the weight matrix is read once per token with minimal reuse, pushing architectures toward output-stationary or hybrid dataflow patterns.
Numerics in AI acceleration
Systolic arrays and Tensor Cores often support reduced-precision arithmetic. FP16 halves the bytes per value relative to FP32, and a target may provision more low-precision multiply-accumulate capacity. A 2\(\times\) speedup is not automatic: it requires a supported kernel and sufficient parallel work without another bottleneck. Building on Model Compression, reduced precision is a hardware-software decision. Input and accumulation formats may differ, affecting accuracy, throughput, energy, and movement.
Precision trade-offs
Lower precision is not free. FP16 has a 5-bit exponent and 10 stored fraction bits, giving it more significand precision than BF16 but much less dynamic range than FP32. Its smallest normal value is about \(6.1\times10^{-5}\) and its largest finite value is 65,504; subnormals extend lower with reduced precision. BF16 retains FP32’s 8-bit exponent and uses 7 fraction bits, preserving similar dynamic range at lower precision. INT8 has no floating-point exponent and depends on scale, zero-point, and clipping choices. Hardware and software must balance these properties against throughput and bandwidth.
The evolution of AI hardware reflects co-design between numerical methods and hardware capability. Earlier GPU generations lacked the dedicated mixed-precision matrix paths now used for deep learning, even when other units supported several formats. As training and inference methods demonstrated acceptable accuracy with selected lower-precision operations, vendors added native FP16, BF16, and integer tensor paths. Frameworks and libraries expose them through compatible kernels, autocasting, quantization, and calibration. Software gains materialize only when the model selects an efficient path; changing storage alone may reduce bytes without producing the advertised arithmetic speedup.
Precision support is integrated into execution units and memory paths. SIMD and SIMT lanes may support several scalar formats, while Tensor Cores (section 1.3.3) and systolic units (section 1.3.6) expose architecture-specific input and accumulation combinations. Lower precision reduces bytes per operand and may increase operations per cycle. Tiling and dataflow, not precision itself, determine reuse, while conversion, scaling, or fallback overhead can offset part of the gain. A format is useful only when the kernel and numerical policy can employ it safely.
Despite the advantages of reduced precision, deep learning models cannot always rely solely on low-bit representations. To address this challenge, modern AI accelerators implement mixed-precision computing, where different numerical formats are used at different stages of execution. These precision choices affect numerical reliability: matrix multiplications may be performed in FP16 or BF16, while accumulations are maintained in FP32 to prevent precision loss. Similarly, inference engines use INT8 arithmetic while preserving key activations in higher precision when necessary.
Mixed-precision computing
Modern AI accelerators increasingly support mixed-precision execution, allowing different numerical formats to be used at various stages of computation. Training workloads often use FP16 or BF16 for matrix multiplications, while maintaining FP32 accumulations to preserve precision (Micikevicius et al. 2017; Mellempudi et al. 2019). The software implementation of mixed-precision training, including loss scaling techniques and framework support, is covered in Mixed-precision training. Inference workloads, by contrast, optimize for INT8 or even INT4, achieving high efficiency while retaining acceptable accuracy.
Micikevicius, Paulius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, et al. 2017. “Mixed Precision Training.” arXiv Preprint arXiv:1710.03740.
Mellempudi, Naveen, Sudarshan Srinivasan, Dipankar Das, and Bharat Kaul. 2019. “Mixed Precision Training with 8-Bit Floating Point.” arXiv Preprint arXiv:1905.12334.
NVIDIA. 2020b. TensorFloat-32 in the A100 GPU Accelerates AI Training, HPC up to 20x.
NVIDIA Corporation. 2018. NVIDIA Tesla T4 Tensor Core GPU. NVIDIA product documentation.
The shift toward precision diversity is evident in the evolution of AI hardware. Early architectures such as NVIDIA Volta provided limited support for lower precision beyond FP16, whereas later architectures, including Turing and Ampere, expanded the range of supported formats. Table 6 traces this progression: Ampere GPUs introduced TF32 as a hybrid between FP32 and FP16 (NVIDIA 2020b), alongside broader support for BF16, INT8, and INT4 (NVIDIA Corporation 2017, 2018, 2020a).
| Architecture | Year | Supported Tensor Core precisions | Supported CUDA Core Precisions |
|---|---|---|---|
| Volta | 2017 | FP16 | FP64, FP32, FP16 |
| Turing | 2018 | FP16, INT8, INT4, INT1 | FP64, FP32, FP16, INT8 |
| Ampere | 2020 | FP64, TF32, BF16, FP16, INT8, INT4 | FP64, FP32, FP16, BF16, INT8 |
Newer architectures incorporate a growing diversity of numerical formats because different workloads bind at different points on the accuracy-throughput-energy trade-off. Precision support is therefore another form of workload matching, not a generic feature checklist.
The precision format used in hardware design has cascading implications across the entire system. Reducing from FP32 to FP16 cuts memory traffic in half, which matters far more than it might seem: because memory access dominates energy consumption, halving memory traffic can substantially reduce energy per inference when data movement is the bottleneck (Horowitz 2014). Simultaneously, Tensor Cores and systolic arrays can pack more lower-precision multiply-accumulate units into the same silicon area, raising peak throughput (Dally et al. 2021; Dally 2023). Lower-precision integer arithmetic also consumes less energy than FP32 in representative technology studies (Horowitz 2014), and the inference-focused TPUv1 was built around 8-bit multiply units (Jouppi et al. 2017). The systems insight is that reduced precision does not merely “save bits”: it simultaneously relieves the memory bandwidth bottleneck and increases compute density, attacking both sides of the roofline at once.
As AI models continue to scale, precision support connects the compute primitive discussion back to the memory wall: lower-bit formats matter when they reduce the bytes moved and keep the hardware’s matrix engines fed. The remaining architectural question is how these execution units, precision formats, and memory paths integrate into complete accelerator systems. Architectural integration determines how efficiently computational primitives become usable accelerator throughput. SIMD lanes, Tensor Cores, and systolic arrays are building blocks, but their full-chip organization varies significantly across AI processors; the choice of execution units, their numerical precision support, and their connectivity shape how effectively hardware can scale for deep learning workloads.
Intra-node interconnects: Scaling the stack
Mastery of the single-machine stack requires understanding how bits move between GPUs and the CPU. In the 1–8 GPU regime, scaling is achieved through high-speed intra-node interconnects such as NVLink and host-to-device PCIe transfers that mitigate the memory wall. These links form a bandwidth taper: data-movement speed falls at each step away from the compute units, from on-package HBM through the GPU-to-GPU NVLink bridge down to the host PCIe link. The PCIe step is much slower than the accelerator-local memory and inter-GPU fabric, so any data path that touches the CPU can become a performance hazard, the “PCIe Wall” that NVLink exists to avoid (NVIDIA Corporation 2020b). Section 1.4.5.1 develops this hierarchy quantitatively, where host-accelerator communication is the operative concern.
Intel Corporation. 2021a. “Intel Advanced Matrix Extensions (Intel AMX).” Intel Architecture Instruction Set Extensions Programming Reference.
Modern AI processors exhibit a range of design trade-offs based on their intended applications, and comparing their configurations reveals how deployment constraints drive architectural divergence. A training-optimized accelerator like the NVIDIA A100 packs many Streaming Multiprocessors with wide SIMD units and FP16 Tensor Cores because training throughput scales with aggregate multiply-accumulate capacity (NVIDIA Corporation 2020a). Google’s TPUv4 makes a radically different bet: just two cores per chip, each containing massive BF16 systolic arrays, a design that trades programmer flexibility for efficiency on dense matrix multiplications (Jouppi et al. 2023). At the inference end, Intel’s Sapphire Rapids dedicates Advanced Matrix Extensions (AMX) tile engines to INT8 and BF16, reflecting the insight from Model Compression that inference models tolerate reduced precision (Intel Corporation 2021a). Mobile neural engines take this further by shrinking matrix engines into low-power system-on-chip (SoC) blocks, prioritizing energy efficiency per operation over peak throughput. Table 7 compares these architectural configurations.
| Processor | Vector execution | Matrix engine | Processing Elements | Primary workloads |
|---|---|---|---|---|
| NVIDIA A100 | CUDA FP/INT lanes | Tensor Core instruction tiles | 108 SMs | Training, HPC |
| Google TPUv4 | Vector unit | \(128{\times}128\) BF16 systolic arrays | 2 cores/chip | Training |
| Intel Sapphire | 512-bit AVX-512 | AMX tiles (up to 16 rows by 64 bytes each) | Up to 60 cores | Inference |
| Mobile NPU | CPU/GPU/DSP vectors | Small matrix engines | Integrated NPU blocks | Mobile inference |
The pattern across these configurations reveals a consistent engineering principle: each design sacrifices generality to optimize for its target workload’s dominant operation and precision. Training chips invest silicon in wide floating-point datapaths; inference chips trade precision for throughput; mobile chips trade throughput for energy efficiency. No single design dominates across all workloads, which is precisely why hardware selection depends on workload analysis rather than headline specifications.
Cost-performance analysis
Architectural specifications define computational potential, but deployment decisions require cost-performance analysis. Raw compute is only one input: delivered performance may be limited by arithmetic, memory, communication, software support, or utilization.
The energy differential established in section 1.1.5 (where memory access costs dominate computation) drives the entire specialized hardware revolution. This disparity helps explain why many accelerators achieve only a fraction of peak compute on memory-bound workloads, while architectures that maximize data reuse (for example, systolic arrays on dense matrix kernels) can sustain substantially higher utilization under favorable conditions.
Consider an organization choosing between more older accelerators and fewer newer ones. Peak FLOP/s can mislead whenever important kernels are bandwidth or communication bound. Large training matrix multiplications may be compute bound, while optimizer steps, embeddings, normalization, and some inference paths may not be. The relevant comparison is delivered workload throughput per dollar under the expected utilization, memory capacity, bandwidth, and scaling efficiency.
These dynamics help explain the rapid adoption of newer accelerators despite higher unit prices. For memory-bound workloads, improvements in effective bandwidth (and the software stack’s ability to use it) can dominate real-world performance. Cloud deployment further complicates the analysis, as rental pricing, utilization, and operational overheads can change the break-even point between purchasing and renting hardware.
Table 8 provides an illustrative cost worksheet for common accelerators. Prices vary by vendor, region, contract, and purchase volume. More importantly, the throughput rows use different precision modes, so price per peak operation is meaningful within a row but not a controlled cross-device benchmark. The table is a prompt to make assumptions explicit, not a ranking.
| Accelerator | List price | Representative Peak Throughput (precision shown) | Memory Bandwidth | Price/performance |
|---|---|---|---|---|
| NVIDIA V100 | ~$10,000 | 125 TFLOP/s | 900 GB/s | $80/(TFLOP/s) |
| NVIDIA A100 | ~$15,000 | 312 TFLOP/s | 2,039 GB/s | $48.1/(TFLOP/s) |
| NVIDIA H100 | ~$25,000–30,000 | 494 TFLOP/s (TF32) | 3,350 GB/s | ~$50.6/(TFLOP/s) |
| Google TPUv4 | ~$8,000* | 275 TFLOP/s (BF16) | 1,200 GB/s | ~$29.1/(TFLOP/s) |
| Intel Gaudi 2 | ~$12,000 | 865 TFLOP/s (FP8) | 2,450 GB/s | $13.9/(TFLOP/s) |
The worksheet shows why a hardware decision cannot be reduced to one column. A bandwidth-bound kernel may value H100’s 3,350 GB/s memory path more than another increase in peak arithmetic, while a compute-bound kernel may show the reverse. Capacity can determine whether a model fits at all, and software support can determine whether the advertised path is reachable. The next section isolates the memory side of that decision.
Framework selection significantly impacts these economic decisions. Detailed hardware-framework optimization strategies are covered in ML Frameworks, while performance evaluation methodologies are discussed in Benchmarking.
The preceding sections introduced vector instructions, matrix-tile operations, and Tensor Cores capable of very high peak throughput. An NVIDIA A100 advertises 312 TFLOP/s on its FP16 tensor path, and newer accelerators add FP8 paths (NVIDIA Corporation 2020a; Kuzmin et al. 2022; Micikevicius et al. 2022). Dividing a ResNet-50 operation count by that peak yields a microsecond-scale arithmetic lower bound, not an inference-time prediction.
Kuzmin, Andrey, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jorn Peters, and Tijmen Blankevoort. 2022. “FP8 Quantization: The Power of the Exponent.” Advances in Neural Information Processing Systems (NeurIPS), 14651–62. https://doi.org/10.52202/068431-1065.
Micikevicius, Paulius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, et al. 2022. “FP8 Formats for Deep Learning.” arXiv Preprint arXiv:2209.05433.
NVIDIA’s Blackwell (B200) architecture extends this trend with an FP4 tensor path, advertised at up to 9 PFLOP/s dense or 18 PFLOP/s sparse peak throughput per chip (NVIDIA Corporation 2024). Reaching those peaks requires a model, numerical method, kernel, and sparsity pattern that use the corresponding path. Lower-bit hardware expands the design space; it does not make every workload safely reducible to FP4.
Yet real ResNet-50 inference takes milliseconds, not microseconds. The gap between theoretical capability and practical performance reveals the chapter’s central tension, first posed in the Purpose section: computational capability has outpaced our ability to feed data to processors. Moving data from memory costs orders of magnitude more energy than arithmetic, and memory bandwidth has improved more slowly than tensor arithmetic throughput. This disparity determines whether those 312 TFLOP/s translate into low sustained utilization or high sustained utilization on a particular workload (Horowitz 2014; Gholami et al. 2024).
Understanding this gap requires examining the memory systems that feed the compute primitives. The memory hierarchy is not merely supporting infrastructure; together with arithmetic intensity and mapping, it determines how much peak throughput a workload can use.
Self-Check: Question
In NVIDIA’s Ampere and Hopper architectures, how does 2:4 structured sparsity achieve an up to \(2\times\) theoretical speedup in Tensor Core matrix multiplication?
- Exactly two non-zero values are preserved in every contiguous four-element block, allowing weights to be stored in half the memory with 2-bit index metadata while sparse Tensor Cores perform math only on non-zeros
- Every alternate row of the weight matrix is dropped completely, allowing the GPU to halve the grid launch dimensions
- Four separate threads simultaneously execute one scalar multiply-accumulate instruction in a single clock cycle
- Floating-point numbers are converted to 2-bit integers, quadrupling register file capacity
Contrast the data movement mechanics and primary use cases of Weight-Stationary (WS) and Output-Stationary (OS) systolic array dataflows.
Place the following steps in the correct execution sequence for processing a matrix multiplication on an accelerator with Sparse Tensor Cores using 2:4 structured sparsity:
- Fine-tune or prune the weight matrix to ensure exactly two non-zero values exist in every four-element contiguous group
- Sparse Tensor Cores load compressed weights and decode metadata to gather matching input activation elements
- Multiply non-zero weights by gathered activations and accumulate into output partial sums at \(2\times\) dense throughput
- Compress the sparse weight matrix by storing only the non-zero values alongside 2-bit per-value selection metadata
What is the primary difference between the FP8 E4M3 and FP8 E5M2 numerical formats used in modern AI accelerators (such as NVIDIA Hopper and Ada Lovelace)?
- E4M3 uses 4 sign bits and 3 exponent bits, whereas E5M2 uses 5 sign bits and 2 exponent bits
- E4M3 has 4 exponent bits and 3 mantissa bits providing higher precision for forward-pass activations/weights, whereas E5M2 has 5 exponent bits and 2 mantissa bits providing wider dynamic range for gradients
- E4M3 is exclusively an integer fixed-point format, whereas E5M2 is a standard IEEE floating-point format
- E4M3 requires twice as many memory bytes per element as E5M2
True or False: In NVIDIA’s SIMT (Single Instruction, Multiple Threads) execution model, when threads within the same 32-thread warp execute divergent branches of an
if-elsecondition, both paths are executed concurrently in parallel at full hardware throughput.Describe the Tiling Principle in deep learning hardware mapping and explain why multi-level hierarchical tiling (from global memory down to registers) is necessary for high-throughput GEMM kernels.
AI Memory Systems
ResNet-50 can expose the gap between accelerator arithmetic and memory: convolution weights, activations, and intermediate results still have to arrive on time. Modern accelerators reach hundreds of TFLOP/s or more on selected low-precision paths (NVIDIA Corporation 2020a, 2024; Choquette 2023), but a kernel cannot use that capacity when its required traffic exceeds the relevant memory bandwidth. The AI memory wall names this growing mismatch between arithmetic demand and the memory system that feeds it.
Unlike conventional workloads, ML models require frequent access to large volumes of parameters, activations, and intermediate results, leading to substantial memory bandwidth demands. This challenge intersects with the data management strategies covered in Data Engineering. Modern AI hardware addresses these demands through advanced memory hierarchies, efficient data movement techniques, and compression strategies that promote efficient execution.
Understanding the AI memory wall
Definition 1.4: AI memory wall
The AI memory wall is the ML accelerator performance constraint that arises when arithmetic throughput \((R_{\text{peak}})\) outpaces memory bandwidth \((\text{BW})\). The core issue is whether the memory system can supply operands and absorb results quickly enough to sustain the specialized primitives introduced in section 1.3.
- Significance: A workload has reached the memory wall when increasing FLOP/s alone no longer improves it because the \(\frac{D_{\text{vol}}}{\text{BW}}\) term dominates the relevant execution time.
- Distinction: Unlike a general-purpose memory wall, which affects all computing, the AI memory wall is driven by the massive model state and activation storage required by deep learning.
- Common pitfall: A frequent misconception is that the memory wall is “fixed” by more memory. In reality, it is a bandwidth-latency gap: even with infinite capacity, the speed of moving data between memory and compute remains the fundamental physical bottleneck.
Data access can dominate the energy budget of a memory-intensive kernel.27 Figure 9 compares representative operation costs from one technology study, revealing the multi-order-of-magnitude gap between simple local arithmetic and an off-chip DRAM access.
27 Von Neumann bottleneck: Separating storage from compute requires instructions and data to cross interfaces rather than remain in local registers. In the representative 45 nm estimates from Horowitz (2014), a DRAM access consumes over 20,000\(\times\) the energy of an INT8 addition. The exact ratio changes with technology and operation, but locality remains a first-order design concern.
Horowitz, Mark. 2014. “1.1 Computing’s Energy Problem (and What We Can Do about It).” 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), 10–14. https://doi.org/10.1109/isscc.2014.6757323.
Quantifying the compute-memory performance gap
The energy disparity that figure 9 captures grows more severe with each hardware generation. Over the past two decades, peak computational capabilities have grown substantially faster than DRAM bandwidth (Gholami et al. 2024). This divergence creates a widening gap where accelerators possess massive computational power but cannot access data quickly enough to use it. Representative high-end accelerators can deliver on the order of \(10^3\) TFLOP/s of peak tensor throughput (for example, NVIDIA H100 delivering 989 TFLOP/s in FP16 or nearly 2,000 TFLOP/s in dense FP8) while providing approximately 3.35 TB/s of memory bandwidth (Choquette 2023). This implies that on the order of \(10^2\) FLOP of work per byte moved is required to fully use the compute, which can exceed the arithmetic intensity of many practical neural network workloads.
Gholami, Amir, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, and Kurt Keutzer. 2024. “AI and Memory Wall.” IEEE Micro 44 (3): 33–39. https://doi.org/10.1109/mm.2024.3373763.
The memory wall manifests through three critical constraints. First, the energy disparity: accessing DRAM can consume orders of magnitude more energy than a multiply-accumulate operation (Horowitz 2014; Sze et al. 2017), which often shifts bottlenecks from raw compute to power and data movement. Second, the bandwidth limitation: even TB/s memory systems may not feed large parallel compute arrays continuously on memory-bound workloads, leaving compute underutilized. Third, the latency hierarchy: off-chip memory access can require hundreds of cycles, creating pipeline stalls that cascade through parallel execution units.
Hardware balance (\(I_{\text{ridge}}\)): The paradigm partition
Different devices place the compute-bandwidth boundary at different intensities. The hardware balance \((I_{\text{ridge}})\) is the ratio of peak arithmetic throughput to peak memory bandwidth. In the roofline model, it is the ridge point where the compute and bandwidth ceilings intersect: \[ I_{\text{ridge}} = \frac{R_{\text{peak}}}{\text{BW}} \]
The ratio sets a device-specific boundary. Depending on the chosen precision, a high-end accelerator such as H100 may require roughly \(150\)–\(300\) FLOP/byte to reach its arithmetic ceiling, whereas a microcontroller may have a much lower ridge point. The same kernel can therefore be bandwidth bound on a compute-dense accelerator and compute bound on an edge processor. Hardware balance does not label either device as universally efficient; it identifies how much reuse a workload needs on that device.
Peak computational capability has expanded faster than memory bus performance across accelerator generations, and the size of that mismatch is what sets how much data reuse a kernel needs to stay fed. Figure 10 separates the two growth rates that produce the ridge point, each normalized to the V100 baseline, so the widening distance between them can be read directly.
The threshold required for a kernel to become compute bound has shifted across accelerator hardware generations. Follow the upward trajectory of hardware ridge points in figure 11 from V100 through B200, observing how the arithmetic intensity requirement increases over time.
Beyond performance limitations, memory access imposes a steep energy cost. Fetching data from off-chip DRAM consumes far more energy than performing arithmetic operations (Horowitz 2014). This inefficiency is particularly evident in machine learning models, where large parameter sizes, frequent memory accesses, and nonuniform data movement patterns exacerbate memory bottlenecks. The energy differential drives architectural decisions: Google’s TPUv1 achieved 30–80\(\times\) better performance per watt than contemporary CPUs and GPUs on Google’s inference benchmarks by minimizing data movement through systolic arrays and large on-chip memory (Jouppi et al. 2017). These design choices demonstrate that energy constraints, not computational limits, often determine practical deployment feasibility. Tracing a single tensor through every level of the memory hierarchy during a real inference pass makes these energy costs concrete.
Memory access patterns in ML workloads
Beyond raw computational throughput, an accelerator’s efficiency depends on its ability to continuously supply data to processing units without stalls. Neural networks impose three concurrent demands on this data supply. Model parameters (weights and biases) may number in the billions, requiring efficient storage and streaming to maintain throughput. Intermediate activations produced at each layer must be temporarily held for subsequent operations, contributing to memory overhead in deep architectures. During training, backpropagation adds a third demand: storing and accessing gradients for every parameter, further increasing data movement volume between compute units and memory.
As models increase in size and complexity, improvements in memory capacity and bandwidth become increasingly important. Although specialized compute units accelerate operations like matrix multiplications, their overall performance depends on the continuous, efficient delivery of data to the processing elements. In large-scale applications such as natural language processing and computer vision, models often incorporate millions to billions of parameters (Brown et al. 2020), and achieving high performance requires minimizing delays and stalls caused by inefficient data movement between memory and compute units (Narayanan et al. 2021; Kwon and Rhu 2018).
Kwon, Youngeun, and Minsoo Rhu. 2018. “TensorDIMM: A Practical Near-Memory Processing Architecture for Embeddings and Tensor Operations in Deep Learning.” Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, 740–53. https://doi.org/10.1145/3352460.3358284.
Lighthouse 1.2: Life of a tensor: GPU-hosted KWS
Recall the Keyword Spotting lighthouse summarized in Five Lighthouse Models. If the same one-second audio clip is batched on a GPU-hosted inference path, its physical journey through the memory hierarchy looks like this:
- DRAM (HBM): The tensor starts here.
- Size: 16,000 samples \(\times\) 2 bytes (FP16) = 32 KB.
- Latency: Fetching this from off-chip memory takes ~300 ns (plus queuing delay).
- Energy: Cost is ~20 pJ/bit. High cost.
- L2 cache: Device-memory requests may be served here when the requested cache lines are resident.
- Latency: ~4 ns.
- Access: Shared across multiple Streaming Multiprocessors (SMs).
- L1 cache/shared memory: A specific SM works on a tile of the audio, using hardware-managed cache or explicitly staged shared memory.
- Latency: ~1 ns.
- Locality: If an L1 request misses, it can still be served by L2 before an HBM access is required.
- Registers: The Tensor Core operates here.
- Latency: Typically a small number of cycles, depending on the instruction and architecture.
- Throughput: 312 TFLOP/s.
- Energy: Cost is ~0.1 pJ/bit.
Systems insight: Performance is bounded by the lower of peak compute throughput and bandwidth times arithmetic intensity. Which memory link supplies the relevant bandwidth depends on where the working set resides; HBM-to-L2 is one possible limiting path, not the roofline’s sole determinant.
One way to quantify this challenge is by comparing the data transfer time with the time required for computations. To do this, five key variables are defined: \(D_{\text{vol}}\) is the total data volume (bytes), \(\text{BW}\) is the available memory bandwidth (bytes/s), \(O\) is the number of floating-point operations, \(R_{\text{peak}}\) is the peak hardware throughput (FLOP/s), and \(\eta_{\text{hw}}\) is the realized hardware utilization.
The memory transfer time \(T_{\text{mem}}\) and compute time \(T_{\text{compute}}\) are expressed as: \[\begin{gather*} T_{\text{mem}} = \frac{D_{\text{vol}}}{\text{BW}} \\ T_{\text{compute}} = \frac{O}{R_{\text{peak}} \cdot \eta_{\text{hw}}} \end{gather*}\]
In a no-overlap model, \(T_{\text{mem}} > T_{\text{compute}}\) indicates memory-bound execution. With overlap, the larger term often dominates elapsed time, but dependencies, queueing, and incomplete overlap determine how much compute actually sits idle and which data-movement optimization can improve performance.
Figure 12 quantifies this disparity for specific public-count models and hardware generations. The gap between these curves represents the engineering challenge that drives accelerator memory-system design (Krizhevsky et al. 2012; Brown et al. 2020; Chowdhery et al. 2022; Dubey et al. 2024). Even high-bandwidth accelerators like NVIDIA’s B200 and AMD’s MI300X/MI325X-class devices cannot close this growth-rate gap by bandwidth alone (NVIDIA Corporation 2024; AMD 2023). Parameter counts for proprietary systems such as GPT-4 and Gemini are not officially disclosed, so they are not plotted as factual data points.
Chowdhery, Aakanksha, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, et al. 2022. “PaLM: Scaling Language Modeling with Pathways.” arXiv Preprint arXiv:2204.02311.
Dubey, Abhimanyu, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783.
NVIDIA Corporation. 2024. NVIDIA Blackwell Architecture. NVIDIA product documentation.
AMD. 2023. AMD Instinct MI300X Accelerators. AMD product documentation.
Irregular memory access
Many ML workloads combine regular dense kernels with irregular memory pressure from sparsity, embedding lookups, variable sequence lengths, attention/KV-cache traffic, and small batches. The dense parts are exactly why accelerators work so well; the irregular parts are where standard caching mechanisms and memory hierarchies struggle, leading to increased memory latency and inefficient bandwidth utilization.
Many ML systems contain both access regimes. Table 9 contrasts regular dense kernels with irregular components such as embeddings, sparse operators, and input-dependent routing.
One key source of irregularity in ML workloads stems from batch size and execution order. The way input data is processed in batches directly affects memory reuse, creating a complex optimization challenge. Small batch sizes decrease the likelihood of reusing cached activations and weights, resulting in frequent memory fetches from slower, off-chip memory. Larger batch sizes can improve reuse and amortize memory access costs, but simultaneously place higher demands on available memory bandwidth, potentially creating congestion at different memory hierarchy levels. This delicate balance requires careful consideration of model architecture and available hardware resources.
| Feature | Regular dense kernels | Irregular ML components |
|---|---|---|
| Access pattern | Contiguous or tiled | Indirect, scattered, or input-dependent |
| Cache locality | Often high after tiling | Often lower and workload-dependent |
| Data reuse | Structured across tiles or batches | Sparse or dynamic, depending on the operator |
| Dependencies | Static loop nests | Input-dependent routing or variable shapes |
| Examples | GEMM, convolution, dense attention | Embedding lookup, unstructured sparsity, MoE routing |
| Primary pressure | Compute or bandwidth, depending on arithmetic intensity | Latency, bandwidth, metadata, and load imbalance |
| Energy consequence | Depends on reuse and memory tier | Extra movement and metadata can increase energy |
Different neural network layers interact with memory in distinct ways beyond batch size considerations. Convolutional layers benefit from spatial locality, as neighboring pixels are processed together and small weight kernels are reused. Fully connected layers and dense attention use regular matrix accesses, but their large weights or key-value tensors can exceed cache capacity and become bandwidth bound when reuse is low. Variable sequence lengths complicate memory planning without making dense attention accesses intrinsically random.
Irregular access can instead arise from unstructured sparsity,28 embeddings, and input-dependent routing. Structured sparsity preserves regular blocks that specialized hardware can process efficiently, whereas compressed unstructured formats often require indirect or scattered accesses. Mixture-of-Experts routing is deterministic for a given input but varies across inputs, complicating batching, prefetching, and load balance.
28 Sparsity and memory irregularity: Unstructured sparse formats commonly gather nonzero elements through indirect addressing, which can weaken sequential access and make metadata overhead significant. Structured sparsity preserves hardware-friendly blocks, so its access behavior differs from arbitrary pruning. Without suitable kernels or hardware support, an unstructured sparse model can run slower than its dense counterpart despite performing fewer arithmetic operations.
Irregularity degrades performance through distinct mechanisms, and treating them as one memory problem can point to the wrong optimization. Scattered addresses may turn one logical access into several memory transactions, so each fetched cache line carries few useful payload bytes. Indirection can also serialize address generation: the system must fetch an index or pointer before it knows which payload to request, limiting memory-level parallelism even when aggregate bandwidth remains available. Input-dependent routing creates a third failure mode when some processing elements receive longer queues than others; completion time follows the slowest queue while neighboring units idle. Variable-sized allocations may fragment capacity, but fragmentation is neither necessary nor sufficient for irregular access. A sequential stream can saturate bandwidth without being irregular, whereas a dependent lookup chain can stall far below peak bandwidth.
Diagnosis should therefore begin with the mechanism that prevents forward progress. A low ratio of useful payload bytes to transferred bytes indicates poor spatial utilization or coalescing. Long memory stalls alongside unused bandwidth indicate dependent misses or too few concurrent requests to hide latency. Uneven queue depths or per-route work reveal load imbalance rather than a cache-capacity problem. Cache hit rate alone cannot distinguish these cases because a miss can waste bandwidth, expose latency, or both. Each diagnosis implies a different remedy: packing or blocking improves spatial utilization, reordering and prefetching expose concurrent requests, structured sparse formats reduce indexing overhead, and routing or partitioning balances work. The memory hierarchy determines the cost of the remaining accesses, while dataflow and tensor-layout choices determine how much irregularity software can remove before execution reaches the hardware.
The optimization order follows that diagnosis. Representation comes first when the format is under system control because a block-structured sparse format or grouped routing can remove indirection before a kernel executes. Request organization comes next: accesses issued together should target adjacent addresses, and enough independent requests must remain in flight to hide unavoidable latency. Additional cache capacity or bandwidth helps only when profiling shows that the regularized request stream still exhausts that resource. More bandwidth cannot resolve a serial pointer dependency, and more capacity cannot rebalance skewed work. The objective is to expose as much regular structure as possible while confining unavoidable irregularity to the smallest portion of execution.
The irregular access patterns and memory wall constraints examined in section 1.4.1 create formidable challenges, but they also reveal optimization opportunities. Although individual memory accesses may appear unpredictable, ML workloads exhibit structured reuse patterns at a higher level: the same weights are applied across batch elements, the same kernels slide across spatial dimensions, and the same attention patterns recur across sequence positions. Hardware designers exploit these regularities through carefully structured memory hierarchies that maintain frequently accessed data close to compute units, even when the specific access sequence varies.
Memory hierarchy
Modern AI accelerators exploit these structured reuse patterns through multilevel memory hierarchies: rather than treating memory as a monolithic resource, they organize storage into distinct tiers optimized for different access patterns, reuse distances, and energy costs. While general-purpose computing contends with unpredictable memory access, ML workloads exhibit structured reuse that can be optimized through careful data organization across multiple memory levels.
At the highest level, large-capacity but slow storage devices provide long-term model storage. At the lowest level, high-speed registers and caches ensure that compute units can access operands with minimal latency. Between these extremes, scratchpad memory provides software-managed on-chip storage, while accelerators use device-local DRAM, such as HBM or GDDR, for larger working sets.
The key pattern in table 10 is that larger, more distant storage generally provides greater capacity at higher latency and energy cost. The ratios are not uniform between adjacent levels, and nearby register or SRAM accesses need not cost more than every arithmetic operation.
| Memory Level | Relative latency | Bandwidth | Relative capacity | Example Use in Deep Learning |
|---|---|---|---|---|
| Registers | Lowest | Highest | Tiny | Storing operands for immediate computation |
| L1/L2 Cache (SRAM) | Very low | High | Small | Caching frequently accessed activations and small weight blocks |
| Scratchpad memory | Very low | High | Small to medium | Software-managed storage for intermediate computations |
| Device DRAM (HBM, GDDR, or LPDDR) | High | Moderate–Very High | Large | Storing model parameters and activations that do not fit on-chip |
| Host DRAM (typically DDR) | High locally; transfer adds latency | Moderate | Large to very large | Staging or offloading data outside accelerator memory |
| Flash Storage (SSD/NVMe) | Very high | Low | Very large | Storing pretrained models and checkpoints for later loading |
The hierarchy invites an apparently simple solution: build larger, faster off-chip memory and eliminate the need for on-chip SRAM entirely. The answer is rooted in physics: signal propagation within and between chips imposes a hard latency floor.
Napkin Math 1.2: The speed of light limit
Problem: Why is on-chip SRAM necessary instead of fetching all data from HBM?
Math:
- Distance: On an H100-class 814 mm² die, signals travel ~20 mm.
- Propagation latency: At \(\approx 0.5c\) in silicon, \(20\text{ mm}/(0.5c) \approx 130\text{ ps}\).
- Clock cycle: At 2 GHz, \(1/(2\text{ GHz}) = 500\text{ ps}\).
- DRAM: Off-chip HBM sits millimeters away on the package, but DRAM access latency plus protocol overhead = 100+ cycles.
Systems insight: The 20 mm propagation estimate is about 130 ps, less than one 500 ps clock cycle, so distance alone does not prove a one-cycle access impossible. A real HBM access includes DRAM-array latency, serialization, protocol traversal, interconnect delay, and queuing, producing this 100-plus-cycle access path. Local registers and SRAM are therefore required to feed compute units at 2 GHz. For transformer models, this complete access path is why weights must be staged in SRAM in tiles rather than read on demand from HBM, and why repeatedly streaming an entire KV cache can collapse inference throughput when its memory traffic cannot be hidden behind arithmetic.
On-chip memory
On-chip memory includes registers, Static Random-Access Memory (SRAM) caches, and software-managed scratchpads on the accelerator die. Each level trades capacity, latency, bandwidth, and management policy. Aggregate on-chip bandwidth can reach many TB/s, but the value depends on access width and concurrency, and capacity is limited and partitioned across cores or tiles. Device DRAM such as HBM provides much greater capacity at lower bandwidth and higher latency; its delivered rate likewise depends on access pattern. Registers provide the most immediate operand and accumulator storage, but they are not limited to only a few operands across the chip. A finite register file is divided among active threads or instructions, so per-thread allocation affects occupancy and tile size. Efficient kernels retain short-lived values in registers while moving reusable tiles through caches or scratchpads, avoiding spills and lower-tier transfers when capacity permits.
To reduce the need for constant data movement between registers and external memory, small but fast caches serve as an intermediary buffer. These caches store recently accessed activations, weights, and intermediate values, ensuring that frequently used data remains available with minimal delay. However, the size of caches is limited, making them insufficient for storing full feature maps or large weight tensors in machine learning models. As a result, only the most frequently used portions of a model’s parameters or activations can reside here at any given time.
Example 1.3: The Tensor Core contract
Scenario: An engineering team migrates a transformer workload to NVIDIA A100 GPUs expecting automatic Tensor Core acceleration, but profiling reveals Tensor Cores remain idle (NVIDIA Corporation 2020a).
Diagnosis: The workload used shapes and data types that selected less efficient kernels rather than the intended Tensor Core path.
Systems lesson: Specialized paths have operator, precision, layout, and shape requirements. Libraries can often handle nonaligned dimensions through padding or alternative kernels, but reaching high utilization requires verifying which path was actually selected.
For larger working datasets, many AI accelerators include scratchpad memory, which offers more storage than caches but with a key difference: it allows explicit software control over what data is stored and when it is evicted. Unlike caches, which rely on hardware-based eviction policies, scratchpad memory enables machine learning workloads to retain key values such as activations and filter weights for multiple layers of computation. This capability is useful in models like convolutional neural networks, where the same input feature maps and filter weights are reused across multiple operations. By keeping this data in scratchpad memory rather than reloading it from external memory, accelerators can significantly reduce unnecessary memory transfers and improve overall efficiency (Chen et al. 2017). On NVIDIA GPUs, the hardware exposes scratchpad memory to programmers as shared memory: a fast, software-managed SRAM region that all threads in a thread block can read and write, distinct from the hardware-managed L1/L2 caches. Custom ML kernels written in CUDA or Triton control this memory explicitly. FlashAttention achieves its substantial throughput gains for transformer attention layers precisely by exploiting this mechanism: rather than materializing the full \(S{\times}S\) attention score matrix in HBM, it tiles queries, keys, and values through SRAM/shared memory and writes only the final output back to HBM (Dao et al. 2022). The reduction in HBM round-trips—not fewer arithmetic operations—is the primary source of the speedup.
Off-chip memory
Once a model’s working set outgrows on-chip SRAM, the design question is whether it fits in the accelerator’s device-local DRAM. An accelerator may use HBM, GDDR, or LPDDR; these are alternative memory technologies rather than sequential tiers. Data that does not fit must be staged or offloaded through host memory or storage, adding interconnect transfers before the accelerator can use it.
Beyond on-chip memory, HBM provides rapid access to larger model parameters and activations that do not fit within caches or scratchpad buffers. It stacks multiple memory dies and uses wide interfaces to deliver much higher aggregate bandwidth than conventional off-chip DRAM interfaces. It is often used for model parameters and activations on high-performance accelerators, while its cost, packaging complexity, and power requirements make it less common in constrained edge devices.
Accelerators without HBM may instead use device-local GDDR or LPDDR. When a model exceeds device-memory capacity, systems can offload data to host DRAM, but the accelerator must move that data across an interconnect such as PCIe before computation. Effective offloading therefore stages only the portions needed for each computation and overlaps transfers when possible.
At the highest level of the hierarchy, flash storage and solid-state drives (SSDs) store large pretrained models, datasets, and checkpointed weights. These storage devices offer large capacities but are too slow for real-time execution, requiring models to be loaded into faster memory tiers before computation begins. For instance, in training scenarios, checkpointed models stored in SSDs must be loaded into DRAM or HBM before resuming computation, as direct execution from SSDs would be too slow to maintain efficient accelerator utilization (Narayanan et al. 2021).
The memory hierarchy thus balances competing objectives of speed, capacity, and energy efficiency. Moving data through multiple levels introduces latency and bandwidth costs, while limited capacity forces additional movement when working sets exceed local storage. Memory bandwidth is therefore a major determinant for bandwidth-bound workloads, alongside compute throughput and software overhead.
Memory bandwidth and architectural trade-offs
Advertised memory bandwidth is only a ceiling; achievable bandwidth depends on access pattern, batching, locality, and the host interface that feeds the accelerator. Modern accelerators exhibit distinct bandwidth-capacity trade-offs that directly shape which workloads they can serve efficiently. Representative data center accelerators provide memory bandwidth on the order of a few TB/s, often paired with tens of GB of high-bandwidth memory. Raw bandwidth alone, however, is misleading: what matters is achievable bandwidth for a given access pattern. Transformer attention, convolution, and fully connected layers can all realize different fractions of peak bandwidth because their reuse, tiling, and access regularity differ. Fully connected layers approach peak bandwidth only when batch sizes are large enough to amortize the cost of loading weight matrices—which connects directly to the batch-size sensitivity discussed in the roofline analysis in section 1.5. The practical consequence is that an accelerator’s effective bandwidth for a specific workload may be well below its advertised peak, making bandwidth-per-dollar a more reliable purchasing metric than peak bandwidth alone.
As established in section 1.4.1, on-chip memory access typically consumes energy in the single-digit-to-tens of picojoules per access, while external DRAM can be on the order of hundreds of picojoules per access, an orders-of-magnitude energy penalty. AI accelerators minimize DRAM access through three key strategies: weight stationarity (keeping model parameters in on-chip memory), input stationarity (buffering input activations locally), and output stationarity (accumulating partial sums on-chip).
Memory bandwidth scaling follows different trajectories across accelerator designs. GPU architectures scale bandwidth by adding memory channels, reaching on the order of 1 TB/s in mainstream products and a few TB/s in high-end systems. TPU-class designs achieve their bandwidth efficiency through systolic array dataflow and aggressive on-chip reuse, often trading flexibility for efficiency on dense tensor kernels. Mobile SoC designs face the tightest constraints, delivering on the order of hundreds of GB/s of unified memory bandwidth within a few-watt power envelope, which demands careful workload scheduling and thermal management.
HBM provides far higher bandwidth than commodity DDR memory, but at substantially higher cost and packaging complexity. High-bandwidth accelerators therefore trade higher memory-system cost for higher sustained performance on bandwidth-bound workloads. Edge accelerators often sacrifice bandwidth to meet tight cost and power targets while maintaining sufficient performance for inference workloads.
These bandwidth characteristics directly influence deployment decisions: cloud training prioritizes raw bandwidth for maximum model capacity, edge inference optimizes bandwidth efficiency for energy constraints, and mobile deployment balances bandwidth with cost limitations. Beyond the accelerator’s internal memory system, host-device traffic can introduce another bottleneck. When an input must cross a 64 GB/s PCIe link before using an accelerator with 2 TB/s of HBM bandwidth, the interface has roughly 30\(\times\) less bandwidth and can dominate small, frequent transfers (NVIDIA Corporation 2020a, 2020b). Resident data and direct storage or network paths need not traverse that host link first.
Host-accelerator communication
Machine learning accelerators, such as GPUs and TPUs, achieve high computational throughput through parallel execution. However, their efficiency is often constrained by host-accelerator data movement between the CPU and accelerator memory. Compared to many traditional workloads that keep most data within a single memory domain, AI workloads can require frequent transfers between CPU memory and accelerator memory, introducing latency, consuming bandwidth, and affecting overall performance.
Figure 13 traces the host-accelerator sequence for a GPU as the concrete accelerator (its “Memory for GPU” lane is the accelerator’s memory). Before computation begins, data is copied from CPU memory to the accelerator’s memory (step 1). The CPU then issues execution instructions (step 2), and the accelerator processes the data in parallel (step 3). Once computation completes, the accelerator writes its output to accelerator memory (the “Store results” arrow), and that result is copied back to the CPU (step 4). Consider the latency cost at every arrow: each transfer represents a potential bottleneck that must be managed to optimize end-to-end performance.
\begin{tikzpicture}[font=\sffamily\small]
\tikzset{%
Line/.style={line width=1.0pt,black!50}
}
\tikzset{
Box/.style={inner xsep=2pt,
draw=GreenLine,
line width=0.75pt,
node distance=1.5,
fill=GreenL!70,
align=flush center,
text width=26mm,
minimum width=26mm,
minimum height=10mm
},
}
\begin{scope}
\node[Box](B1){Main Memory};
\node[Box,right=of B1](B2){CPU};
\node[Box,right=of B2](B3){Memory for GPU};
\node[Box,right=of B3](B4){GPU};
\end{scope}
%
\begin{scope}[shift={(0,-6)}]
\colorlet{GreenL}{OrangeL}
\colorlet{GreenLine}{OrangeLine}
\node[Box](2B1){Main Memory};
\node[Box,right=of 2B1](2B2){CPU};
\node[Box,right=of 2B2](2B3){Memory for GPU};
\node[Box,right=of 2B3](2B4){GPU};
%
\end{scope}
%
\foreach \x in {1,2,3,4} {
\draw[Line] (B\x) -- (2B\x);
}
%
\draw[Line,-latex]($(B1)!0.2!(2B1)$)--
node[above,text=black,pos=0.26]{Copy processing data (1)}
($(B3)!0.2!(2B3)$);
\draw[Line,-latex]($(B2)!0.37!(2B2)$)--
node[above,text=black,pos=0.26]{Instruct the processing (2)}
($(B4)!0.37!(2B4)$);
%
\draw[Line,-latex]($(B4)!0.75!(2B4)$)--
node[above,text=black,pos=0.5]{Store results}
($(B3)!0.75!(2B3)$);
\draw[Line,-latex]($(B3)!0.85!(2B3)$)--
node[above,text=black,pos=0.25]{Copy the result (4)}
($(B1)!0.85!(2B1)$);
%
\draw[Line,-latex]($(B4)!0.57!(2B4)$)
to [out=10,in=350,distance=42]
node[above,text=black,pos=0.1,fill=white]{Execute parallel in each core (3)}
($(B4)!0.62!(2B4)$);
\end{tikzpicture}The key challenges in host-accelerator data movement include latency, bandwidth constraints, and synchronization overheads. The efficiency of ML accelerators depends as much on a continuous data supply as on raw computational power. Even high-performance GPUs and TPUs remain underutilized if data transfers are inefficient. Host and accelerator memory exist as separate domains, requiring explicit transfers over interconnects such as PCIe, NVLink, or proprietary links. Ineffective data movement causes execution stalls, making transfer optimization a priority.
Node-level interconnect topology
Optimizing data movement requires understanding the physical topology of the compute node. A typical AI server is not a flat mesh of connected devices but a hierarchy of bandwidths that tapers as data moves further from the chip.
Bandwidth tapers steeply as data moves farther from the accelerator.
At node level, three links define the bandwidth taper:
- Device-device interconnect (NVLink/Infinity Fabric): Modern multi-GPU nodes use specialized high-speed bridges like NVLink29 to connect accelerators directly, bypassing the host CPU. Bandwidth ranges from 600 GB/s to 900 GB/s per GPU (NVIDIA Corporation 2020b; Choquette 2023). This link matters whenever tensors must move between accelerators within one server, including model partitioning, activation exchange, and gradient synchronization during training. The hardware lesson for this chapter is the boundary: traffic that stays on the accelerator fabric is far cheaper than traffic that falls back through the host.
- Host-device interconnect (PCIe): The link between the CPU and the accelerator. Bandwidth ranges from 32 to 64 GB/s (PCIe Gen4/Gen5). Host-staged training data can bottleneck here, although resident data, direct storage or network I/O, and coherent accelerator links can use other paths. Multi-GPU servers may also provide multiple PCIe links rather than a single shared 64 GB/s path.
- Network interface card: The link from the host to the outside world, connecting to other nodes. Bandwidth ranges from 25 to 50 GB/s (200 Gb/s to 400 Gb/s Ethernet/InfiniBand30). This interconnect is the first step from single-node hardware reasoning into the next frontier: scale. Here, the point is that leaving the node moves traffic onto a much narrower and higher-latency path.
29 NVLink (NVIDIA Link): This direct GPU-to-GPU interconnect exists to keep accelerator-to-accelerator traffic off PCIe when tensors must move inside a server. Its 600–900 GB/s aggregate bidirectional bandwidth is roughly 300–450 GB/s per direction under balanced full-duplex traffic, several times the directional bandwidth of a standard PCIe link, so workloads with frequent cross-device tensor movement can remain in the fast part of the bandwidth taper (NVIDIA Corporation 2020b; Choquette 2023). The training algorithms that determine how much tensor traffic must cross this link are developed later; the hardware fact needed here is the bandwidth gap.
NVIDIA Corporation. 2020b. “NVLink: Scalable High-Performance Interconnect.” NVIDIA Technical Report 2020.
Choquette, Jack. 2023. “NVIDIA Hopper H100 GPU: Scaling Performance.” IEEE Micro 43 (3): 9–17. https://doi.org/10.1109/mm.2023.3256796.
30 InfiniBand: Its key feature for multi-node scaling is RDMA (Remote Direct Memory Access). With a GPU-aware path such as GPUDirect RDMA and registered GPU memory, a network adapter can transfer data directly between GPUs across nodes without staging it through host memory. This reduces host involvement and protocol overhead, so scaling is more often constrained by physical bandwidth, topology, and collective implementation rather than CPU-managed packet processing.
These three levels produce a characteristic bandwidth taper:
\[\begin{aligned} \text{HBM (3350 GB/s)} &\gg \text{NVLink (900 GB/s)} \\ &\gg \text{PCIe (64 GB/s)} \gg \text{Network (50 GB/s)} \end{aligned}\]
System efficiency depends on keeping data on the fastest suitable path. PCIe and network links provide substantially less bandwidth and higher latency than on-package memory, so placement and scheduling should prevent avoidable host and network crossings.
Offloading computation to specialized hardware introduces bus transfer overheads between host memory and accelerator VRAM.31 Trace the numbered steps in figure 13, following the sequence from initial PCIe memory copy to command launch, kernel execution, and result retrieval.
31 DMA (direct memory access): A dedicated hardware unit that manages the data copy (step 1) without direct CPU management, freeing the CPU to immediately issue computation commands (step 2). This concurrency is critical: without it, the accelerator can idle between compute batches, especially when host-to-device movement is on the critical path.
Latency and bandwidth limitations directly impact AI workloads. PCIe-class host interconnects are typically much slower than an accelerator’s on-package high-bandwidth memory, so large transfers can become bottlenecks, particularly in deep learning tasks. Synchronization overheads compound this problem when computation must wait for data transfers to complete. Efficient scheduling and overlapping transfers with execution are necessary to mitigate these inefficiencies.
Transfer optimization
The bandwidth taper described in this section creates a clear optimization hierarchy. Practitioners have two complementary strategies for mitigating transfer overheads: asynchronous data movement and unified memory abstraction.
DMA engines execute bulk transfers after software configures them, avoiding CPU-managed copying of every byte. When separate engines and buffers are available, the next batch can move while the accelerator computes on the current one. This overlap helps only when dependencies permit concurrency and neither path contends for the same limiting resource.
Unified Memory provides the second strategy, offering a single address space accessible by both CPU and accelerator. Rather than requiring explicit copies, the runtime migrates memory pages on demand when either processor accesses them. The programming model simplifies dramatically (a single malloc replaces complex staging logic), but introduces performance unpredictability. Page migrations triggered by access patterns can cause latency spikes, and small or scattered accesses may thrash pages back and forth across the interconnect. For this reason, production training workloads typically use explicit DMA-based transfers for predictable performance, while Unified Memory finds its niche in prototyping and workloads where development speed outweighs absolute throughput.
These overheads shape how model operations map to hardware. Convolution commonly exposes spatial reuse that maps well to tiling. Dense attention also has regular matrix structure, but its intermediate state and sequence-length scaling create different capacity and bandwidth pressure. Embeddings, sparse experts, and KV-cache reads add still other access patterns.
Model memory pressure
Model architecture determines which memory term binds. While multilayer perceptrons (MLPs), convolutional neural networks (CNNs), and transformer networks each require large parameter sets, their distinct access patterns create different pressure on weights, activations, bandwidth, and host transfers, so each demands a different accelerator optimization strategy.
The lighthouse models introduced in Lighthouse Models as Reference Workloads ground this analysis: ResNet-50 represents CNN workloads with high spatial reuse, GPT-2/Llama exemplifies transformer memory pressure, DLRM illustrates sparse embedding lookups that stress memory systems differently than dense operations, and MobileNetV2 demonstrates efficiency-optimized architectures with depthwise convolutions. Their different memory characteristics reveal how workload structure translates to hardware utilization.
Multilayer perceptrons
An MLP layer is a dense matrix multiplication followed by optional bias, activation, and normalization. Its weight traffic is expensive at small batch sizes, while larger batches reuse each weight across more examples and can make the matrix multiplication compute bound.
From a memory perspective, MLPs rely on large, dense weight matrices that frequently exceed on-chip SRAM capacity, necessitating accesses to device-local DRAM. PCIe or another host-device interconnect becomes part of the critical path only when inputs are staged from the host or weights and activations are offloaded beyond device memory.
MLPs have regular access patterns that support tiled prefetching and streaming. Accelerators stage weight and activation tiles in on-chip SRAM and overlap movement with arithmetic when dependencies permit. Delivered performance then depends on batch size, tensor shape, precision, and whether the working set remains in device memory (Chen et al. 2017).
Convolutional neural networks
CNNs are widely used in image processing and computer vision tasks. Unlike MLPs, which require dense matrix multiplications, CNNs process input feature maps using small filter kernels that slide across the image. This localized computation structure results in high spatial data reuse, where the same input pixels contribute to multiple convolutions.
CNN accelerators benefit from on-chip memory optimizations, as convolution filters exhibit extensive reuse, allowing weights to be stored in fast local SRAM instead of repeatedly accessing device DRAM. Activation maps require careful management due to their size, so CNN accelerators divide them into tiles that fit within on-chip buffers. This reduces device-memory traffic and improves efficiency (Chen et al. 2017).
Chen, Yu-Hsin, Joel Emer, and Vivienne Sze. 2017. “Using Dataflow to Optimize Energy Efficiency of Deep Neural Network Accelerators.” IEEE Micro 37 (3): 12–21. https://doi.org/10.1109/mm.2017.54.
CNNs can expose substantial weight and activation reuse, but intermediate feature maps may still dominate traffic. Tiling and on-chip buffering reduce device-memory movement; host-device transfer is a separate concern that matters when inputs or outputs are not resident. Eyeriss introduced a row-stationary dataflow to exploit convolutional reuse (Chen et al. 2016). In the broader taxonomy of section 1.7, it is a hybrid that keeps selected rows and partial sums local.
Transformer networks
The transformer architectures introduced in Transformers: Parallel Sequence Processing have become the dominant architecture for natural language processing and are increasingly used in other domains such as vision and speech recognition. Unlike CNNs, which rely on local computations, transformers perform global attention32 mechanisms, where each token in an input sequence can interact with all other tokens.
32 Attention mechanism: Bahdanau, Cho, and Bengio introduced attention for sequence-to-sequence models; Transformer self-attention later allowed each token to attend to every token in the sequence (Bahdanau et al. 2015; Vaswani et al. 2017); materializing full attention scores for a sequence of length \(S\) requires an \(S{\times}S\) matrix, producing quadratic intermediate storage. FlashAttention tiles the computation to avoid materializing that full matrix in HBM (Dao et al. 2022). The inference KV cache is separate: its capacity grows linearly with sequence length per layer, although each decode step reads an increasing cache (see Memory and KV cache).
Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2015. “Neural Machine Translation by Jointly Learning to Align and Translate.” International Conference on Learning Representations (ICLR).
Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. “Attention Is All You Need.” Advances in Neural Information Processing Systems (NeurIPS) 30: 5998–6008.
Dao, T., D. Y. Fu, S. Ermon, A. Rudra, and C. Ré. 2022. “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.” Advances in Neural Information Processing Systems (NeurIPS) 35: 16344–59. https://doi.org/10.52202/068431-1189.
These models are particularly challenging for accelerators because global token interaction creates large attention state while GPT-3-scale language models (Brown et al. 2020) can exceed on-chip memory capacity through sheer parameter count. As a result, frequent movement between HBM, caches, and compute units creates substantial latency and bandwidth pressure. If the model spills beyond accelerator memory or uses host offload, PCIe or NVLink transfers add another bottleneck. Unified Memory architectures can mitigate some programming complexity by handling movement between host and device memory at runtime, but they introduce additional latency when page migrations occur unpredictably. These pressures make high-bandwidth memory, tensor tiling, and memory partitioning central accelerator design concerns for transformer workloads.
Attention tiling, KV caching, and tensor layouts reduce or reorganize memory traffic. When transformer training partitions tensors across GPUs within a node, NVLink can provide more bandwidth than PCIe for the required exchanges; it offers no benefit if the workload does not cross that link. Asynchronous transfers can overlap movement with computation when the schedule and resources permit (Narayanan et al. 2021).
Narayanan, Deepak, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, et al. 2021. “Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM.” Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, 1–15. https://doi.org/10.1145/3458817.3476209.
Accelerator design implications
The diverse memory requirements of MLPs, CNNs, and transformers highlight the need for workload-specific accelerator design. Table 11 reveals how memory access patterns vary dramatically across model types.
| Model type | Weight size | Activation reuse | Memory access pattern | Primary Bottleneck |
|---|---|---|---|---|
| MLP (Dense) | Large, dense | Low | Regular, sequential (streamed) | Bandwidth (off-chip) |
| CNN | Small, reused | High | Spatial locality | Feature map movement |
| Transformer | Large, usually dense; sparse in MoE/pruned variants | Low-to-medium | Mostly regular GEMM plus KV-cache/attention traffic | Memory capacity + bandwidth |
Each model type creates different pressure. Small-batch MLP inference may stream large resident weight matrices from device memory, whereas larger batches increase reuse. CNNs can exploit spatial tiling and filter reuse. Transformer memory pressure depends on phase: training retains activations and exchanges partitioned tensors, while autoregressive inference repeatedly reads weights and an expanding KV cache.
To address these challenges, modern AI accelerators incorporate multi-tier memory hierarchies that balance speed, capacity, and energy efficiency. On-chip SRAM caches and scratchpad memories store frequently accessed data, while high-bandwidth external memory provides scalability for large models. Efficient interconnects, such as NVLink, help alleviate host-accelerator transfer bottlenecks, particularly in transformer workloads where memory movement constraints can dominate execution time.
Memory efficiency can matter as much as raw compute power. Representative energy studies place DRAM access far above simple on-chip arithmetic, while model architectures differ in reuse, capacity, and communication demand. The remaining question is quantitative: on a given accelerator, is a specific kernel limited by arithmetic or by the relevant memory path? The roofline model answers that question before engineers optimize the wrong resource.
Self-Check: Question
An accelerator delivers \(R_{\text{peak}} = 1{,}000\text{ TFLOP/s}\) (\(10^{15}\text{ FLOP/s}\)) in FP16 and has a High-Bandwidth Memory (HBM) subsystem delivering \(\text{BW} = 2.0\text{ TB/s}\) (\(2\times 10^{12}\text{ bytes/s}\)). What is the hardware balance point (ridge point \(I_{\text{ridge}}\)) of this system?
- \(50\text{ FLOP/byte}\)
- \(200\text{ FLOP/byte}\)
- \(500\text{ FLOP/byte}\)
- \(2{,}000\text{ FLOP/byte}\)
When training a 7-billion parameter model using standard mixed-precision (FP16/BF16) with the Adam optimizer, calculate the minimum memory required purely for model states (weights, gradients, and optimizer states) and explain why optimizer states dominate this footprint.
Arrange the following levels of a modern GPU memory hierarchy in order of access latency, from lowest latency (fastest) to highest latency (slowest):
- High-Bandwidth Memory (HBM3)
- Register File
- Pinned Host System Memory (DDR5 via PCIe)
- Shared Memory / L1 Cache
- On-Chip L2 Cache
Why does High-Bandwidth Memory (HBM) achieve significantly higher bandwidth (e.g., \(>2\text{ TB/s}\)) than traditional GDDR6X memory (e.g., \(\approx 760\text{ GB/s}\)) while maintaining comparable or lower power per bit?
- HBM operates at a \(10\times\) higher clock frequency than GDDR6X on standard PCB traces
- HBM uses optical photonic signaling to transmit data across the motherboard
- HBM eliminates all error correction codes and row buffer precharge cycles
- HBM vertically stacks DRAM dies using Through-Silicon Vias (TSVs) and connects to the GPU via a wide 1024-bit per stack silicon interposer bus at lower clock speeds
Memory allocated on the host CPU that is locked into physical RAM and prevents operating system paging, enabling direct DMA transfers over PCIe to the GPU, is called ____ host memory.
True or False: The AI Memory Wall refers solely to the limited physical capacity (GBs) of GPU DRAM, meaning that if an accelerator has sufficient gigabytes to store model weights, memory bandwidth will never bottleneck execution.
Roofline Model
Low arithmetic intensity pins the workload in the memory-bound regime.
The Roofline Model answers this question by plotting arithmetic intensity against attainable performance, revealing whether an operation meets a compute ceiling or a memory-bandwidth ceiling. Peak FLOP/s remains one input, but the model combines it with workload traffic and platform bandwidth to produce a more useful bound.
The Roofline Model33 (Williams et al. 2009) provides the standard framework for understanding whether workloads are compute bound or memory bound, directly connecting the memory wall discussion to practical performance analysis. This model enables quantitative reasoning about accelerator utilization and guides optimization decisions.
33 Roofline model: Introduced by Williams et al. (2009) at UC Berkeley, building on earlier I/O complexity work from the 1980s. Their specific contribution was making the compute vs. bandwidth trade-off visual and actionable: the characteristic roofline plot immediately reveals whether a kernel is compute bound (hitting the flat ceiling) or memory bound (hitting the sloped bandwidth line) and quantifies the gap to hardware limits. A kernel operating at only 50 percent of its ceiling has a clear 2\(\times\) utilization gap to close, making this the standard diagnostic tool for accelerator optimization.
Williams, Samuel, Andrew Waterman, and David Patterson. 2009. “Roofline: An Insightful Visual Performance Model for Multicore Architectures.” Communications of the ACM 52 (4): 65–76. https://doi.org/10.1145/1498765.1498785.
Performance is bounded by two ceilings, as equation 2 formalizes. Here, attainable performance \(R_{\text{attain}}\) and peak compute \(R_{\text{peak}}\) are in FLOP/s (often reported as TFLOP/s), peak bandwidth \(\text{BW}\) is in bytes/s (often TB/s), and arithmetic intensity \(I\) is in FLOP/byte: \[R_{\text{attain}} = \min(R_{\text{peak}}, \text{BW} \times I) \tag{2}\]
Definition 1.5: Arithmetic intensity
Arithmetic Intensity is the ratio of operations to bytes transferred across the memory interface being modeled (\(\text{FLOP}/\text{byte}\)). Together with that level’s bandwidth and the relevant compute peak, it determines which idealized roofline ceiling is lower.
- Significance: The threshold between the two ceilings is the ridge point \(R_{\text{peak}} / \text{BW}\). For an A100 (312 TFLOP/s FP16/BF16, 2.04 TB/s), it is roughly 153 FLOP/byte. Under the stated traffic models, a large well-tiled 1024 \(\times\) 1024 multiply reaches about 341.3 FLOP/byte, while standalone ReLU is about 0.125. They therefore meet different ceilings on the same hardware.
- Distinction: Unlike total FLOPs (a count of operations), arithmetic intensity is a ratio that characterizes the shape of a workload’s hardware demand. Two kernels with identical FLOPs but different memory access patterns have different arithmetic intensities and will be bottlenecked by different hardware resources.
- Common pitfall: A frequent misconception is that arithmetic intensity is a fixed property of an operation. In practice, it depends on implementation details: a naive matrix-multiply that reloads operands from DRAM for each output element has low arithmetic intensity; a blocked (tiled) implementation that reuses data from fast SRAM achieves high arithmetic intensity—the same mathematical operation, orders of magnitude apart in hardware efficiency.
With \(O\) as the dimensionless operation count and \(D_{\text{vol}}\) as data volume in bytes, equation 3 makes the definition operational. \[I = \frac{O}{D_{\text{vol}}} \tag{3}\]
The roofline visualization shows performance (TFLOP/s) on the vertical axis and arithmetic intensity (FLOP/byte) on the horizontal axis. At low arithmetic intensity, performance increases linearly with intensity (memory-bound region). Above the ridge point, performance saturates at peak compute (compute-bound region). Bottleneck diagnostic maps each regime to the optimizations that pay off and the ones that waste effort, so that classifying a workload as memory-bound or compute-bound tells the engineer directly whether a faster accelerator or more bandwidth is the binding investment.
Hardware ridge points
The ridge point, the hardware balance \(I_{\text{ridge}}\) established in section 1.4.2, is the arithmetic-intensity threshold at which an accelerator turns from memory-bound to compute-bound. Table 12 quantifies how different accelerators exhibit distinct characteristics based on their compute-to-bandwidth ratios:
| NVIDIA Accelerator | Peak FP16 | Bandwidth | Ridge Point |
|---|---|---|---|
| V100 (2017) | 125 TFLOP/s | 0.9 TB/s | 138.9 FLOP/byte |
| A100 (2020) | 312 TFLOP/s | 2.04 TB/s | 153 FLOP/byte |
| H100 (2022) | 989 TFLOP/s | 3.35 TB/s | 295.2 FLOP/byte |
These ridge points show that later tensor paths require more reuse to reach their arithmetic ceiling. A ridge-point comparison makes that trend concrete.
Napkin Math 1.3: The utilization gap
Problem: Why is it harder to get 100 percent utilization on an H100 than a V100?
Metric: The ridge point \(I_{\text{ridge}} = R_{\text{peak}} / \text{BW}\) (FLOP/byte) from section 1.4.2: how many math operations the hardware must perform for every byte of data loaded to keep the compute units busy.
Evolution:
- V100 (2017): 125 TFLOP/s / 0.9 TB/s ≈ 138.9 FLOP/byte.
- A100 (2020): 312 TFLOP/s / 2.04 TB/s ≈ 153 FLOP/byte.
- H100 (2022): 989 TFLOP/s / 3.35 TB/s ≈ 295.2 FLOP/byte.
Result: The intensity needed to reach the tensor peak has increased. Under this simple HBM roofline, a kernel with \(I\) = 200 FLOP/byte lies above the A100 ridge but below the H100 ridge. If its intensity and implementation remain unchanged, its idealized gain is then closer to the 1.6× bandwidth ratio than the 3.2× peak-compute ratio.
Systems insight: A standalone ReLU under a one-read, one-write FP32 traffic model performs about 1 operation per 8 bytes, far below the H100 tensor ridge. A well-tiled 1024 \(\times\) 1024 dense multiply reaches about 341.3 FLOP/byte under its idealized traffic model. Fusion is valuable for low-intensity chains because it can remove intermediate round-trips; it is not automatically the best optimization for every kernel.
Depthwise convolution, embedding lookup, LayerNorm, and softmax are useful low-intensity reference points because they spend more time moving bytes than doing arithmetic. Table 13 maps common neural network operations to the Roofline Model.
| Operation | Arithmetic Intensity | Classification | Lighthouse example |
|---|---|---|---|
| Conv2D (Dense) | 50–200 FLOP/byte | Straddles ridge; high-reuse cases compute-bound | ResNet-50 |
| Dense MatMul (large batch, well-tiled) | 64–256+ FLOP/byte | Often compute-bound at large batch | GPT-2 (batched projections) |
| Depthwise conv | 10–20 FLOP/byte | Memory-bound | MobileNet |
| Attention Softmax | Convention-dependent | Memory-bound | GPT-2 (Generation) |
| LayerNorm | 1–2 FLOP/byte | Memory-bound | GPT-2/Llama |
| Embedding lookup | \(<1\) FLOP/byte | Memory-bound | DLRM |
Napkin Math 1.4: Transformer layer analysis
For a transformer with hidden_dim = 768, batch = 32, seq = 512:
Attention QKV projection:
- FLOPs: 2 \(\times\) 3 \(\times\) 32 \(\times\) 512 \(\times\) 768 \(\times\) 768 = 58 GFLOP
- Bytes: (input + weights + output) = (32 \(\times\) 512 \(\times\) 768 + 3 \(\times\) 768 \(\times\) 768 + 32 \(\times\) 512 \(\times\) 768 \(\times\) 3) \(\times\) 2 ≈ 104.2 MB
- AI = 58 GFLOP / 104.2 MB = 556.4 FLOP/byte, which is compute bound on A100 (above 153 FLOP/byte threshold)
Softmax:
- FLOPs: 32 \(\times\) 12 \(\times\) 512 \(\times\) 512 \(\times\) 3 ≈ 302 MFLOP (exp, sum, div)
- Bytes: 32 \(\times\) 12 \(\times\) 512 \(\times\) 512 \(\times 2 \times 2\) = 402.7 MB
- AI = 302 MFLOP / 402.7 MB = 0.75 FLOP/byte under this simple operation count, which is memory-bound. Counts that weight exponentials and division as multiple operations produce a larger numerical intensity without changing the bottleneck classification.
Systems insight: This analysis explains why FlashAttention focuses on reducing memory traffic in attention rather than reducing FLOPs.
These classifications directly inform optimization strategy. Memory-bound operations benefit from reducing data movement through operator fusion, using reduced precision (FP16, INT8), and increasing arithmetic intensity through algorithmic changes like FlashAttention. Compute-bound operations, by contrast, benefit from maximizing hardware utilization through batching and parallelism, exploiting Tensor Cores and specialized compute units, and optimizing compute efficiency through tiling and scheduling.
Calculating memory bandwidth bounds
The Roofline Model’s memory-bound region is determined by the peak memory bandwidth. For an operation to achieve throughput \(R_{\text{ops}}\) (FLOP/s, often expressed in TFLOP/s) in the memory-bound regime, equation 4 gives the required bandwidth: \[\text{BW}_{\text{req}} = \frac{R_{\text{ops}}}{I} \text{ bytes/s} \tag{4}\]
When required bandwidth exceeds peak bandwidth, performance is capped according to equation 5. Here \(R_{\text{ops}}\) and \(R_{\text{attain}}\) are in FLOP/s and \(I\) is in FLOP/byte. \[R_{\text{attain}} = \text{BW} \times I \tag{5}\]
A convolution layer provides the compute-bound contrast.
Napkin Math 1.5: Convolutional layer analysis
Consider a Conv2D layer with input shape (batch = 32, channels = 128, height = 56, width = 56), output channels = 256, kernel size \(3{\times}3\) on an A100 GPU, which provides the compute-bound contrast:
Computational requirements:
- Output size: \(32 \times 256 \times 56 \times 56\) = 25.7M elements
- FLOPs per output: \(128 \times 3 \times 3 \times 2\) = 2,304 (multiply-add)
- Total FLOPs: 25.7M \(\times\) 2,304 = 59.2 GFLOP
Memory traffic analysis:
- Input: \(32 \times 128 \times 56 \times 56 \times 2\) = 25.7 MB (FP16)
- Weights: \(256 \times 128 \times 3 \times 3 \times 2\) ≈ 0.6 MB (FP16)
- Output: \(32 \times 256 \times 56 \times 56 \times 2\) = 51.4 MB (FP16)
- Total: 77.7 MB
Arithmetic intensity: \(I\) = 59.2 GFLOP / 77.7 MB = 762.2 FLOP/byte
Systems insight: This is well above A100’s ridge point of 153 FLOP/byte, so the simple Roofline Model places the operation in the compute-limited region. Peak throughput of ~312 TFLOP/s (FP16 with Tensor Cores) is therefore the ceiling, but realized performance still depends on tiling, occupancy, instruction mix, and library efficiency.
The convolutional layer’s high arithmetic intensity arises from its weight reuse pattern: the same \(3{\times}3\) kernel is applied across all spatial locations, amortizing the cost of loading weights across millions of output computations. This is the architectural pattern that makes CNNs so efficient on modern accelerators.
Napkin Math 1.6: Dense layer analysis
Consider a fully connected layer: input (batch = 32, features = 2048) → output (batch = 32, features = 2048) on the same A100:
Computational requirements:
- Matrix multiply: \((32 \times 2048) \times (2048 \times 2048)\)
- Total FLOPs: \(2 \times 32 \times 2048 \times 2048\) = 268.4 MFLOP
Memory traffic analysis:
- Input: \(32 \times 2048 \times 2\) = 131.1 KB (FP16)
- Weights: \(2048 \times 2048 \times 2\) = 8.4 MB (FP16)
- Output: \(32 \times 2048 \times 2\) = 131.1 KB (FP16)
- Total: 8.7 MB
Arithmetic intensity: \(I\) = 268.4 MFLOP / 8.7 MB = 31 FLOP/byte
This intensity sits below A100’s ridge point of 153 FLOP/byte, making this operation memory-bound. Attainable performance: \(R_{\text{attain}}\) = 2,039 GB/s \(\times\) 31 FLOP/byte = 63.3 TFLOP/s
Systems insight: Delivered throughput reaches only 20.3 percent of peak compute capability, demonstrating the memory wall effect for small batch sizes.
However, not all layers in a neural network exhibit this favorable profile. The fully connected (dense) layers that typically appear at the end of classification networks, or as the projection layers in transformers, have different arithmetic intensity characteristics. A dense layer provides the memory-bound contrast needed to predict where bottlenecks will occur in end-to-end model execution.
The dense layer’s lower arithmetic intensity stems from limited weight reuse: each weight is reused across the batch but lacks the additional spatial reuse of convolutional filters, so small-batch dense layers have much lower arithmetic intensity than convolutions. This difference explains why transformer inference (dominated by dense projections) is typically memory bound while CNN inference can be compute bound.
Napkin Math 1.7: LayerNorm analysis
LayerNorm with input shape (batch = 32, seq = 512, hidden = 768):
Computational requirements:
- Elements: \(32 \times 512 \times 768\) = 12.6M
- Approximate operations per element: mean reduction (1 ADD), variance (2 ADD, 1 MUL), normalization and affine transform (2 ADD, 2 MUL) ≈ 8; reciprocal-square-root overhead is amortized across each hidden vector
- Total FLOPs: 12.6M \(\times\) 8 = 100.7 MFLOP
Memory traffic:
- Input: 12.6M \(\times\) 2 = 25.2 MB
- Parameters (scale, bias): \(768 \times 2 \times 2\) = 3.1 KB (negligible)
- Output: 12.6M \(\times\) 2 = 25.2 MB
- Total: 50.3 MB
Arithmetic intensity: \(I\) = 100.7 MFLOP / 50.3 MB = 2.0 FLOP/byte
This arithmetic intensity sits severely below the A100 ridge point (77× below). Performance is limited to: \(R_{\text{attain}}\) = 2039 GB/s \(\times\) 2.0 FLOP/byte = 4.1 TFLOP/s
Systems insight: The simple HBM roofline bounds this standalone LayerNorm at about 1 percent of A100’s tensor peak. Actual latency also depends on fusion, reduction strategy, launch overhead, and which memory level supplies the data.
Standalone normalization performs little arithmetic relative to the bytes it touches, as the LayerNorm example shows. Within that isolated pass, each element is read and written with little reuse; fusion with adjacent operators can remove some of those round-trips.
Optimization by intensity regime
The roofline analysis directly informs optimization priorities, summarized in table 14.
| Intensity regime | Typical operations | Optimization priority | Common techniques | Expected impact |
|---|---|---|---|---|
| High AI (\(>200\) FLOP/byte) | Large convolutions | Maximize compute utilization. | Tensor Cores, thread-block tuning, and high occupancy. | Raises sustained compute utilization when the kernel already clears the ridge. |
| Medium AI (20–200 FLOP/byte) | Medium-sized dense layers | Balance compute and memory optimization. | Larger batches, register tiling, and fusion with adjacent operations. | Can move the binding bottleneck between memory and compute. |
| Low AI (\(<20\) FLOP/byte) | Small dense layers and element-wise operations | Reduce memory traffic. | Aggressive operator fusion, reduced precision (FP16 → INT8), and algorithmic changes. | Reduces external-memory traffic when fusion or layout changes are legal. |
| Very low AI (\(<2\) FLOP/byte) | Normalization layers and activation functions | Eliminate memory round-trips. | Fuse with adjacent operations and use in-place computation where legal. | Removes intermediate round-trips when adjacent operations can be fused. |
For low-AI operations, operator fusion is often the decisive optimization: LayerNorm combined with the Gaussian Error Linear Unit (GELU), for example, can become a single fused kernel. One of the most accessible levers for moving an operation up and right on the roofline is batching.
Napkin Math 1.8: Batch size and arithmetic intensity
Increasing batch size improves arithmetic intensity for matrix operations by amortizing weight loading. Equation 6 formalizes this relationship for an idealized FP16/BF16 dense layer \((B{\times}M){\times}(M{\times}N)\), counting one read each of the input and weights and one write of the output, where \(B\) is batch size and \(M\) and \(N\) are the input and output dimensions: \[I = \frac{2BMN}{2BM + 2MN + 2BN} \approx \frac{2BMN}{2MN} = B \quad (\text{when } 2MN \gg 2B(M+N)) \tag{6}\]
Example: Dense layer with M=N=2048 (FP16)
- Batch = 1: AI ≈ 1 FLOP/byte (memory bound)
- Batch = 32: AI ≈ 31 FLOP/byte (memory bound)
- Batch = 256: AI ≈ 204.8 FLOP/byte (compute bound on A100)
Systems insight: This explains why batching can produce large throughput improvements in production inference systems, as MLPerf Inference, the standardized benchmark suite covered in Benchmarking, demonstrates by separating bulk-throughput runs from latency-constrained serving runs (Reddi et al. 2019).
Reddi, Vijay Janapa, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, et al. 2019. “MLPerf Inference Benchmark.” 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), 446–59. https://doi.org/10.1109/isca45697.2020.00045.
The batch size analysis reveals why inference serving systems are designed around batching: it changes the arithmetic intensity regime of memory-bound workloads. However, batching introduces latency trade-offs, since requests must wait in a queue until a batch forms. This tension between throughput (favoring large batches) and latency (favoring small batches) is a central challenge in ML serving systems, explored in depth in Dynamic batching latency-throughput trade-offs.
When service latency limits batching, batch-1 or small-batch LLM decode often retains low weight reuse and low arithmetic intensity. The next worked model quantifies that particular regime rather than treating all LLM inference as identical.
Napkin Math 1.9: The throughput ceiling
Problem: A batch-1 GPT-2 calculation makes this bandwidth ceiling concrete: what is the maximum possible utilization of an NVIDIA A100 when running GPT-2 inference (batch size 1)?
The hardware constraints (the denominators)
Peak compute: 312 TFLOP/s (FP16 Tensor Core).
Peak bandwidth: 2.04 TB/s (HBM2e).
Ridge point \((R_{\text{peak}}/\text{BW})\): 312 TFLOP/s / 2.04 TB/s = 153 FLOP/byte (for FP16 Tensor Core).
Interpretation: Saturating this chip at FP16 precision requires 153 FLOP/byte operations for every byte loaded. The ridge point varies by precision: FP32 operations (19.5 TFLOP/s peak) have a ridge point of only ~9.6 FLOP/byte.
The workload characteristics (the numerator)
- Model: GPT-2 XL (1.5 billion parameters).
- Operation: Autoregressive generation (1 token at a time).
- Data movement: Must load all weights (3 GB @ FP16) for every token.
- Compute: Vector-Matrix multiplication. 2 \(\times\) Params ≈ 3 GFLOP.
- Arithmetic intensity: 3 GFLOP / 3 GB = 1 FLOP/byte
The prediction (iron law)
Since Actual Intensity (1) \(\ll\) Ridge Point (153 FLOP/byte), the system is bandwidth bound.
- Maximum throughput: 1 FLOP/byte \(\times\) 2.04 TB/s = 2.04 TFLOP/s.
- Utilization ceiling: 2.04 TFLOP/s (Actual) / 312 TFLOP/s (Peak) ≈ \(0.7\%\)
Systems insight: Without batching, a $15,000 GPU runs at less than 1 percent compute utilization in this weight-streaming decode model. Batching and weight quantization address that gap; key-value caching instead avoids recomputing prior attention states.
Through this derivation, the Roofline Model provides a diagnostic framework for identifying whether operations are compute bound or memory bound. Knowing that a workload is memory bound at 0.7 percent utilization is only the first step; the next challenge is translating this diagnosis into efficient execution plans that exploit accelerator architectures.
Self-Check: Question
A developer runs a LayerNorm kernel on an accelerator with \(R_{\text{peak}} = 312\text{ TFLOP/s}\) and \(\text{BW} = 1.5\text{ TB/s}\) (\(I_{\text{ridge}} = 208\text{ FLOP/byte}\)). The LayerNorm has an arithmetic intensity of \(I = 4\text{ FLOP/byte}\). What is the maximum attainable performance of this kernel, and what is the binding bottleneck?
- \(6.0\text{ TFLOP/s}\), bound by memory bandwidth
- \(312\text{ TFLOP/s}\), bound by peak compute capacity
- \(78\text{ TFLOP/s}\), bound by warp scheduler instruction issue rate
- \(1.5\text{ TFLOP/s}\), bound by PCIe bus transfer limits
Why has the hardware ridge point (\(I_{\text{ridge}}\)) increased dramatically across successive GPU generations (e.g., from Volta to Ampere to Hopper), and what pressure does this trend place on compiler and kernel developers?
True or False: When a kernel operates in the memory-bound regime of the Roofline model (\(I < I_{\text{ridge}}\)), doubling the accelerator’s peak tensor compute capability (\(R_{\text{peak}}\)) without changing memory bandwidth will double the kernel’s execution speed.
In the Roofline model, the transition point on the horizontal axis where the memory-bandwidth ceiling intersects the peak-compute ceiling is known as the hardware ____ point.
An engineer profiles a transformer inference workload and discovers that the attention Softmax kernel is heavily memory bandwidth bound. Which of the following optimization techniques directly increases arithmetic intensity to move the kernel closer to the compute-bound regime?
- Upgrading host CPU RAM to DDR5 to decrease kernel enqueue latency
- Fusing the scale, mask, Softmax, and dropout operations into a single kernel to keep intermediate activations in registers/SRAM
- Increasing the clock frequency of the GPU Tensor Cores by \(15\%\)
- Disabling warp scheduler out-of-order instruction issue
Hardware Mapping
Consider a \(3{\times}3\) convolution running on an accelerator tile. The mathematical operation is fixed, but the execution plan is not. One schedule can keep a filter in local registers while many output pixels stream past it; another can advance pixel by pixel and reload the same filter values repeatedly from a slower memory tier. Both schedules compute the same tensor. Only one turns the reuse in the convolution into real bandwidth savings.
Definition 1.6: Mapping in AI acceleration
Mapping in AI Acceleration is the accelerator-compiler process of binding the Logical Computation Graph to the Physical Hardware Topology by deciding which operations execute on which processing elements, which data resides in which memory tier, and in what temporal order.
- Significance: Within the D·A·M taxonomy, mapping is a machine-axis decision that helps determine how closely an operation approaches the roofline bound \(\min(R_{\text{peak}},\; \text{BW} \times I)\). A poor tiling choice that forces unnecessary DRAM accesses can reduce effective arithmetic intensity enough to move a nominally compute-bound operation into the bandwidth-bound regime.
- Distinction: Traditional compilation often emphasizes instruction selection, register allocation, and scheduling for general-purpose processors. Accelerator mapping gives equal prominence to spatial placement and explicit data movement across a Dataflow Architecture. The reason is physical: in the 45 nm comparison introduced earlier, off-chip DRAM access consumed roughly 200\(\times\) the energy of a local integer multiply-accumulate, although the ratio depends on the technology and operation.
- Common pitfall: A frequent misconception is that mapping is automatically handled by frameworks. For general GPU workloads, compilers like Accelerated Linear Algebra (XLA) can find strong mappings for common kernels; for specialized accelerators (systolic arrays, custom ASICs), compiler-generated mappings may still lag hand-tuned schedules because the compiler’s search space is limited by the time budget at compilation.
The convolution example exposes the three decisions that recur throughout accelerator compilation. Placement assigns the multiply-accumulate work to processing elements so parallelism does not turn into idle time or interconnect congestion. Allocation keeps weights, activations, and partial sums in the memory tier where their next use will occur, rather than letting reuse spill back to DRAM. Scheduling orders loops and kernels so the chosen placement and allocation remain valid over time. A poor choice in any one dimension can collapse a high-arithmetic-intensity operation back into a bandwidth-bound execution. In practice, these choices are too coupled for developers to manage by hand at model scale, which is why systems such as XLA, TVM, and TensorRT lower high-level models and search or select execution plans within their compile-time and hardware budgets. Section 1.8 examines that compiler support in detail.
Placement and allocation
Translating a model’s computational graph into efficient hardware execution requires solving two tightly coupled problems. Computation placement determines which operations run on which processing elements, balancing parallelism against communication costs. Memory allocation determines where data resides within the memory hierarchy, trading capacity against access latency. These two decisions interact: placing operations on distant processing elements increases the memory bandwidth required to shuttle data between them, while allocating data to fast but small on-chip memory limits which operations can execute concurrently. Getting either wrong leaves thousands of processing elements idle or starved for data.
Computation placement
Computation placement is the process of assigning operations to an accelerator’s processing elements (PEs) to expose parallelism, limit idle time, and reduce unnecessary data movement. Modern accelerators contain many such resources: the NVIDIA H100 has more than 16,000 CUDA cores and more than 500 Tensor Cores (Choquette 2023), TPUs organize thousands of multiply-accumulate units into systolic arrays (Jouppi et al. 2017), and wafer-scale processors such as Cerebras’ CS-2 integrate more than 850,000 cores (Systems 2021). At these scales, placement inefficiencies become measurable because idle cores and redundant transfers waste both time and energy.
The difficulty of placement depends on workload regularity. CNNs and dense transformer operations are structured and tile well, although transformers combine kernels with different shapes and reuse patterns. Graph Neural Networks (GNNs) are harder to partition because sparse, input-dependent neighborhoods can create load imbalance and scattered communication. Table 15 lists the core challenges placement must address across these workload types. Static schedules work well for many fixed-shape dense kernels; runtime-aware placement is useful when shapes, sparsity, routing, or resource availability vary.
| Challenge | Impact on execution | Key Considerations for Placement |
|---|---|---|
| Workload imbalance | Some processing elements finish early while others remain overloaded, leading to idle compute resources. | Distribute operations evenly to prevent stalls and ensure full utilization of PEs. |
| Irregular computation patterns | Sparse graphs, routing, and variable shapes can create nonuniform work that complicates static placement. | Use adaptive placement when workload characteristics vary at run time. |
| Excessive data movement | Frequent memory transfers introduce latency and increase power consumption. | Keep frequently used data close to the compute units and minimize off-chip memory accesses. |
| Limited interconnect bandwidth | Poorly placed operations can create congestion, slowing data movement between PEs. | Optimize spatial and temporal placement to reduce communication overhead. |
| Model-specific execution needs | CNNs, transformers, and GNNs require different execution patterns, making a single placement strategy ineffective. | Tailor placement strategies to match the computational structure of each model type. |
Good placement can substantially reduce latency, while poor placement leaves processing elements idle or increases communication. Accelerators therefore combine static mappings for regular kernels with runtime-aware scheduling where workload behavior varies. Placement decisions also interact directly with the next concern: where the data those processing elements need resides in the memory hierarchy.
Memory allocation
While computation placement determines where operations execute, memory allocation defines where data resides and how it flows through the hierarchy. The goal is to keep reused data near the processing elements without exceeding faster-tier capacity. GPUs expose global memory, shared memory, caches, and registers that kernels coordinate through tiling (NVIDIA Corporation 2020a). TPUs use on-chip buffers to stage activations and weights for systolic-array execution (figure 8) (Jouppi et al. 2017), while wafer-scale processors partition memory and computation to control interconnect traffic (Systems 2021). General-purpose processors and accelerators both use memory hierarchies, but many accelerators expose more placement decisions to compilers and kernels. Poor allocation can therefore impose three related penalties: additional latency, greater energy from higher-tier accesses, and lower throughput when processing elements wait for data.
Systems, Cerebras. 2021. “Wafer-Scale Deep Learning Acceleration with the Cerebras CS-2.” Cerebras Technical Paper 2021.
The severity of these penalties varies by workload. CNNs rely on structured, localized access patterns and benefit from well-defined memory layouts that facilitate predictable reuse (Chen et al. 2016). Transformer models require access to large parameter sets and intermediate activations, making them sensitive to capacity and bandwidth. GNNs add irregular sparse structures that complicate allocation and prefetching.
Capacity is only the first allocation test. A tensor can fit in accelerator memory yet still dominate execution if each kernel rereads it from a distant tier; conversely, tiling can keep a larger working set efficient by preserving reuse locally. The allocation diagnostic is therefore traffic, not fit alone: designers must count how often values cross each memory boundary and keep frequently reused values in the fastest feasible tier. Table 16 summarizes these challenges. Systems combine static memory plans for known shapes with dynamic buffer management when shapes vary, while device capacity limits which models fit without partitioning or offload.
| Challenge | Impact on Execution | Key Considerations for Allocation |
|---|---|---|
| High memory latency | Slow data access delays execution and reduces throughput. | Prioritize placing frequently accessed data in faster memory locations. |
| Limited on-chip storage | Small local memory constrains the amount of data available near compute units. | Allocate storage efficiently to maximize data availability without exceeding hardware limits. |
| High off-chip bandwidth demand | Frequent access to external memory increases delays and power consumption. | Reduce unnecessary memory transfers by carefully managing when and how data is moved. |
| Irregular memory access patterns | Some models require accessing data unpredictably, leading to inefficient memory usage. | Organize memory layout to align with access patterns and minimize unnecessary data movement. |
| Model-specific memory needs | Different models require different allocation strategies to optimize performance. | Tailor allocation decisions based on the structure and execution characteristics of the workload. |
Combinatorial complexity
The small convolution example also explains why hardware mapping becomes a combinatorial search problem. Keeping a filter local improves reuse only if the chosen processing elements have enough nearby storage and if the loop order revisits that filter before eviction. Parallelizing across more processing elements improves throughput only until synchronization and interconnect traffic consume the gain. Table 17 lists recurring tensions among placement, allocation, and scheduling. Because changing one decision alters the feasible and useful choices for the others, practical mappers search or optimize them jointly rather than choosing each dimension independently.
These interacting factors define a vast combinatorial design space where small variations in mapping decisions lead to large differences in performance and energy efficiency. Unlike traditional workloads with predictable execution patterns, machine learning models introduce diverse computational structures that require mappings adapted to data reuse, parallelization opportunities, and memory constraints. The search space grows combinatorially, making exhaustive search infeasible. Three sources of variation contribute to this complexity:
| Dimension | Placement considerations | Allocation and Scheduling Considerations |
|---|---|---|
| Computational granularity | Fine-grained placement enables greater parallelism but increases synchronization overhead. | Coarse-grained scheduling reduces synchronization overhead but may limit flexibility. |
| Spatial vs. Temporal Mapping | Spatial placement enhances parallel execution but can lead to resource contention and memory congestion. | Temporal scheduling balances resource sharing but may reduce overall throughput. |
| Memory and Data Locality | Placing data closer to compute units minimizes latency but may reduce overall memory availability. | Allocating data across multiple memory levels increases capacity but introduces higher access costs. |
| Communication and synchronization | Co-locating compute units reduces communication latency but may introduce contention. | Scheduling synchronization mechanisms mitigates stalls but can introduce additional overhead. |
| Dataflow and Execution Ordering | Static placement simplifies execution but limits adaptability to workload variations. | Dynamic scheduling improves adaptability but adds scheduling complexity. |
Ordering computation and execution
Machine learning workloads are often structured as nested loops that iterate over various dimensions of computation. For instance, a matrix multiplication kernel may loop over batch size (\(B\)), input features (\(C_{\text{in}}\)), and output features (\(C_{\text{out}}\)). The order in which these loops execute has a profound effect on data locality, reuse patterns, and computational efficiency.
Ignoring dependences and equivalent orderings, the number of ways to arrange \(n_{\text{loops}}\) loops has the toy upper bound: \[ N_{\text{order}} = n_{\text{loops}}! \] which scales rapidly. A typical convolutional layer may involve up to seven loop dimensions, leading to: \[ 7! = 5,040 \text{ possible execution orders.} \]
Mapping choices explode combinatorially as loop dimensions grow.
If each memory level could choose an ordering independently, the corresponding toy upper bound would expand as: \[ (n_{\text{loops}}!)^{N_{\text{mem}}} \] where \(N_{\text{mem}}\) is the number of memory hierarchy levels. This rapid expansion shows why execution order optimization matters: poor loop ordering can lead to excessive memory traffic, while an optimized order improves cache utilization (Sze et al. 2017).
Parallelization across processing elements
Modern AI accelerators use thousands of processing elements to maximize parallelism, but determining which computations should be parallelized requires careful analysis. Excessive parallelization can introduce synchronization overheads and increased bandwidth demands, while insufficient parallelization leads to underutilized hardware.
Before legality and capacity constraints are applied, the number of ordered ways to select loops for parallel execution has the upper bound: \[ \mathcal{P}_{\text{parallel}} = \frac{n_{\text{loops}}!}{(n_{\text{loops}}-k_{\text{parallel}})!} \] where \(n_{\text{loops}}\) is the number of loops, and \(k_{\text{parallel}}\) is the number selected for parallel execution. For a six-loop computation where three loops are selected, this unconstrained count is: \[ \frac{6!}{(6-3)!} = 120. \]
Even for a single layer, the candidate count can reach hundreds before invalid or equivalent strategies are removed. Each remaining strategy affects data synchronization, memory contention, and overall compute efficiency.
Memory placement and data movement
The hierarchical memory structure of AI accelerators introduces additional constraints, as data must be efficiently placed across registers, caches, shared memory, and off-chip DRAM. Data placement impacts latency, bandwidth consumption, and energy efficiency. Frequent access to slow memory creates bottlenecks, while optimized placement reduces costly memory transfers.
If each computational dimension had \(n\) independent choices at every memory level, the resulting toy upper bound would be: \[ \mathcal{M}_{\text{placement}} = n^{N_{\text{comp}} \times N_{\text{mem}}} \] where:
- \(n\) = number of placement choices per level,
- \(N_{\text{comp}}\) = number of computational dimensions,
- \(N_{\text{mem}}\) = number of memory hierarchy levels.
For a model with:
- \(N_{\text{comp}} = 5\) computational dimensions,
- \(N_{\text{mem}} = 3\) memory levels,
- \(n = 4\) possible placement choices per level,
the number of possible memory allocations is: \[ 4^{5 \times 3} = 4^{15} = 1,073,741,824. \]
Mapping search space
This unconstrained mapping example exceeds a billion combinations, although many are illegal, coupled, or equivalent. Multiplying the toy counts gives an upper-bound illustration of the mapping search space: \[ \mathcal{S}_{\text{mapping}} = \left( n^{N_{\text{comp}}} \times n_{\text{loops}}! \times \frac{n_{\text{loops}}!}{(n_{\text{loops}}-k_{\text{parallel}})!} \right)^{N_{\text{mem}}} \] where:
- \(n^{N_{\text{comp}}}\) represents memory placement choices,
- \(n_{\text{loops}}!\) accounts for computation ordering choices,
- \(\frac{n_{\text{loops}}!}{(n_{\text{loops}}-k_{\text{parallel}})!}\) captures parallelization possibilities,
- \(N_{\text{mem}}\) is the number of memory hierarchy levels.
This equation is not an exact count of legal schedules because choices interact and hardware constraints prune the space. It illustrates why candidate spaces can grow rapidly and why exhaustive search becomes impractical. A concrete example makes the impact of these choices tangible.
Example 1.4: Loop ordering in a small convolution
Consider a convolution applying 16 filters of size \(3{\times}3\) to an \(8{\times}8\) single-channel input. The computation can be expressed as five nested loops iterating over output rows (\(H_{\text{out}}\)), output columns (\(W_{\text{out}}\)), filter count (\(C_{\text{out}}\)), filter height (\(F_h\)), and filter width (\(F_w\)). The \(5! = 120\) permutations are an unconstrained upper bound: legal orderings preserve the mathematical result, although floating-point reduction order can change rounding, and they generate different memory traffic.
Ordering A (weight-stationary): Place the filter loops (\(C_{\text{out}}\), \(F_h\), \(F_w\)) outermost and the spatial loops (\(H_{\text{out}}\), \(W_{\text{out}}\)) innermost. Each \(3{\times}3\) filter is loaded into registers once and then applied across all 36 output positions before the next filter is loaded. Total weight loads: \(16 \times 9 = 144\) values, each loaded exactly once.
Ordering B (output-stationary): Place the spatial loops outermost and the filter loops innermost. For every output position, all 16 filters must be loaded, applied, and their partial sums accumulated before advancing to the next position. If the register file cannot hold all 16 filters simultaneously, filters are repeatedly fetched from cache or DRAM. In the worst case, each of the 36 output positions reloads all 144 filter weights, producing \(36 \times 144 = 5{,}184\) weight reads.
Systems insight: Under these two limiting assumptions, Ordering A reduces modeled weight reads by 36\(\times\) relative to the worst case for Ordering B. Real kernels also account for activation and partial-sum traffic, register capacity, cache behavior, and vectorization. The example nevertheless shows why two mathematically equivalent loop orders can place very different demands on the memory system.
The combinatorial growth revealed by this analysis poses a practical challenge: explaining how practitioners achieve strong performance despite a large candidate space. Exhaustive enumeration is often impractical, yet production systems routinely find useful schedules for common kernels. The answer lies in a small set of principled dataflow patterns that reduce the search to a manageable set of strategic choices.
Self-Check: Question
In neural network hardware mapping, what distinguishes a spatial mapping decision from a temporal mapping decision?
- Spatial mapping refers to compiling graph IR, whereas temporal mapping refers to runtime CUDA kernel launches
- Spatial mapping determines precision formats (FP16 vs INT8), whereas temporal mapping determines memory allocation sizes
- Spatial mapping assigns computational tasks to specific physical execution units (e.g., PEs or SMs) simultaneously in parallel, whereas temporal mapping determines the execution ordering and loop scheduling over time on those units
- Spatial mapping operates only on convolutional layers, whereas temporal mapping operates only on transformer attention layers
Explain why finding the optimal hardware mapping (tiling sizes, loop orders, and spatial partitioning) for a deep neural network on a target accelerator is a combinatorially hard optimization problem.
Explain why reordering loop nests in a tensor contraction (e.g., changing from \(I \to J \to K\) to \(K \to I \to J\)) alters memory bandwidth demands and execution speed without changing the total mathematical operation count.
When mapping a tensor computation to a multi-level memory hierarchy, what is the primary objective function optimized by spatial and temporal tiling?
- Maximizing the total number of intermediate tensors written to host DDR memory
- Maximizing data reuse in the fastest, closest memory tiers (registers and SRAM) to minimize traffic to slower, energy-expensive DRAM
- Ensuring every warp thread executes different instruction streams simultaneously
- Converting all 2D matrix multiplications into 1D scalar operations
Dataflow Optimization
The mapping strategies from section 1.6 establish where computations execute and where data resides, but they do not specify dataflow optimization: how data flows through processing elements during execution. A systolic array might process a matrix multiplication with weights in local memory, but the order in which weights, inputs, and outputs move through the array directly determines memory bandwidth consumption and energy efficiency. The choice among strategies directly impacts whether an accelerator operates in the compute-bound or memory-bound region identified by the Roofline analysis—which is why compilers (section 1.8) and runtime systems (section 1.9) must select appropriate dataflow patterns based on workload characteristics.
Three recurring decisions structure the dataflow choices considered here:
- Locality: Weight-stationary, output-stationary, and input-stationary strategies each make different choices about what to cache near compute units, trading off different memory access patterns.
- Organization: Tensor layouts (NHWC vs. NCHW) determine whether memory accesses align with hardware preferences, with performance impacts that can be large when layout conversions or uncoalesced access block the fast path.
- Combination: Kernel fusion and tiling restructure computation to minimize memory traffic, often producing large speedups on low-arithmetic-intensity operations by avoiding intermediate writes and reloads.
These patterns provide a compact vocabulary for reasoning about many dataflow decisions without exhaustive search. The next sections examine each decision in turn, then show how they combine for specific neural network architectures including ResNet-50, GPT-2, and MLPs.
Building blocks of mapping strategies
These three decisions map to four foundational techniques: data movement patterns (weight-stationary, output-stationary, input-stationary), memory-efficient tensor layouts (channels-last vs. channels-first), kernel fusion (combining operations to eliminate intermediate writes), and tiling (partitioning computations into memory-friendly blocks). Together, these building blocks reduce the mapping search space: heuristic and model-driven optimizers can combine them instead of rediscovering the same data-movement choices from scratch.
Data movement patterns
While computational mapping determines where and when operations occur, its success depends heavily on how efficiently data is accessed and transferred across the memory hierarchy. Some machine learning workloads are regular but exceed cache capacity; others, such as sparse lookups and routed models, also have irregular access patterns. Both cases make data movement strategy critical to overall system performance.
Even when computational units are mapped efficiently, poor data movement strategies degrade performance by causing frequent memory stalls and leaving hardware resources idle. If data cannot be supplied to processing elements at the required rate, computational units stall, increasing latency, memory traffic, and energy consumption (Chen et al. 2016). Listing 15 illustrates how data movement inefficiencies affect the backbone computation of many machine learning models through a typical matrix multiplication operation.
## Matrix multiplication where:
## weights: [512x256] - model parameters
## input: [256x32] - batch of activations
## Z: [512x32] - output activations
## Computing each output element Z[i,j]:
for i in range(512):
for j in range(32):
for k in range(256):
Z[i, j] += weights[i, k] * input[k, j]This computation reveals several critical dataflow challenges. The first challenge is the number of memory accesses required. For each output \(Z_{ij}\), the computation must fetch an entire row of weights from the weight matrix and a full column of activations from the input matrix. Since the weight matrix contains 512 rows and the input matrix contains 32 columns, this results in repeated memory accesses that place a heavy burden on memory bandwidth.
The second challenge comes from weight reuse. The same weights are applied to multiple inputs, meaning that an ideal mapping strategy should maximize weight locality to avoid redundant memory fetches. Without proper reuse, the accelerator would waste bandwidth loading the same weights multiple times (Chen et al. 2018).
The third challenge involves the accumulation of intermediate results. Since each element in \(Z_{ij}\) requires contributions from 256 different weight-input pairs, partial sums must be stored and retrieved before the final value is computed. If these intermediate values are stored inefficiently, the system will require frequent memory accesses, further increasing bandwidth demands.
One way to mitigate these challenges is to use SIMD and SIMT execution models, which allow multiple values to be fetched in parallel. However, even with these optimizations, data movement remains a bottleneck. The primary bottleneck is not retrieval speed alone, but transfer frequency and placement within the memory hierarchy (Han et al. 2016).
Han, Song, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz, and William J. Dally. 2016. “EIE: Efficient Inference Engine on Compressed Deep Neural Network.” 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), 243–54. https://doi.org/10.1109/isca.2016.30.
Because moving data dominates the energy budget that figure 9 charts, the single most important goal of an accelerator is to minimize memory access. Dataflow strategies achieve this by maximizing data reuse. The central decision is which data is most valuable to keep local. Accelerators answer that decision by determining which data remains fixed in memory and which data streams dynamically: weight-stationary keeps model parameters local, input-stationary maintains activation data, and output-stationary preserves intermediate results. Each approach trades off different memory access patterns to maximize data reuse and minimize the energy-intensive transfers that constitute the primary bottleneck in AI acceleration.
Weight stationary
The weight stationary strategy keeps weights fixed in local memory, while input activations and partial sums are streamed through the system. Weight stationary approaches prove particularly beneficial in CNNs and matrix multiplications, where the same set of weights is applied across multiple inputs. By ensuring weights remain stationary, this method reduces redundant memory fetches, which helps alleviate bandwidth bottlenecks and improves energy efficiency.
A key advantage of weight stationary is that it maximizes weight reuse, reducing the frequency of memory accesses to external storage. Since weight parameters are often shared across multiple computations, keeping them in local memory eliminates unnecessary data movement, lowering the overall energy cost of computation. This makes it particularly effective for architectures where weights represent the dominant memory overhead, such as systolic arrays and custom accelerators designed for machine learning. Listing 16 demonstrates how Weight Stationary execution keeps weights fixed in local memory while streaming inputs and accumulating partial sums.
## Weight Stationary Matrix Multiplication
## - Weights remain fixed in local memory
## - Input activations stream through
## - Partial sums accumulate for final output
for weight_block in weights: # Load and keep weights stationary
load_to_local(weight_block) # Fixed in local storage
for input_block in inputs: # Stream inputs dynamically
for output_block in outputs: # Compute results
output_block += compute(weight_block, input_block)
# Reuse weights across inputsIn weight stationary execution, weights are loaded once into local memory and remain fixed throughout the computation while inputs stream dynamically, reducing redundant memory accesses. Partial sums accumulate efficiently, minimizing unnecessary data movement. Because weights need not be reloaded for each new computation, bandwidth requirements drop significantly, making this dataflow highly effective for workloads with heavy weight reuse patterns such as CNNs and matrix multiplications.
However, while this strategy reduces weight-related memory traffic, it introduces trade-offs in input and output movement. Since inputs must be streamed dynamically while weights remain fixed, the efficiency of this approach depends on how well input activations can be delivered to the computational units without causing stalls. Partial sums, which represent intermediate results, must also be carefully accumulated to avoid excessive memory traffic. The total performance gain depends on the size of available on-chip memory, as storing larger weight matrices locally can become a constraint in models with millions or billions of parameters.
The weight stationary strategy is well-suited for workloads where weights exhibit high reuse and memory bandwidth is a limiting factor. It is commonly employed in CNNs, systolic arrays, and matrix multiplication kernels, where structured weight reuse leads to measurable performance improvements. However, for models where input or output reuse is more critical, alternative dataflow strategies, such as output stationary or input stationary, may provide better trade-offs.
Output stationary
Weight stationary keeps weights local and streams inputs through the system. The dominant cost shifts, however, when the bottleneck is not weight loading but the frequent writes of partial sums. In fully connected layers and transformer attention mechanisms, each output element accumulates contributions from hundreds or thousands of weight-input pairs. Writing those intermediate partial sums to external memory after every accumulation step would create a write-bandwidth bottleneck far more severe than the read overhead that weight stationary addresses. The output stationary strategy inverts the priority: it keeps partial sums fixed in local memory while streaming both weights and input activations through the system, so that each output element is written to external memory only once, after all its contributions have been accumulated (Chen et al. 2016).
Chen, Yu-Hsin, Joel S. Emer, and Vivienne Sze. 2016. “Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks.” 2016 ACM/IEEE 43rd Annual International Symposium on Computer Architecture (ISCA), 367–79. https://doi.org/10.1109/ISCA.2016.40.
Listing 17 demonstrates how accumulating partial sums locally minimizes memory writes and enhances efficiency during matrix multiplication. In this implementation, the accumulator buffer stays in local registers or scratchpad throughout the inner loop; weights and inputs stream in, contribute to the running sum, and are discarded. The final result is written out only once per output element, eliminating the repeated write traffic that would otherwise dominate bandwidth.
This approach aligns naturally with systolic arrays, where computation progresses through a grid of processing elements and partial sums can flow along one axis without leaving the chip. The trade-off is that both weights and activations must now be streamed dynamically, so the system must sustain high read bandwidth for two data streams simultaneously. Parallel implementations also require careful synchronization when multiple PEs contribute to the same output element. Output stationary is therefore most effective for workloads where accumulation dominates, such as fully connected layers and attention mechanisms, but less suitable when input reuse is the critical bottleneck.
## - Partial sums remain in local memory
## - Weights and input activations stream through dynamically
## - Final outputs are written only once
for output_block in outputs: # Keep partial sums stationary
accumulator = 0 # Initialize accumulation buffer
for weight_block, input_block in zip(weights, inputs):
accumulator += compute(weight_block, input_block)
# Accumulate partial sums
store_output(accumulator) # Single write to memoryInput stationary
The two strategies examined so far each fix a different operand in local memory: weight stationary fixes weights to reduce read bandwidth for parameters, and output stationary fixes partial sums to reduce write bandwidth for accumulations. The third strategy completes the picture by fixing the remaining operand: input activations. Within a transformer layer, one activation can feed multiple projections or attention heads; the next layer receives a newly computed representation rather than the same token activation. When activation reuse is the dominant memory cost, keeping inputs stationary and streaming weights through the system can reduce energy and bandwidth. Listing 18 illustrates this approach.
## - Input activations remain in local memory
## - Weights stream through dynamically
## - Partial sums accumulate and are written out
for input_block in inputs: # Keep input activations stationary
load_to_local(input_block) # Fixed in local storage
for weight_block in weights: # Stream weights dynamically
for output_block in outputs: # Compute results
output_block += compute(weight_block, input_block)
# Reuse inputs across weightsHere, input activations are loaded once and held fixed while weights stream through. Partial sums accumulate and are eventually written out, but unlike output stationary, the accumulation buffer is not the primary beneficiary of locality; instead, the input data is.
The trade-off mirrors the other two strategies: weights must now be streamed dynamically, so the system needs sustained read bandwidth for the weight stream, and partial sums require buffering before write-back. Input stationary is most effective where an activation tile feeds multiple projections, heads, gates, or other computations before eviction.
Taken together, the three dataflow strategies provide a decision rule rather than a hierarchy of quality. Weight stationary minimizes read traffic for parameters and suits CNNs with small, heavily reused filters. Output stationary minimizes write traffic for accumulations and suits fully connected layers with high fan-in. Input stationary minimizes read traffic for activations and suits transformers and batch processing with high activation reuse. No single strategy dominates; the optimal choice depends on which data element has the highest reuse ratio relative to its size, a determination that the compiler and hardware designer must make based on the specific workload and memory hierarchy. Convolution-specific designs add a fourth label, row-stationary (as in Eyeriss), but it is not a separate primitive: it is a hybrid that keeps rows of inputs and partial sums local when that mapping yields higher reuse than fixing any single operand (section 1.4.6.2).
Memory-efficient tensor layouts
The dataflow strategies in section 1.7.1.1 determine which data stays close to compute; tensor layouts determine whether that data can be accessed efficiently once it arrives. A perfectly chosen weight-stationary dataflow still suffers if weights are stored in a format that causes scattered memory accesses. Tensor layout is therefore a kernel contract: the physical arrangement of multidimensional data must match the access pattern expected by the selected hardware path, or the accelerator pays in memory stalls, inefficient cache usage, and increased data movement.
In AI accelerators, tensor layout optimization is particularly important because data is frequently accessed in patterns dictated by the underlying hardware architecture. Choosing the right layout ensures that memory accesses align with hardware-friendly access patterns, minimizing overhead from costly memory transactions (NVIDIA Corporation 2021).
While developers can sometimes manually specify tensor layouts, the choice is often determined automatically by machine learning frameworks such as TensorFlow, PyTorch, and JAX, by compilers, or by AI accelerator runtimes. Low-level optimization tools such as cuDNN (for NVIDIA GPUs), XLA (for TensorFlow graphs), and MLIR-based compiler stacks may impose or transform tensor layouts as they lower operations to backend-specific kernels (NVIDIA Corporation 2021; Google 2025; Lattner et al. 2020). In high-level frameworks, layout transformations are typically applied transparently, but developers working with custom kernels or low-level libraries such as CUDA, Metal, or OpenCL may have direct control over tensor format selection.
For example, PyTorch exposes view operations such as tensor.permute() and uses tensor.contiguous() to materialize contiguous storage when a downstream kernel requires it (PyTorch Contributors 2026). TensorFlow recommends channels-last (NHWC) for convolutional models on NVIDIA GPUs and warns that NCHW can introduce automatic transpose overhead (TensorFlow Developers 2024). Hardware-aware libraries such as cuDNN for GPUs and oneDNN for CPUs use specific memory layouts to maximize cache locality and SIMD efficiency. The practical rule is to treat layout as part of the selected backend path: the fastest tensor format is the one that avoids conversion overhead and makes the kernel’s memory accesses contiguous.
PyTorch Contributors. 2026. Tensor Views.
TensorFlow Developers. 2024. Optimize TensorFlow Performance Using the Profiler.
Channels-last layout
In an NHWC channels-last layout, the channel index varies fastest in a contiguous tensor, so the channel values for each pixel are adjacent. NHWC describes logical dimension order, not whether the underlying linear storage is row-major; both NHWC and NCHW tensors can use row-major linearization of their respective dimensions.
To understand channels-last layout, consider a single RGB image represented as a tensor of shape (Height, Width, Channels). If the image has a size of \(3{\times}3\) pixels with 3 channels (RGB), the corresponding tensor is structured as (3, 3, 3). The values are stored in memory as follows: \[\begin{gather*} I(0,0,0), I(0,0,1), I(0,0,2), I(0,1,0), I(0,1,1), \\ I(0,1,2), I(0,2,0), I(0,2,1), I(0,2,2), \ldots \end{gather*}\]
The channels for each pixel are adjacent, and pixels advance across width before height. This ordering can support sequential access when a kernel vectorizes over adjacent channel values.
The benefit of channels-last storage depends on the operator and backend. CPU and accelerator kernels that vectorize over channels can use the contiguous channel dimension, while other kernels may prefer channels-first or blocked internal layouts.
However, layout choice becomes subtle for convolutions because logical dimension order and physical memory format are not the same thing. A tensor may be described as NHWC or NCHW, but the backend ultimately cares whether the memory addresses consumed by a kernel are contiguous, aligned, and coalesced for the specific operator and precision mode.
TensorFlow commonly uses NHWC conventions, while PyTorch commonly exposes NCHW tensors with a separate channels-last memory format option. When targeting GPUs, frameworks and libraries may insert, propagate, or internally perform layout transformations to match the fastest kernel path.
Channels-first layout
In a contiguous NCHW channels-first layout, the width index varies fastest and all spatial values for a channel are stored together before the next channel. GPUs process data in parallel across threads, and when threads access consecutive memory addresses, the hardware can combine these requests into a single efficient transaction (memory coalescing). Historically, many GPU convolution paths used NCHW effectively, while modern Tensor Core convolution and fusion paths often prefer NHWC or channels-last physical layouts because those layouts align better with vectorized tensor-core kernels.
To understand channels-first layout, consider the same RGB image tensor with dimensions reordered as (Channels, Height, Width) = (3, 3, 3). Retaining \(I(h,w,c)\) as semantic coordinate notation, the data are stored in channels-first order as follows: \[\begin{gather*} I(0,0,0), I(0,1,0), I(0,2,0), I(1,0,0), I(1,1,0), I(1,2,0), \ldots, \\ I(0,0,1), I(0,1,1), I(0,2,1), \ldots, I(0,0,2), I(0,1,2), I(0,2,2), \ldots \end{gather*}\]
In this format, all red channel values for the entire image are stored first, followed by all green values, and then all blue values. This ordering can allow some hardware accelerators to efficiently load and process data across channels in parallel, which is important for convolution operations and SIMD execution models (Chetlur et al. 2014).
Chetlur, Sharan, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer. 2014. “cuDNN: Efficient Primitives for Deep Learning.” arXiv Preprint arXiv:1410.0759.
The advantage of a given layout becomes clear only relative to a specific backend. Convolutional layers process images by applying a shared set of filters across all channels. Depending on the kernel implementation, NCHW, NHWC, or a blocked internal layout may minimize scattered memory fetches, reduce memory latency, and improve data locality for the lowered matrix multiplications that implement convolution.
Because GPUs and TPUs rely on memory coalescing,34 a technique in which consecutive threads fetch contiguous memory addresses, the best layout is the one that makes the kernel’s actual thread-access pattern contiguous. For example, in NVIDIA GPU convolution paths, cuDNN may use or internally convert to NHWC/channels-last for Tensor Core kernels, while other kernels may still perform well with NCHW. The rule is backend- and operator-dependent rather than a universal CPU=NHWC, GPU=NCHW split.
34 Memory coalescing: The GPU hardware mechanism that fuses memory requests from threads in a warp into a single transaction when those threads access contiguous memory. Tensor layout affects coalescing, but the sign of the effect depends on the kernel: NCHW can be efficient for some convolution implementations, while NHWC/channels-last is often preferred for modern Tensor Core convolution and fusion paths. Poor layout choices can still create multi-fold performance gaps, but the correct fix is to match the layout to the backend rather than memorize one universal format.
Despite its advantages for some accelerator kernels, channels-first layout can introduce inefficiencies in kernels optimized for channels-last access. The most efficient layout depends on the operation, framework convention, and library implementation.
Modern AI frameworks and compilers often transform tensor layouts dynamically depending on the execution environment, but this is not guaranteed for every model or operation.35 TensorFlow, XLA, cuDNN, and TensorRT may insert or choose layout conversions internally; PyTorch exposes explicit channels-last conversion and propagation paths. Developers still need to profile layout choices when convolution performance is material.
35 NHWC vs. NCHW: NHWC lists dimensions as batch, height, width, channel; NCHW lists batch, channel, height, width. Physical memory format determines whether adjacent threads read adjacent addresses, so the performance effect is backend-specific. Layout-to-hardware mismatch is not a micro-optimization; it can create multi-fold performance gaps, especially when a conversion prevents Tensor Core convolution or fusion paths from being used.
Comparing channels-last and channels-first layouts
Both channels-last (NHWC) and channels-first (NCHW) layouts serve distinct purposes in machine learning workloads, with their efficiency largely determined by the hardware architecture, memory access patterns, and computational requirements. The choice of layout directly influences cache utilization, memory bandwidth efficiency, and processing throughput. Table 18 contrasts the performance trade-offs and hardware compatibility between these two approaches.
| Feature | Channels-Last (NHWC) | Channels-First (NCHW) |
|---|---|---|
| Memory storage order | Pixels are stored row-by-row, channel interleaved | All values for a given channel are stored together first |
| Best for | CPU loops, element-wise operations, many channels-last kernels | Many legacy GPU convolution paths and channel-first model code |
| Cache efficiency | High cache locality for sequential row/channel-last access | Can improve coalescing for channel-first kernels |
| Convolution performance | Often preferred by modern Tensor Core convolution/fusion paths | Efficient for many cuDNN and framework kernels |
| Memory fetching | Good when kernels vectorize over adjacent channel data | Good when kernels process channel-major tiles |
| Default in frameworks | Common TensorFlow convention; PyTorch channels-last option | Common PyTorch tensor shape convention |
The decision to use channels-last (NHWC) or channels-first (NCHW) layouts is not always made manually by developers. Instead, machine learning frameworks and AI compilers often determine the optimal layout based on the target hardware and operation type.
In practice, modern AI compilers such as TensorFlow’s XLA, cuDNN, TensorRT, and PyTorch compilation paths may perform layout transformations or propagate layout metadata. The result can be high throughput without manual tensor rewrites, but performance-sensitive deployments should still profile both layout and conversion overhead on the target hardware.
Kernel fusion
One of the most impactful optimization techniques in AI acceleration involves reducing the overhead of intermediate data movement between operations. Kernel fusion36 transforms multiple separate computations into unified operations, dramatically improving memory efficiency and execution performance. The memory bottlenecks created by intermediate writes motivate kernel fusion, which eliminates these inefficiencies.
36 Kernel fusion: Separate GPU kernels expose intermediate results through device memory, although caches can absorb some traffic. A fused kernel can keep intermediates in registers or local storage and avoid corresponding external write/read cycles. The resulting speedup depends on cache behavior, occupancy, launch overhead, and whether memory traffic was the bottleneck; it is not necessarily proportional to the byte reduction.
Intermediate memory write
AI model performance is often constrained by memory bandwidth and intermediate memory writes rather than pure arithmetic operations. Every time an operation produces an intermediate result that must be written to memory and later read back, execution stalls from the data movement overhead.
Kernel fusion represents the critical bridge between the software optimization techniques introduced in Operator fusion and the memory bandwidth constraints analyzed in section 1.4.1. Many AI workloads introduce unnecessary intermediate memory writes, increasing memory bandwidth consumption and reducing execution efficiency (NVIDIA Corporation 2017).
NVIDIA Corporation. 2017. NVIDIA Tesla V100 GPU Architecture. NVIDIA Whitepaper.
Listing 19 reveals how each operation becomes a separate kernel in a naïve execution model, forcing intermediate results to be written to memory and then read back for the next operation. The first two operations produce intermediate tensors that a naïve multi-kernel execution exposes through device memory for the next operation. On large tensors, this movement can outweigh the arithmetic cost (Jia et al. 2019). Table 19 illustrates the live-tensor footprint when buffers are not reused.
import torch
import torch.nn.functional as F
## Input tensor
X = torch.randn(1024, 1024).cuda()
running_mean = torch.zeros(1024, device=X.device)
running_var = torch.ones(1024, device=X.device)
## Step-by-step execution (naïve approach)
X1 = torch.relu(X) # Intermediate tensor stored in memory
X2 = F.batch_norm(
X1, running_mean, running_var, training=False
) # Frozen inference BatchNorm
Y = 2.0 * X2 + 1.0 # Final resultKernel fusion for memory efficiency
The two intermediate tensors consume memory capacity when they remain live and create external traffic when separate kernels materialize them. Kernel fusion can eliminate those intermediate writes and reloads (Jia et al. 2019) by propagating values through registers or local memory within one kernel.
| Tensor | Size (MB) for \(1024{\times}1024\) Tensor |
|---|---|
| X | 4.2 MB |
| X’ | 4.2 MB |
| X’’ | 4.2 MB |
| Y | 4.2 MB |
| Total Memory | 16.8 MB |
A machine learning inference sequence might apply ReLU, frozen batch normalization, and then an affine scaling with scale \(\alpha\) and offset \(\beta\). In a naïve implementation, each separate kernel generates an intermediate tensor that is written to memory and read back by the next kernel: \[ \begin{aligned} \mathbf{X}' &= \text{ReLU}(\mathbf{X}) \\ \mathbf{X}'' &= \text{BatchNorm}(\mathbf{X}') \\ \mathbf{Y} &= \alpha \cdot \mathbf{X}'' + \beta \end{aligned} \]
With kernel fusion, these operations are combined into a single computation step, allowing the entire transformation to occur without generating unnecessary intermediate tensors: \[ \mathbf{Y} = \alpha \cdot \text{BatchNorm}\big(\text{ReLU}(\mathbf{X})\big) + \beta \]
Table 20 highlights the impact of operation fusion on memory efficiency. By keeping intermediate results in registers or local memory rather than writing them to main memory, fusion reduces external traffic. For three separate pointwise inference kernels, the idealized naïve path reads and writes one tensor per operation, whereas the fused path reads the input once and writes the output once.
| Execution Model | External tensor payloads | Traffic (MB) |
|---|---|---|
| Naïve execution | 3 reads + 3 writes | 25.2 MB |
| Fused execution | 1 read + 1 write | 8.4 MB |
Performance benefits and constraints
Kernel fusion brings several key advantages that enhance memory efficiency and computation throughput. By reducing memory accesses, fused kernels ensure that intermediate values stay within registers instead of being repeatedly written to and read from memory. This significantly lowers memory traffic, which is one of the primary bottlenecks in machine learning workloads. GPUs and TPUs, in particular, benefit from kernel fusion because high-bandwidth memory is a scarce resource, and reducing memory transactions leads to better utilization of compute units (NVIDIA Corporation 2020a).
However, not all operations can be fused arbitrarily. Element-wise operations and frozen inference normalization are strong candidates because their computations do not require batch-wide reductions. Training-mode batch normalization computes batch statistics and is not purely element-wise. Matrix multiplications and convolutions constrain fusion because they involve reductions and large data movement; they are often fused with element-wise epilogues such as bias or activation, but cannot be freely fused with unrelated global operations.
Another major consideration is register pressure. Fusing multiple operations means all temporary values must be kept in registers rather than memory. While this eliminates redundant memory writes, it also increases register demand. If a fused kernel exceeds the available registers per thread, the compiler can spill excess values to per-thread local memory, which is backed by device memory and may be cached, introducing additional latency and potentially negating the benefits of fusion. On GPUs, where thread occupancy (the number of threads that can run in parallel) is limited by available registers, excessive fusion can reduce parallelism, leading to diminishing returns.
Different AI accelerators and compilers handle fusion in distinct ways. NVIDIA GPUs, for example, favor warp-level parallelism, where element-wise fusion is straightforward (NVIDIA Corporation 2020a). TPUs, on the other hand, prioritize systolic array execution for dense matrix operations (Jouppi et al. 2017). Compiler and inference stacks such as TVM, XLA, TensorRT, and MLIR apply graph rewrites, lowering passes, or engine-building heuristics to balance memory savings against execution constraints (Chen et al. 2018; Google 2025; NVIDIA 2024b; Lattner et al. 2020).
Despite its advantages, fusion is not always beneficial. Some AI frameworks allow developers to disable fusion selectively, especially when debugging performance issues or making frequent model modifications. The decision to fuse operations must consider trade-offs between memory efficiency, register usage, and hardware execution constraints to ensure that fusion leads to tangible performance improvements.
Checkpoint 1.3: Data movement and kernel fusion
The dataflow building blocks are now in place: data locality, tensor layout, and kernel fusion. These fusion decisions are ultimately about data locality, which ties together the chapter’s core data movement strategies:
Memory-efficient tiling strategies
While modern AI accelerators offer high computational throughput, their performance is often limited by memory bandwidth rather than raw processing power. If data cannot be supplied to processing units fast enough, execution stalls occur, leading to wasted cycles and inefficient hardware utilization.
Tiling37 mitigates this issue by restructuring computations into smaller, memory-friendly subproblems. When memory bandwidth cannot be increased directly, systems must reduce trips to main memory. Instead of processing entire matrices or tensors at once, which leads to excessive memory traffic, tiling partitions computations into smaller blocks (tiles) that fit within fast local memory (for example, caches, shared memory, or registers) (Lam et al. 1991).
37 Tiling (loop blocking): This restructuring partitions a computation into blocks sized for a fast local tier. A naive loop makes repeated references to the same elements; caches may capture some, while blocking deliberately concentrates reuse before eviction (Lam et al. 1991). This reduction in lower-tier traffic is a primary source of the gap between naive matrix multiplication and optimized general matrix multiplication (GEMM) routines.
Lam, Monica D., Edward E. Rothberg, and Michael E. Wolf. 1991. “The Cache Performance and Optimizations of Blocked Algorithms.” Proceedings of the Fourth International Conference on Architectural Support for Programming Languages and Operating Systems - ASPLOS-IV, 63–74. https://doi.org/10.1145/106972.106981.
Matrix multiplication, widely used in AI models, demonstrates inefficient memory access when implemented naively. Listing 20 shows how, without tiling, repeated memory accesses for the same data lead to unnecessary bandwidth consumption.
for i in range(N):
for j in range(N):
for k in range(N):
C[i, j] += A[i, k] * B[k, j] # Repeatedly fetching
# A[i, k] and B[k, j]The loop repeatedly references elements of \(\mathbf{A}\) and \(\mathbf{B}\). Some references can hit in cache, but large matrices and an unfavorable traversal order can still cause excessive lower-tier traffic.
Tiling addresses this problem by ensuring that smaller portions of matrices are loaded into fast memory, reused efficiently, and only written back to main memory when necessary. This technique is especially important in AI accelerators, where memory accesses dominate execution time. Figure 14 labels the product \(\mathbf{C} = \mathbf{A}\mathbf{B}\) by its three dimensions: \(M\) rows and \(K\) columns in \(\mathbf{A}\), \(K\) rows and \(N\) columns in \(\mathbf{B}\), and \(M{\times}N\) in the output \(\mathbf{C}\). Each highlighted tile is the working set that fits in fast memory at one moment: a green \(M_{\text{tile}}{\times}K_{\text{tile}}\) row band of \(\mathbf{A}\) multiplies a pink \(K_{\text{tile}}{\times}N_{\text{tile}}\) column band of \(\mathbf{B}\) to accumulate one blue \(M_{\text{tile}}{\times}N_{\text{tile}}\) output block (\(\text{Block}_{m,n}\)) of \(\mathbf{C}\). Processing all computations for one tile before moving to the next avoids repeatedly paying the DRAM access penalty.
\scalebox{0.65}{%
\begin{tikzpicture}[line join=round,font=\sffamily,x=1mm,y=1mm]
\tikzset{%
Line/.style={draw,line width=1.25pt,black,text=black},
LineT/.style={draw,line width=0.75pt,black,text=black},
}
%Bmatrix
\node[Line,rectangle,anchor=south west,
minimum width=66mm,minimum height=60mm](BM)at(0,0){};
\scoped[on background layer]
\node[LineT,rectangle,anchor=south west,fill=RedFill,
minimum width=18mm,minimum height=60mm](BM1)at(18mm,0){};
\node[LineT,rectangle,anchor=south west,fill=RedFill,
minimum width=18mm,minimum height=9mm](BM2)at(18mm,30mm){};
%
\draw[thick,decorate,decoration={brace, amplitude=7pt}]([yshift=2mm]BM.north west)--
([yshift=2mm]BM.north east)node[midway,above=9pt]{N};
\draw[thick,decorate,decoration={brace, amplitude=7pt,mirror}]([xshift=-2mm]BM.north west)--
([xshift=-2mm]BM.south west)node[midway,left=9pt]{K};
\draw[thick,decorate,decoration={brace, amplitude=5pt}]([xshift=2mm]BM2.north east)--
([xshift=2mm]BM2.south east)node[midway,right=6pt]{Ktile};
\draw[thick,decorate,decoration={brace, amplitude=5pt,mirror}]([yshift=-2mm,xshift=1mm]BM2.south west)--
([yshift=-2mm,xshift=-1mm]BM2.south east)node[midway,below=6pt]{Ntile};
\node[below left=2 of BM.north east]{B matrix};
%Cmatrix
\node[Line,rectangle,anchor=north west,
minimum width=66mm,minimum height=48mm](CM)at(0,-10){};
\node[LineT,rectangle,anchor=north west,
minimum width=18mm,minimum height=48mm](CM1)at(18mm,-10){};
\node[LineT,rectangle,anchor=south west,
minimum width=66mm,minimum height=15mm](CM2)at(CM.south west){};
\node[LineT,rectangle,anchor=south west,fill=BlueL,
minimum width=18mm,minimum height=14.8mm](CM3)at(CM1.south west){};
%
\draw[thick,decorate,decoration={brace, amplitude=5pt}]([yshift=-1mm,xshift=2mm]CM3.north east)--
([yshift=1mm,xshift=2mm]CM3.south east)node[midway,right=6pt]{Mtile};
\draw[thick,decorate,decoration={brace, amplitude=5pt,mirror}]([yshift=-2mm,xshift=1mm]CM3.south west)--
([yshift=-2mm,xshift=-1mm]CM3.south east)node[midway,below=6pt]{Ntile};
\node[above right=2 of CM3.north east]{Block \textsubscript{m,n}};
%Amatrix
\node[Line,rectangle,anchor=north east,
minimum width=60mm,minimum height=48mm](AM)at(-10,-10){};
\node[LineT,rectangle,anchor=south west,fill=GreenL!40,
minimum width=60mm,minimum height=15mm](AM1)at(AM.south west){};
\node[LineT,rectangle,anchor=south west,fill=GreenL,
minimum width=9mm,minimum height=15mm](AM2)at($(AM.south west)+(21mm,0)$){};
\node[below left=2 of CM.north east]{C matrix};
\node[Line,rectangle,anchor=north east,
minimum width=60mm,minimum height=48mm](AM)at(-10,-10){};
%
\draw[thick,decorate,decoration={brace, amplitude=7pt}]([yshift=2mm]AM.north west)--
([yshift=2mm]AM.north east)node[midway,above=9pt]{K};
\draw[thick,decorate,decoration={brace, amplitude=7pt,mirror}]([xshift=-2mm]AM.north west)--
([xshift=-2mm]AM.south west)node[midway,left=9pt]{M};
\draw[thick,decorate,decoration={brace, amplitude=5pt}]([yshift=-1mm,xshift=2mm]AM2.north east)--
([yshift=1mm,xshift=2mm]AM2.south east)node[midway,right=6pt]{Mtile};
\draw[thick,decorate,decoration={brace, amplitude=5pt,mirror}]([yshift=-2mm,xshift=1mm]AM2.south west)--
([yshift=-2mm,xshift=-1mm]AM2.south east)node[midway,below=6pt]{Ktile};
\node[below left=2 of AM.north east]{A matrix};
\end{tikzpicture}}Tiling fundamentals
Tiling divides a computation into smaller tiles that fit within available fast memory rather than operating on an entire data structure at once. This structure maximizes data reuse, reducing redundant memory accesses and improving efficiency.
Consider matrix multiplication, a key operation in machine learning workloads. The operation computes \(\mathbf{C} = \mathbf{A} \times \mathbf{B}\) where each element \(C_{ij} = \sum_{k} A_{ik} \times B_{kj}\). Listing 20 exposes the core problem: every iteration of the innermost loop fetches elements from matrices \(\mathbf{A}\) and \(\mathbf{B}\) from memory, performs a multiplication, and updates matrix \(\mathbf{C}\). Because matrices are large, the processor repeatedly reloads the same values from memory, even though they were just used in previous computations.
This data movement overhead is expensive: DRAM access has much higher latency and energy cost than access to on-chip cache or registers (Horowitz 2014; Sze et al. 2017). The solution is tiling.
Performance benefits of tiling
Instead of computing one element at a time and constantly moving data in and out of slow memory, tiling processes submatrices (tiles) at a time, keeping frequently used values in fast memory. The idea is to divide the matrices into smaller blocks that fit within the processor’s cache or shared memory, ensuring that once a block is loaded, it is reused multiple times before moving to the next one. Listing 21 demonstrates cache-friendly loop blocking: the loop bounds partition the matrices into tiles, and the hardware cache hierarchy keeps recently used tile data close to the compute units when access order has locality.
TILE_SIZE = 32 # Choose a tile size based on hardware constraints
# Cache blocking: partition data via loop bounds.
# Loads are implicit through the hardware cache hierarchy.
for i in range(0, N, TILE_SIZE):
for j in range(0, N, TILE_SIZE):
for k in range(0, N, TILE_SIZE):
# Each tile computed independently
for ii in range(i, min(i + TILE_SIZE, N)):
for jj in range(j, min(j + TILE_SIZE, N)):
for kk in range(k, min(k + TILE_SIZE, N)):
C[ii, jj] += A[ii, kk] * B[kk, jj]This restructuring significantly improves performance through three reinforcing effects. Memory reuse improves because the approach visits a small tile repeatedly while it is likely to remain in cache before moving on to the next tile, rather than fetching elements from \(\mathbf{A}\) and \(\mathbf{B}\) repeatedly from slow memory, which minimizes redundant memory accesses. Memory bandwidth usage drops as a direct consequence: since each tile is used multiple times before being evicted, most required data is available in L1/L2 cache or shared memory rather than DRAM, so traffic falls and execution speeds up. Compute efficiency rises in turn, because processors spend less time waiting for data and more time performing useful work; in architectures like GPUs and TPUs, where thousands of parallel processing units operate simultaneously, tiling keeps data read and processed in a structured manner that avoids unnecessary stalls.
This technique is particularly effective in AI accelerators, where machine learning workloads consist of large matrix multiplications and tensor transformations. Without tiling, these workloads quickly become memory bound, meaning performance is constrained by how fast data can be retrieved rather than by the raw computational power of the processor.
Tiling methods
While the general principle of tiling remains the same, which involves partitioning large computations into smaller subproblems to improve memory reuse, there are different ways to apply tiling based on the structure of the computation and hardware constraints. The two primary tiling strategies are spatial tiling and temporal tiling. These strategies optimize different aspects of computation and memory access, and in practice, they are often combined to achieve the best performance.
Spatial tiling partitions data structures into smaller blocks that fit within fast memory. The tiled matrix multiplication in listing 21 demonstrates cache-friendly loop blocking: the code does not issue explicit scratchpad loads, but the tile-shaped access pattern lets hardware caches reuse nearby values before they are evicted. This strategy is particularly beneficial for large tensors that exceed fast memory capacity—by breaking computations into smaller tiles, data movement between memory levels is minimized, keeping operations localized within cache hierarchies.
Temporal tiling complements spatial tiling by explicitly staging data in shared memory or registers and reorganizing the computation order around that staged data. Many ML workloads access the same data repeatedly across iterations—without temporal tiling, this results in redundant memory fetches. Temporal tiling restructures the computation to ensure that frequently used data stays in fast memory for as long as possible before the next computation begins.
A classic example where temporal tiling is beneficial is convolutional operations, where the same set of weights is applied to multiple input regions. Without loop blocking, these weights might be loaded from memory multiple times for each computation. With temporal tiling, the computation is reordered so that the weights remain in fast memory across multiple inputs, reducing unnecessary memory fetches and improving overall efficiency. Listing 22 illustrates explicit tile staging: the code loads blocks of \(\mathbf{A}\) and \(\mathbf{B}\) into temporary fast storage, then reuses them across multiple inner-loop operations.
# Explicit tile staging: load data into fast
# temporary storage before the inner loops.
for i in range(0, N, TILE_SIZE):
for j in range(0, N, TILE_SIZE):
for k in range(0, N, TILE_SIZE):
# Pseudocode: map these tile objects to fast memory
A_tile = A[i : i + TILE_SIZE, k : k + TILE_SIZE]
B_tile = B[k : k + TILE_SIZE, j : j + TILE_SIZE]
# Reuse loaded tiles for all inner iterations
for ii in range(A_tile.shape[0]):
for jj in range(B_tile.shape[1]):
for kk in range(A_tile.shape[1]):
C[i + ii, j + jj] += (
A_tile[ii, kk] * B_tile[kk, jj]
)Explicit tile staging improves performance when the backend places the temporary tiles in fast memory and reuses them before eviction. The pseudocode includes shortened boundary tiles when \(N\) is not divisible by TILE_SIZE.
This technique is particularly useful in workloads where certain values are used repeatedly, such as convolutions, recurrent neural networks (RNNs), and self-attention mechanisms in transformers. By applying loop blocking, AI accelerators can significantly reduce memory stalls and improve execution throughput.
Tiling challenges and trade-offs
Tiling improves performance only when the tile matches the locality budget of the hardware. If the tile is too small, memory fetches still dominate execution time because reuse is too limited. If the tile is too large, it exceeds fast memory and causes cache thrashing or scratchpad spills. Selecting the right tile size therefore directly determines computational efficiency and memory bandwidth usage.
The tile choice also controls load balance. In architectures such as GPUs and TPUs, computations execute in parallel across thousands of processing units. If tiles are not evenly distributed, some units remain idle while others are overloaded, leading to suboptimal utilization of computational resources. Effective tile scheduling keeps parallel execution balanced and efficient.
Data movement remains the limiting cost even after tiling. Although tiling reduces the number of slow memory accesses, transferring tiles between hierarchy levels still incurs latency and energy cost, especially when data falls from cache or scratchpad back to DRAM. Efficient memory prefetching and scheduling strategies minimize this residual movement and ensure that data is available when needed.
Hybrid tiling combines spatial and temporal strategies when neither dimension alone captures the workload’s reuse pattern. Some AI accelerators use spatial tiling for matrix multiplications while employing temporal tiling for weight reuse in convolutional layers, dynamically adjusting tile sizes or reordering computations based on real-time execution conditions.
Register blocking, double buffering, and hierarchical tiling extend the same locality principle at smaller and larger memory tiers. AI compilers and runtime systems such as TensorFlow XLA, TVM, and MLIR automatically select these tiling strategies based on hardware constraints, enabling fine-tuned performance optimization without manual intervention. Table 21 provides a comparative overview of spatial, temporal, and hybrid tiling approaches, highlighting their respective benefits and trade-offs.
When machine learning models grow in size and complexity, tiling remains a critical tool for improving hardware efficiency, ensuring that AI accelerators operate near their practical potential. While manual tiling strategies can provide substantial benefits, compilers and hardware-aware optimization techniques further enhance performance by automatically selecting effective tiling strategies for a given workload.
| Aspect | Spatial Tiling (Data Tiling) | Temporal Tiling (Loop Blocking) | Hybrid tiling |
|---|---|---|---|
| Primary Goal | Reduce memory accesses by keeping data in fast memory longer | Increase data reuse across loop iterations | Adapt dynamically to workload constraints |
| Optimization Focus | Partitioning data structures into smaller, memory-friendly blocks | Reordering computations to maximize reuse before eviction | Balancing spatial and temporal reuse strategies |
| Memory usage | Improves cache locality and reduces DRAM access | Keeps frequently used data in fast memory for multiple iterations | Minimizes data movement while ensuring high reuse |
| Common use cases | Matrix multiplications, CNNs, self-attention in transformers | Convolutions, recurrent neural networks (RNNs), iterative computations | AI accelerators with hierarchical memory, mixed workloads |
| Performance gains | Reduced memory bandwidth requirements, better cache utilization | Lower memory fetch latency, improved data locality | Maximized efficiency across multiple hardware types |
| Challenges | Requires careful tile size selection, inefficient for workloads with minimal spatial reuse | Can increase register pressure, requires loop restructuring | Complexity in tuning tile size and execution order dynamically |
| Best when | Data is large and needs to be partitioned for efficient processing | The same data is accessed multiple times across iterations | Both data partitioning and iteration-based reuse are important |
Applying mapping strategies to neural networks
While these foundational mapping techniques apply broadly, their effectiveness varies based on the computational structure, data access patterns, and parallelization opportunities of different neural network architectures. Each architecture imposes distinct constraints on data movement, memory hierarchy, and computation scheduling, requiring tailored mapping strategies to optimize performance.
A structured approach to mapping is required to address the combinatorial explosion of choices that arise when assigning computations to AI accelerators. Rather than treating each model as a separate optimization problem, the same principles apply across different architectures; only their priority shifts based on workload characteristics. The goal is to systematically select and apply mapping strategies that maximize efficiency for different types of machine learning models.
These principles apply to three representative AI workloads, each characterized by distinct computational demands. CNNs benefit from spatial data reuse, making weight-stationary execution and tiling especially effective. Transformer behavior depends on phase and shape: training and prefill matrix multiplications can be compute bound, while low-batch decoding often depends on weight and KV-cache bandwidth. MLPs involve substantial matrix multiplication and benefit from structured tiling, optimized weight layouts, and memory-aware execution.
Despite their differences, each of these models follows a common set of mapping principles, with variations in how optimizations are prioritized. Table 22 summarizes the suitability of different optimization strategies for CNNs, transformers, and MLPs.
Convolutional neural networks (ResNet-50)
For ResNet-50 and similar CNNs, a key mapping opportunity is spatial weight reuse. A small filter is applied across many spatial locations, so its weights participate in many multiply-accumulate operations. Weight-stationary mappings can capture that reuse by retaining filter tiles locally, while row-stationary or output-stationary mappings may better balance activation and partial-sum traffic on other architectures. Favorable convolution shapes can achieve high arithmetic intensity, which makes them well suited to matrix and systolic execution paths. The model family does not determine one universal dataflow: layer shape, batch size, local-storage capacity, and the target’s supported kernels decide which reuse pattern is worth retaining.
| Optimization technique | CNNs | Transformers | MLPs | Rationale |
|---|---|---|---|---|
| Dataflow strategy | Weight, row, or output stationary | Phase- and kernel-dependent | Weight or output stationary | CNNs expose filter and spatial reuse; transformer attention and MLP blocks impose different traffic; dense MLPs reuse weights across a batch and accumulate outputs. |
| Memory-aware tensor layouts | Backend-dependent (often NCHW for cuDNN convolutions, channels-last on Tensor Cores) | Backend-dependent (row-major activations typical) | Row-major typical | Layout choice depends on backend kernel path and precision mode (see section 1.7.1.2); the entries name common defaults, not universal prescriptions. |
| Kernel Fusion | Convolution + Activation | Fused Attention | GEMM Fusion | CNNs optimize convolution+activation fusion; Transformers fuse attention mechanisms; MLPs benefit from fused matrix multiplications. |
| Tiling for Memory Efficiency | Spatial Tiling | Temporal Tiling | Blocked Tiling | CNNs tile along spatial dimensions; Transformers use loop blocking to improve sequence memory efficiency; MLPs use blocked tiling for large matrix multiplications. |
This spatial regularity also enables aggressive fusion and tiling. For inference with frozen BatchNorm statistics, compilers can fold or fuse convolution, normalization, and activation to avoid unnecessary intermediate writes. Training-mode BatchNorm requires reductions over batch statistics and does not have the same element-wise fusion contract. Spatial tiling partitions the feature map into subregions sized to fit within on-chip SRAM, so the fused kernel processes each tile from fast memory before moving to the next.
Transformer architectures (GPT-2/Llama)
Where CNNs are defined by weight reuse, transformers are defined by the memory pressure of the key-value (KV) cache. During attention computation, every query vector must access stored key and value pairs across the entire sequence length. As sequences grow, the full KV cache usually lives in HBM or device DRAM, while attention kernels tile active blocks through SRAM, registers, or shared memory. This access pattern motivates activation stationary execution: keep the currently used KV tiles close to compute while streaming queries through them, rather than repeatedly materializing large attention intermediates in external memory.
The memory traffic created by standard attention implementations explains why fused attention kernels, such as FlashAttention (Dao et al. 2022), can deliver large performance gains: by fusing the query-key dot product, softmax normalization, and value-weighted summation into a single kernel that tiles along the sequence dimension, these implementations avoid materializing the full attention matrix in main memory. This temporal tiling approach processes sequence blocks that fit within on-chip SRAM, substantially reducing HBM traffic while preserving the \(\mathcal{O}(S^2)\) attention computation. Transformer mapping depends on the phase and shape: training and prefill GEMMs can be compute-bound, while low-batch autoregressive decode is commonly constrained by weight and KV-cache bandwidth.
Multilayer perceptrons and DLRM
MLPs present the most straightforward mapping problem because their computation reduces largely to dense GEMM.38 Each fully connected layer multiplies an activation matrix by a weight matrix. The weight matrix is fixed across samples, so batching reuses each loaded weight across more activations and raises arithmetic intensity. A batch size of one provides little cross-sample weight reuse and often underutilizes wide matrix engines, while larger batches can move execution toward the compute-bound regime. General matrix multiply (GEMM) derives the scaling that governs this sensitivity.
38 GEMM: The operation \(\mathbf{C} = \alpha \mathbf{A}\mathbf{B} + \beta \mathbf{C}\) is the dense linear-algebra primitive that many deep-learning layers lower to. Optimized GEMM libraries such as cuBLAS and oneDNN use register blocking, vectorization, and hierarchical tiling to approach hardware limits on favorable shapes (NVIDIA 2024a; Intel Corporation 2021b). Modern AI accelerators are heavily specialized for GEMM-like tiles: Tensor Cores, systolic arrays, and matrix extensions all exist to accelerate this primitive, which is why GEMM performance is an important predictor of end-to-end throughput across architectures.
NVIDIA. 2024a. cuBLAS: CUDA Basic Linear Algebra Subprograms.
Intel Corporation. 2021b. oneDNN: Intel’s Deep Learning Neural Network Library.
Naumov, Maxim, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, et al. 2019. “Deep Learning Recommendation Model for Personalization and Recommendation Systems.” arXiv Preprint arXiv:1906.00091.
Because MLP layers are typically followed by activation functions and bias additions, GEMM fusion combines these steps into a single kernel, avoiding intermediate memory writes. Blocked tiling partitions the large matrix multiplications into sub-blocks sized for the accelerator’s shared memory, ensuring high cache utilization throughout computation. The simplicity of the MLP mapping, dominated by a single primitive with predictable access patterns, is precisely why hardware vendors optimize GEMM libraries so aggressively: gains in GEMM performance translate directly to MLP throughput. DLRM is not purely an MLP workload, however. Its large sparse embedding tables and feature-interaction stage can dominate capacity and memory traffic, so the dense mapping described here applies to its bottom and top MLP towers rather than to the entire model (Naumov et al. 2019).
Hybrid mapping strategies
The preceding architectural subsections treat each architecture in isolation, but real models rarely consist of a single layer type. A vision transformer, for example, combines a patch embedding stage, self-attention layers, and MLP blocks (Dosovitskiy et al. 2021). Those layers create different reuse patterns: the embedding stage can benefit from weight-stationary mapping, attention emphasizes activation movement and tiling, and MLP blocks demand blocked GEMM tiling and fusion. No single dataflow strategy is optimal across all these layers, so hardware mapping becomes hybrid and layer-specific.
Dosovitskiy, Alexey, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, et al. 2021. “An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.” International Conference on Learning Representations (ICLR).
Hybrid mapping addresses this heterogeneity by allowing the accelerator to switch strategies at layer boundaries. Each layer presents a different balance of compute intensity, data reuse, and memory access pattern, and the optimal mapping must shift accordingly (Sze et al. 2017). Rather than committing to one dataflow for the entire model, hybrid approaches select weight stationary execution for layers with high weight reuse, activation stationary execution for attention layers with large KV caches, and output stationary execution for layers where minimizing write traffic matters most.
Modern accelerators provide the architectural features needed to realize hybrid mapping in practice. TPU-style systolic arrays, NVIDIA GPUs, and tile-based accelerator designs expose different combinations of local memory, tensor layouts, fusion, and scheduling controls, allowing compilers and runtimes to choose layer-specific strategies rather than one global dataflow for the whole model (Jouppi et al. 2023; NVIDIA Corporation 2020a; Chen et al. 2018). These implementations require programmable memory hierarchies, efficient interconnects, and specialized execution pipelines, reinforcing the hardware-software co-design principle.
However, hybrid mapping remains a design-time optimization. In production workloads, execution conditions change dynamically due to varying input sizes, memory contention, and hardware resource availability. Machine learning compilers and runtime systems extend these static mapping choices by introducing dynamic scheduling, memory optimizations, and automatic tuning, ensuring that deep learning workloads operate efficiently across diverse accelerators and deployment environments.
Self-Check: Question
Why do modern deep learning libraries (e.g., cuDNN, TensorRT) strongly prefer the Channels-Last (NHWC) tensor layout over Channels-First (NCHW) when executing convolutions on NVIDIA Tensor Cores?
- NHWC eliminates the need for spatial convolutions by flattening images into 1D vectors
- NCHW requires floating-point numbers to be stored in big-endian byte order
- NHWC reduces model parameter count by sharing channel weights across batches
- NHWC places channel values for a given spatial location in contiguous memory, aligning with the packed vector/matrix multiply requirements of Tensor Cores
Explain the mechanism of vertical kernel fusion (e.g., fusing
Conv2D\(\to\)BatchNorm\(\to\)ReLU), and identify both its performance benefit and its primary architectural constraint.Place the following loop transformation steps in the logical order applied by an optimizing compiler when targeting a matrix multiplication dataflow to an accelerator:
- Loop Reordering: Permute loop indices to establish a specific stationary dataflow (e.g., Output-Stationary)
- Loop Unrolling: Fully or partially unroll the innermost loop to expose instruction-level parallelism and map to hardware registers
- Loop Tiling (Blocking): Partition global loop iterations into sub-tiles that fit into on-chip shared memory / SRAM
- Spatial Partitioning: Assign outer tile loops to physical hardware compute clusters (e.g., GPU thread blocks / SMs)
In a multi-head self-attention layer where a single input activation tensor is projected across multiple query, key, and value weight matrices (\(W_q, W_k, W_v\)), which stationary dataflow strategy provides the highest data reuse in local scratchpad memory?
- Input-Stationary (Activation-Stationary)
- Output-Stationary
- No Local Reuse (NLR)
- Weight-Stationary
True or False: Row-Stationary (RS) dataflow, as implemented in architectures like Eyeriss, keeps only the final output activation stationary in registers while streaming 2D convolutional filter rows and input rows from DRAM on every clock cycle.
Compiler Support
A single convolution can be implemented with dozens of valid tiling strategies, kernel variants, and memory layouts, most of which perform poorly on a given target. Machine learning compilers navigate this complexity by translating dataflow strategies into target-specific executable code. Compiling ResNet-50 for GPU inference exemplifies the process:
- Graph optimization fuses repeated Conv2D-BatchNorm-ReLU patterns into fewer kernels, eliminating intermediate writes that would otherwise consume bandwidth
- Kernel selection chooses Tensor Core implementations for compatible convolutions, exploiting the high arithmetic intensity calculated in the Roofline analysis
- Memory planning determines whether intermediate activations fit in accelerator memory and whether buffers can be reused safely
- Computation scheduling overlaps memory transfers with computation when dependencies allow, hiding part of the transfer latency
In this illustrative scenario, inference time drops from 47 ms to 8 ms, a 5.9× improvement without changing the model’s mathematical function. These numbers are not a ResNet-50 benchmark. They show how graph rewrites, kernel selection, memory planning, and scheduling can translate fusion (section 1.7.1.3) and tiling (section 1.7.1.4) into delivered performance (Chen et al. 2018; NVIDIA 2024b).
This process exemplifies the hardware-software co-design principle established in Acceleration Fundamentals. Machine learning compilers bridge high-level model representations and low-level execution by restructuring computations, selecting kernels, and planning tensor storage (Chen et al. 2018). They build on conventional compiler techniques while adding tensor- and graph-level transformations.
ML compiler design
Machine learning compilers specialize conventional compilation for programs expressed as tensor graphs and operator libraries. General-purpose compilers already optimize loops, vector instructions, parallel code, and memory access; ML compiler stacks add graph rewrites, tensor layouts, shape specialization, accelerator kernel selection, and execution planning (Li et al. 2021).
Li, Mingzhen, Yi Liu, Xiaoyan Liu, Qingxiao Sun, Xin You, Hailong Yang, Zhongzhi Luan, Lin Gan, Guangwen Yang, and Depei Qian. 2021. “The Deep Learning Compiler: A Comprehensive Survey.” IEEE Transactions on Parallel and Distributed Systems 32 (3): 708–27. https://doi.org/10.1109/tpds.2020.3030548.
Table 23 contrasts the emphasis of the two layers. The boundary is not absolute: an ML stack often lowers tensor programs into an LLVM, GPU, or vendor backend, so graph-level and instruction-level compilation cooperate.
| Aspect | Traditional compiler | Machine learning compiler |
|---|---|---|
| Input representation | Source or intermediate-representation program | Tensor graph or tensor-level intermediate representation |
| Execution Model | Control flow, loops, threads, vectors, and tasks | Tensor operators, kernels, graphs, and accelerator streams |
| Optimization priorities | Instruction selection, loop transforms, vectorization, registers | Graph rewrites, fusion, layouts, shapes, and kernel selection |
| Memory management | Objects, stack/heap, caches, locality, and prefetching | Tensor lifetimes, buffer reuse, layouts, tiling, and device transfers |
| Target Hardware | CPUs, GPUs, and other programmable targets | CPUs, GPUs, TPUs, and custom accelerators |
| Compilation output | Machine code or lower-level intermediate representation | Kernels plus a hardware-specific graph or execution plan |
The table explains why compiler configuration can change performance even when model code is unchanged. ML compilers own the hidden layer that maps graph-level tensor operations onto hardware-specific kernels, layouts, and schedules; when that mapping is poor, the model leaves arithmetic units idle or moves the same bytes repeatedly.
ML compilation pipeline
Machine learning models, as defined in modern frameworks, are initially represented in a high-level computation graph that describes operations on tensors. These representations must be lowered into executable code for target hardware such as CPUs, GPUs, TPUs, and custom AI chips. An ML compilation pipeline performs this transformation (Chen et al. 2018; Google 2025; Lattner et al. 2020).
A useful compilation model has five responsibilities, although real systems interleave them. Graph optimization restructures the computation. Kernel selection maps operations to target implementations. Memory planning assigns layouts, buffers, and lifetimes. Scheduling orders kernels and transfers subject to dependencies. Code generation then emits or packages the target-specific executable components.
At each stage, the compiler applies the optimizations developed in section 1.7: kernel fusion, tiling, data movement strategies, and computation placement. These optimizations are systematically incorporated into the final execution plan, which is why machine learning acceleration depends as much on compiler-driven software optimization as on hardware improvements.
Graph optimization
AI accelerators provide specialized hardware to speed up computation, but raw model representations are not inherently optimized for execution on these accelerators. Machine learning frameworks define models using high-level computation graphs, where nodes represent operations (such as convolutions, matrix multiplications, and activations), and edges define data dependencies. However, if executed as defined, these graphs often contain redundant operations, inefficient memory access patterns, and suboptimal execution sequences that can prevent the hardware from operating at peak efficiency.
For example, transformer self-attention can create large intermediate score and probability matrices. A naïve implementation that materializes and rereads those intermediates from high-bandwidth memory pays excessive memory traffic, while IO-aware attention kernels tile the computation through fast memory to avoid that traffic (Dao et al. 2022). Similarly, in a CNN, applying batch normalization and activation functions as separate operations after each convolution leads to unnecessary intermediate memory writes, increasing memory bandwidth usage. These inefficiencies are addressed during graph optimization, where the compiler restructures the computation graph to eliminate unnecessary operations and improve memory locality (Chen et al. 2018).
Jia, Zhihao, James Thomas, Todd Warszawski, Mingyu Gao, Matei Zaharia, and Alex Aiken. 2019. “Optimizing DNN Computation with Relaxed Graph Substitutions.” Proceedings of Machine Learning and Systems (MLSys).
Graph optimization transforms this high-level computation graph into an optimized execution plan before hardware mapping. Rather than requiring manual optimization, the compiler systematically applies transformations that improve data movement, reduce redundant computations, and restructure operations for efficient parallel execution (Chen et al. 2018; Jia et al. 2019). At this stage, the compiler works at a hardware-agnostic level, focusing on high-level restructuring before hardware-specific optimizations are applied in subsequent compilation phases (section 1.8.6.2).
Graph optimization first removes traffic that the high-level graph representation would otherwise create. Kernel fusion merges consecutive operations to eliminate unnecessary memory writes and reduce the number of kernel launches, which is particularly effective in convolutional neural networks where convolution, batch normalization, and activation functions appear in fixed sequences. Computation reordering adjusts execution order to improve data locality and parallel execution; in transformer models, this reordering enables reuse of cached key-value pairs rather than repeated memory reloads.
Redundant computation elimination serves the same goal from the compute side. By identifying and removing duplicate or unnecessary operations, the compiler avoids repeated work in models with residual connections where common subexpressions might otherwise be recomputed. Memory-aware dataflow adjustments then refine tensor layouts and optimize movement; for example, tiling matrix multiplications to meet the structural requirements of systolic arrays in TPUs aligns the graph with the accelerator’s strengths. Together, these techniques prepare the model for acceleration by minimizing overhead and balancing computation against memory resources.
Modern AI compilers implement these rewrites through automated pattern recognition and structured rules, but the core responsibilities are the same across compiler stacks: find fusible patterns, choose layouts that match the target memory hierarchy, and preserve the model’s mathematical meaning while exposing hardware-specific optimization opportunities. XLA, TVM, TensorRT, and MLIR are representative systems that emphasize different target constraints, from graph-level fusion to tensor-layout search and multi-stage lowering. The systems lesson is not the product list; it is that compiler restructuring turns a framework graph into an execution plan the accelerator can sustain. Without this restructuring, a large transformer model on an edge device may suffer excessive memory stalls; with it, reduced bandwidth consumption and latency can make real-time inference feasible on resource-constrained devices.
After graph-level rewrites expose the available operations, kernel selection determines which target-specific implementation should realize each one. Later memory and scheduling decisions may feed back into that choice.
Kernel selection
Kernel selection turns the optimized graph into a hardware contract. A kernel is a specialized implementation of a computational operation designed to run efficiently on a particular hardware architecture. Most accelerators, including GPUs, TPUs, and custom AI chips, provide multiple kernel implementations for the same operation, each optimized for different execution scenarios. Choosing the right kernel determines whether the accelerator maximizes computational throughput, avoids memory stalls, and keeps specialized processing elements busy (Chen et al. 2018; Zheng et al. 2020).
Kernel selection builds upon graph optimization, mapping the structured execution plan to the most efficient implementation available for each operation. Poor kernel choices can nullify the benefits of prior optimizations by introducing unnecessary computation overhead or memory bottlenecks (Chen et al. 2018).
Transformer matrix multiplications illustrate the choice. A CPU backend may select a vectorized library kernel, a GPU backend may use Tensor Cores when shape and precision permit, and a TPU backend maps compatible tiles to its matrix unit. An INT8 inference kernel is useful only when the model has been quantized and validated at that precision; lower precision changes both performance and numerical behavior.
In many cases, the compiler’s decision is which existing implementation to trust rather than whether to generate a kernel from scratch. cuDNN and cuBLAS offer optimized kernels for deep learning on NVIDIA GPUs, oneDNN provides optimized execution for Intel architectures, ACL (Arm Compute Library) targets Arm-based devices, and Eigen and BLIS provide efficient CPU-based implementations. These libraries encode hardware-specific knowledge so the compiler can choose a preoptimized kernel rather than reinventing an execution strategy for each platform.
AI compilers use heuristics,39 profiling, and cost models to decide among these options. The selection method depends on how much uncertainty the compiler can tolerate before execution begins.
39 Heuristic in kernel selection: A practical rule that chooses a promising implementation without exhaustively benchmarking the full candidate space. Tile sizes, layouts, precision modes, fusion choices, and shape constraints can create many legal GEMM variants. Heuristics reduce tuning cost but may miss a faster candidate, which is why autotuners such as AutoTVM profile selected options on the target hardware.
Rule-based selection applies predefined heuristics based on known hardware capabilities. For instance, XLA, the compiler used in TensorFlow, automatically selects Tensor Core-optimized kernels for NVIDIA GPUs when mixed-precision execution is enabled. These predefined rules allow fast, reliable decisions without extensive analysis.
Profile-guided selection pays more search cost to reduce uncertainty. TVM uses AutoTVM to benchmark kernel options empirically and tune execution strategies based on real execution times, so operations are assigned to implementations that perform well under actual deployment conditions.
Cost model-based selection estimates execution time and memory consumption before profiling every option. MLIR provides compiler infrastructure in which target-specific passes can implement this kind of selection (Lattner et al. 2020). A concrete compiler can model how candidate kernels interact with the accelerator’s compute units and memory hierarchy, then select a kernel that minimizes estimated execution cost.
Lattner, Chris, Mehdi Amini, Uday Bondhugula, Albert Cohen, Andy Davis, Jacques Pienaar, River Riddle, Tatiana Shpeisman, Nicolas Vasilache, and Oleksandr Zinenko. 2020. “MLIR: Scaling Compiler Infrastructure for Domain Specific Computation.” 2021 IEEE/ACM International Symposium on Code Generation and Optimization (CGO), 2–14. https://doi.org/10.1109/cgo51591.2021.9370308.
Precision-aware selection adds the numerical constraint to the same decision. Training workloads often prioritize FP32 or BF16 to maintain model accuracy, whereas inference workloads favor FP16 or INT8 to increase speed and reduce power consumption. For example, an NVIDIA GPU running inference with TensorRT can select among calibrated FP16 and INT8 engine profiles that were built for the model’s accuracy constraints and input shapes. This trade-off between precision and performance is a key aspect of kernel selection, especially in resource-constrained environments.
Some compilers extend selection into adaptive tuning, where execution strategies adjust to workload and resource conditions. AutoTVM in TVM measures kernel performance across workloads and refines execution strategies; TensorRT applies optimized engine profiles based on batch size, memory constraints, and supported precision; Google’s TPU compiler specializes execution plans for the target TPU topology and shape profile. The consequences of poor kernel selection are significant: a transformer model assigned a nontensor-core kernel for matrix multiplications may execute at only a fraction of possible performance, while a model designed for FP32 execution may lose accuracy if forced onto an INT8-optimized kernel. Kernel selection is therefore as much about numerical correctness as performance.
After an initial kernel selection, memory planning and scheduling determine buffer lifetimes, workspaces, launch order, and legal overlap. These decisions can also cause the compiler to revisit a kernel whose workspace or layout no longer fits the emerging plan.
Memory planning
The memory planning phase ensures that data is allocated and accessed in a way that minimizes memory bandwidth consumption, reduces latency, and maximizes cache efficiency (Roesch et al. 2018; Chen et al. 2018). Even with the most optimized execution plan, a model can still suffer from severe performance degradation if memory is not managed efficiently.
Machine learning workloads are memory-intensive, requiring frequent movement of large tensors between different levels of the memory hierarchy. The compiler must determine how tensors are stored, how they are accessed, and how intermediate results are handled to prevent memory from becoming the bottleneck.
The memory planning phase optimizes tensor layouts, memory access patterns, and buffer reuse to prevent unnecessary stalls and memory contention during execution. Tensors are arranged in formats that align with hardware access patterns, minimizing format conversions. Memory accesses are structured to reduce cache misses and stalls, lowering overall bandwidth consumption. Buffer reuse reduces redundant memory allocations by managing intermediate results so that completed buffers are reclaimed promptly. Together, these strategies ensure that data is efficiently placed and accessed, enhancing both computational performance and energy efficiency.
Balancing memory availability, reuse, and access efficiency across multiple hierarchy levels makes memory planning one of the most complex compiler problems. AI compilers use several strategies to manage memory effectively and prevent unnecessary data movement.
Tensor layout optimization determines how tensors should be arranged in memory to maximize locality and prevent unnecessary format conversions. As section 1.7.1.2 established, different hardware accelerators favor different physical layouts depending on the backend kernel and precision mode. NVIDIA’s cuDNN convolution path historically expected NCHW for many FP32 kernels, while the channels-last NHWC layout aligns with Tensor Core memory coalescing for FP16 and INT8 paths; TensorFlow/XLA may choose internal layouts during lowering for the target backend. Compiler and library stacks transform tensor layouts based on the kernel and precision selected for the target hardware, ensuring that memory accesses are aligned for maximum efficiency (NVIDIA Corporation 2021; Google 2025).
Google. 2025. XLA: Optimizing Compiler for Machine Learning.
Roesch, Jared, Steven Lyubomirsky, Logan Weber, Josh Pollock, Marisa Kirisame, Tianqi Chen, and Zachary Tatlock. 2018. “Relay: A New IR for Machine Learning Frameworks.” Proceedings of the 2nd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, 58–68. https://doi.org/10.1145/3211346.3211348.
Buffer allocation and reuse complements layout optimization: the compiler minimizes memory footprint by reusing intermediate storage whenever possible. Deep learning workloads generate many temporary tensors, such as activations and gradients, which can quickly overwhelm on-chip memory if not carefully managed. Instead of allocating new memory for each tensor, the compiler analyzes the computation graph to identify opportunities for buffer reuse, ensuring that intermediate values are stored and overwritten efficiently (Roesch et al. 2018).
Minimizing data movement between hierarchy levels is equally critical. AI accelerators typically have a mix of high-speed on-chip memory (such as caches or shared SRAM) and larger, but slower, external DRAM. If tensor data is repeatedly moved between these memory levels, the model may become memory bound, reducing computational efficiency. To prevent this, compilers use tiling strategies that break large computations into smaller, memory-friendly chunks, allowing execution to fit within fast, local memory and reducing the need for costly off-chip memory accesses. The consequences of neglecting memory planning are concrete: a CNN running on a GPU may achieve high computational efficiency in theory, but if its convolutional feature maps are stored in an incompatible layout that necessitates repeated format conversions, the resulting overhead can negate the gains from graph optimization and kernel selection entirely. With memory allocation determined, the compiler must next decide when and where each computation executes.
Computation scheduling
With graph optimization completed, kernels selected, and memory planning finalized, computation scheduling determines the execution order and resource assignment for each operation. This phase determines when and where each computation should be executed, ensuring that workloads are efficiently distributed across available processing elements while avoiding unnecessary stalls and resource contention (Zheng et al. 2020).
Without effective scheduling, massive parallelism goes to waste: computational units sit idle, memory bandwidth goes underutilized, and execution efficiency degrades. Computation scheduling keeps processing elements active, manages execution dependencies correctly, and distributes workloads across the hardware schedule space (Chen et al. 2018; Zheng et al. 2020).
The scheduling phase coordinates parallel execution, synchronization, and resource allocation. Task partitioning decomposes computations into units that can be distributed among multiple compute cores. Execution order optimization determines the sequence for launching operations, maximizing hardware performance while reducing stalls. Resource allocation and synchronization ensure that compute cores, memory bandwidth, and shared caches are used without contention.
Implementation in AI compilers
Scheduling strategies are highly dependent on the underlying hardware architecture, since different AI accelerators have unique execution models. AI compilers implement several strategies to optimize scheduling for efficient execution.
Task partitioning divides large computational graphs into smaller units that can execute in parallel. On GPUs, this typically means mapping matrix multiplications and convolutions to thousands of CUDA cores, while on TPUs, tasks are partitioned to fit within systolic arrays that operate on structured data flows (Norrie et al. 2021). In CPUs, partitioning is often focused on breaking computations into vectorized chunks that align with SIMD execution. In each case, the goal is to keep every core active throughout execution.
Norrie, Thomas, Nishant Patil, Doe Hyun Yoon, George Kurian, Sheng Li, James Laudon, Cliff Young, Norman Jouppi, and David Patterson. 2021. “The Design Process for Google’s Training Chips: TPUv2 and TPUv3.” IEEE Micro 41 (2): 56–63. https://doi.org/10.1109/mm.2021.3058217.
Beyond task partitioning, scheduling involves optimizing execution order to minimize dependencies and maximize throughput. Many AI models include operations that can be computed independently (for example, different batches in a batch processing pipeline) alongside operations that have strict dependencies (for example, recurrent layers in an RNN). AI compilers analyze these dependencies and attempt to rearrange execution where possible, reducing idle time and improving parallel efficiency. In transformer attention, IO-aware kernels make this scheduling problem concrete by loading blocks of queries, keys, and values into fast memory, using them while resident, and evicting them in an order that reduces high-bandwidth-memory traffic (Dao et al. 2022).
Resource allocation and synchronization determine how compute cores share memory and coordinate execution. Modern AI accelerators often support overlapping computation and data transfers, meaning that while one task executes, the next task can begin fetching its required data. Compilers take advantage of this by scheduling tasks in a way that hides memory latency, ensuring that execution remains compute bound rather than memory-bound (Chen et al. 2018). In production inference stacks, optimized runtimes and compiler-generated schedules coordinate kernel launch order, stream execution, and synchronization so the accelerator does not stall between dependent kernels (NVIDIA 2024b; Zheng et al. 2020). Poor scheduling decisions can negate the benefits of all prior compilation phases: a CNN with highly optimized kernels and efficient memory layouts will still suffer reduced throughput if compute units remain idle between kernel launches, and a transformer on a TPU may underperform if attention layers are not scheduled to overlap with memory transfers.
Chen, Tianqi, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Q. Yan, Haichen Shen, Meghan Cowan, et al. 2018. “TVM: An Automated End-to-End Optimizing Compiler for Deep Learning.” Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’18), 578–94.
NVIDIA. 2024b. NVIDIA TensorRT: Programmable Inference Accelerator.
Zheng, Lianmin, Chengfan Jia, Minmin Sun, Zhao Wu, Cody Hao Yu, Ameer Haj-Ali, Yida Wang, et al. 2020. “Ansor: Generating High-Performance Tensor Programs for Deep Learning.” Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 863–79.
Code generation
With scheduling complete, the final compilation stage translates this optimized execution plan into hardware-specific instructions. Unlike the previous phases, which required AI-specific optimizations, code generation follows many of the same principles as traditional compilers. This process includes instruction selection, register allocation, and final optimization passes, ensuring that execution makes full use of hardware-specific features such as vectorized execution, memory prefetching, and instruction reordering. Crucially, however, instruction selection for ML targets is not generic: the compiler must emit instructions that engage the hardware’s matrix-specific ISA extensions. On NVIDIA GPUs, this means emitting Parallel Thread Execution (PTX) instructions such as mma.sync.aligned to invoke Tensor Cores directly, as shown in listing 14. On Intel CPUs with Advanced Matrix Extensions (AMX), the compiler targets tile-multiply instructions operating on 2D register tiles. On Arm CPUs with the Scalable Matrix Extension, the target is outer-product accumulation across scalable matrix tiles. A code generation backend that emits generic floating-point instructions instead of these extensions leaves the hardware’s primary matrix engines idle, which can reduce effective throughput by an order of magnitude regardless of how well prior compilation phases performed. For CPUs and GPUs, AI compilers typically generate machine code or optimized assembly instructions, while for TPUs, field-programmable gate arrays (FPGAs),40 and other accelerators, the output may be optimized bytecode or execution graphs that are interpreted by the hardware’s runtime system.
40 Field-programmable gate array: “Field-programmable” means the logic fabric is configurable after manufacturing, contrasting with fixed-function ASICs. FPGAs can improve performance for latency-sensitive data center services by implementing custom pipelines matched to a particular workload (Putnam et al. 2014). This reconfigurability makes FPGAs attractive for rapidly evolving ML architectures where committing to an ASIC risks obsolescence, but the requirement for hardware description languages (Verilog/VHDL) and compilation times measured in hours creates a productivity barrier that limits adoption to deployments where the efficiency benefit justifies the engineering cost.
Putnam, Andrew, Adrian M. Caulfield, Eric S. Chung, Derek Chiou, Kypros Constantinides, John Demme, Hadi Esmaeilzadeh, et al. 2014. “A Reconfigurable Fabric for Accelerating Large-Scale Datacenter Services.” ACM SIGARCH Computer Architecture News 42 (3): 13–24. https://doi.org/10.1145/2678373.2665678.
From compilation to runtime
The compiler transforms a high-level model into an execution plan tailored to target hardware, but that plan still embeds assumptions about future conditions: shapes, workspace, device capabilities, and sometimes concurrency. Graph optimization, kernel selection, memory planning, and scheduling can make execution efficient; they cannot guarantee full utilization under every runtime workload.
Ahead-of-time compiler and engine-building optimizations occur before execution begins. This static nature enables aggressive graph optimization but limits adaptation when input shapes or resource conditions diverge from the prepared profiles. Graph restructuring, fusion, tactic selection, tiling, precision choices, and much of memory planning are therefore based on the shapes and hardware known at build time. Just-in-time compilation systems may compile additional variants later, but each generated kernel still has a fixed implementation.
Production AI systems inhabit a dynamic world that may not match these static assumptions. Batch sizes vary, multiple workloads may compete for an accelerator, and thermal throttling can reduce sustained performance below short-benchmark peaks. Runtimes and serving systems respond by selecting among supported profiles, managing buffers and streams, batching requests, and applying admission control; they do not generally invent new fusion or tiling strategies for a built engine. The serving chapter treats batching, admission control, and service-level objectives as end-to-end system problems (Model Serving); here the narrower question is how the accelerator runtime dispatches prepared execution plans once a request or batch reaches the hardware.
Self-Check: Question
What is the primary purpose of lifetime analysis in an ML compiler’s static memory planner?
- To calculate the physical degradation and failure rate of HBM memory cells over time
- To determine the precise intervals during which each intermediate activation tensor is needed, allowing disjoint tensors to share the same physical memory buffer
- To predict the number of training epochs required for a neural network to converge
- To prevent the compiler from generating out-of-order instruction streams
Explain how double buffering (software pipelining) implemented by an ML compiler hides memory access latency during loop execution on an accelerator.
Place the following compilation stages in the correct order as an end-to-end ML compiler (such as TVM or XLA) transforms a high-level deep learning model into executable machine code:
- Target Code Generation: Emit hardware-specific binary (e.g., PTX or machine instructions)
- High-Level Graph Optimization: Perform operator fusion, constant folding, and dead code elimination on the computation graph
- Front-End Ingestion: Parse framework model (e.g., PyTorch/ONNX) into High-Level Graph IR
- Low-Level IR & Auto-Tuning: Lower fused operators to loop-level IR and optimize tile sizes, thread bindings, and unroll factors
- Static Memory Planning: Analyze tensor lifetimes and allocate shared physical buffers
How does an auto-tuning ML compiler (such as TVM/Ansor) differ from a traditional handwritten library approach (such as cuDNN) for kernel selection?
- Auto-tuning compilers execute code only on the host CPU, whereas handwritten libraries run on GPUs
- Handwritten libraries search an infinite combinatorial loop space at runtime, whereas auto-tuning compilers use static heuristics
- Auto-tuning compilers explore large parameterized search spaces of loop transformations and tile sizes using cost models to generate custom kernels, whereas handwritten libraries rely on expert-tuned templates for specific fixed shapes
- Auto-tuning compilers require all tensors to be quantized to 1-bit integers
The optimization where an ML compiler merges two or more independent operators at the same graph depth into a single batched kernel to maximize GPU parallelism is known as ____ fusion.
Runtime Support
AI runtimes execute compiled graphs and engines while managing input-dependent shapes, buffers, streams, and supported execution profiles; TensorRT is one representative production inference stack. TensorRT’s builder compiles the network and selects tactics when creating an engine, while its runtime supplies inputs, selects a compatible optimization profile, and dispatches the prepared implementation (NVIDIA Corporation 2026a, 2026b).
NVIDIA Corporation. 2026a. How TensorRT Works.
NVIDIA Corporation. 2026b. Working with Dynamic Shapes.
Huang, Y., Y. Cheng, A. Bapna, O. Firat, D. Chen, M. X. Chen, H. Lee, et al. 2019. “GPipe: Efficient Training of Giant Neural Networks Using Pipeline Parallelism.” Advances in Neural Information Processing Systems (NeurIPS) 32: 103–12.
Mirhoseini, Azalia, Hieu Pham, Quoc V. Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, Naveen Kumar, Mohammad Norouzi, Samy Bengio, and Jeff Dean. 2017. “Device Placement Optimization with Reinforcement Learning.” International Conference on Machine Learning (ICML) 70: 2430–39.
AI runtimes manage three interrelated aspects of execution. First, they dispatch kernels or prepared graph variants that are legal for the current shapes and device. Second, they bind and reuse tensor buffers, workspaces, streams, and transfer resources. Third, higher-level framework runtimes can participate in multi-device execution, as in pipeline-parallel systems such as GPipe (Huang et al. 2019) and device-placement methods (Mirhoseini et al. 2017). The low-level inference runtime, training framework, and serving scheduler may divide these responsibilities differently.
AI runtimes complement compiler-based optimizations by handling these execution aspects dynamically. Comparing AI runtimes to traditional software runtimes clarifies why machine learning workloads require specialized execution strategies.
ML runtime architecture
General-purpose runtimes manage threads, tasks, objects, memory, and asynchronous I/O across diverse programs. AI runtimes build on those mechanisms but represent work in terms of tensors, kernels, graphs, device buffers, and accelerator queues.
Table 24 contrasts their emphasis. AI runtimes must respect graph dependencies and tensor shapes while coordinating kernels and transfers, but they coexist with conventional language, operating-system, and device runtimes.
| Aspect | Traditional runtime | AI runtime |
|---|---|---|
| Execution Model | Functions, threads, tasks, events, and asynchronous I/O | Tensor graphs, kernels, streams, and device events |
| Task scheduling | Threads and tasks across processor resources | Dependency-aware kernel and transfer dispatch |
| Memory Management | Objects, stacks, heaps, pools, and virtual memory | Tensor buffers, workspaces, device memory, and buffer reuse |
| Optimization Priorities | Responsiveness, throughput, locality, and resource sharing | Shape compatibility, transfer overlap, reuse, and utilization |
| Adaptability | Dynamic scheduling and allocation within program semantics | Runtime choices within compiled graphs and available kernels |
| Target Hardware | CPUs and heterogeneous devices | CPUs, GPUs, TPUs, and custom accelerators |
Tensor memory is a major specialization. AI runtimes often combine static lifetime plans for fixed shapes with pools or dynamic allocation for variable shapes, then bind those buffers to device kernels and transfers. Poor planning can increase peak memory, prevent buffer reuse, or add transfers; cache behavior remains largely under the compiled kernels and hardware.
AI runtimes support variability within the bounds of their compiled engines or available kernel libraries. They can manage dynamic tensor buffers, select a compatible shape profile, coordinate streams, and participate in multi-device execution. Request batching and fleet-level resource allocation usually belong to the serving scheduler rather than the low-level accelerator runtime.
Production conditions can differ from the benchmark environment. The A100 SXM has a maximum thermal design power (TDP) of 400 W, while the A100 PCIe has a 300 W maximum TDP; these are different form factors with different cooling envelopes, so a deployment may not sustain the throughput measured on another variant. Batch size, memory availability, and resource contention can also vary. The system must therefore prepare appropriate engine profiles and use serving controls such as batching and admission control rather than assume that the runtime will retune compiled kernels in response to temperature or load.
To see how these runtime mechanisms work together, consider a transformer inference request arriving at a production server. The serving system forms a supported batch, the runtime selects a compatible prebuilt profile and manages its buffers, and the engine dispatches the kernels chosen during compilation or engine construction. The subsections that follow examine these bounded runtime choices using the request as a running example.
Dynamic kernel execution
While static compilation provides a foundation, efficient execution may require choosing among prepared variants. When a transformer request arrives, its batch and sequence dimensions determine which supported optimization profile and kernels can execute it. Shapes outside those profiles require a different engine or additional compilation rather than arbitrary runtime retuning.
Individual operations are assigned implementations during compilation or engine construction. At execution time, the runtime binds buffers, selects legal variants when the engine provides them, and launches work in dependency order.
The same constraint appears in image classification. If an incoming batch of high-resolution images falls outside the memory assumptions of a prepared profile, the serving system must select a compatible profile or engine, split the batch, or rebuild the engine. The runtime cannot invent a new tiling strategy for an already built engine.
For the running transformer inference request, sequence length may vary between calls. A plan optimized for one fixed length can underutilize compute resources on shorter sequences or exceed its workspace assumptions on longer ones. A runtime can select a compatible prepared profile and allocate its required buffers; unsupported lengths require another compiled variant or a more general kernel.
Overlapping computation with memory movement can mitigate transfer bottlenecks. When dependencies, page-locked buffers, streams, and copy engines permit concurrency, asynchronous execution and double buffering transfer batch \(n{+}1\) while computing batch \(n\). This can hide part of the host-to-device transfer latency, but it does not eliminate stalls when transfer time exceeds compute time or resources serialize.
Convolutional layers illustrate this boundary: TensorRT may fuse operations and benchmark convolution tactics while building the engine. During inference, the runtime dispatches those prepared layers; the GPU’s hardware schedulers place their thread blocks on available SMs.
Selecting among prepared execution strategies in response to request shape and system conditions can improve both training and inference performance. This adaptation depends on having the right kernel available. In the transformer inference scenario, the runtime must choose among implementations prepared for the request’s supported shape and precision.
Runtime kernel selection
While compilers perform an initial selection of kernels based on static analysis, AI runtimes may still choose among precompiled or library-provided variants during execution. Real-time factors, such as available memory, hardware utilization, and workload priorities, may differ from the assumptions made during compilation. In the transformer scenario, the compiler and framework determine the legal precision paths, while the runtime selects the kernel variant that best fits the current sequence length, batch shape, and available hardware resources. Runtime selection adapts execution to changing conditions, but it remains bounded by the numerical formats and kernels the model has been prepared to use.
For instance, transformer-based language models spend substantial time in matrix multiplications. Mixed-precision systems such as Megatron-LM use FP16 execution on GPU Tensor Cores to increase throughput (Shoeybi et al. 2019). Which operations remain in FP32 is a model-conversion or engine-building decision validated before deployment; the runtime does not infer numerical instability and change precision on its own.
Shoeybi, Mohammad, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. “Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism.” arXiv Preprint arXiv:1909.08053.
Memory constraints also influence kernel selection. When memory bandwidth is limited, the runtime may select a prepared kernel or profile whose tiling and workspace requirements fit the available resources. Creating a new tiling strategy requires engine building or compilation rather than an arbitrary runtime adjustment.
Batch size also influences kernel selection. When a stack provides multiple compatible variants, the runtime may use a latency-oriented implementation for small batches and a throughput-oriented one for larger batches. The benefit depends on the prepared kernel set and the cost of switching profiles. The final pipeline stage then determines when each selected kernel runs.
Kernel scheduling and utilization
Kernel dispatch completes the runtime pipeline. Returning to the transformer request, the runtime launches the prepared attention, normalization, and activation kernels on streams while respecting graph dependencies. On a GPU, hardware block schedulers assign thread blocks to SMs; the runtime does not directly distribute individual operations across cores. Other accelerators expose analogous queues or execution commands (Jouppi et al. 2017).
In image recognition models, independent kernels may overlap on separate streams when their dependencies and resource use permit. Within each GPU kernel, however, the compiled grid defines the work and GPU hardware schedules its thread blocks; the runtime does not distribute individual filters across processing units.
Memory management reinforces the same goal through buffer reuse, asynchronous transfers, and explicit staging where the API supports it. Hardware caches remain hardware-managed; a runtime cannot generally place arbitrary intermediate tensors into cache.
Together, profile selection, buffer management, and kernel dispatch form the runtime pipeline. For the transformer request, the runtime chooses among implementations and shapes prepared by the compiler or engine builder, then launches them efficiently. Changes to fusion, tiling, or legal precision generally require rebuilding or recompiling rather than continuous runtime retuning.
The compiler and runtime systems examined thus far optimize execution within single accelerators, but the largest AI workloads exceed what any single chip can deliver. Single-chip optimizations can achieve impressive results through compiler optimization, dataflow selection, fusion, and memory planning. Yet for the largest AI workloads, even well-optimized single-chip execution proves insufficient.
Consider the estimated \(3.14 \times 10^{23}\) floating-point operations associated with training GPT-3 (Brown et al. 2020). As a scale comparison, divide that historical operation count by a later H100’s peak FP8 rate of 1.98 PFLOP/s. The arithmetic lower bound is about 5 years on one device, and 8.4–12.6 years under the illustrative 40–60 percent utilization range (Choquette 2023). This is not a reconstruction of GPT-3 training: the precision, hardware, communication, and workload are counterfactual. It simply shows why model-scale compute or service-scale request volume can exceed the useful capacity of one accelerator.
Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. 2020. “Language Models Are Few-Shot Learners.” Advances in Neural Information Processing Systems 33: 1877–901. https://doi.org/10.48550/arxiv.2005.14165.
Self-Check: Question
Why do deep learning frameworks implement custom caching memory allocators (such as PyTorch’s
caching_allocator) rather than directly invokingcudaMallocandcudaFreefor every intermediate tensor?cudaMallocoperates only in FP32 precision and cannot allocate FP16 memory buffers- Direct OS memory allocation encrypts all tensor data, introducing cryptographic decryption latency
- GPU DRAM can only be allocated once during system boot time
cudaMallocis a synchronous operation that stalls GPU execution and causes expensive driver and OS page-table synchronization overhead
Explain how CUDA Graphs eliminate CPU kernel launch overhead during repeated training or inference iterations on small or low-latency models.
True or False: In an asynchronous GPU runtime model, when a Python script executes
y = torch.matmul(a, b), the CPU thread blocks and waits until the GPU hardware finishes computing the matrix multiplication before executing the next line of Python code.A sequence of asynchronous GPU operations that execute strictly in FIFO order on an accelerator is called a CUDA ____.
An inference serving system processes dynamic batch sizes ranging from 1 to 32 tokens per request. Why does dynamic batching pose a significant challenge to runtime kernel selection and hardware efficiency?
- Optimal tile sizes, thread block configurations, and memory bandwidth requirements change across batch sizes, making a single static kernel inefficient across all shapes
- Dynamic batching forces the GPU to switch from FP16 to FP64 precision for even batch sizes
- Tensor Cores cannot execute matrix multiplication when batch dimensions are not powers of two
- The GPU memory controller must physically power down DRAM banks when batch size decreases
Multi-Chip Scaling
A single H100 has a large FP8 peak, yet a training-time or throughput target may still require many accelerators. The techniques covered earlier remain the foundation for multi-chip scaling; each device still needs an efficient dataflow. The new lesson is that communication adds further hierarchy boundaries. Moving data through on-chip storage, HBM, an intra-node link, and a cluster fabric generally increases latency and energy, so scaling is not merely a question of adding chips.
When single-accelerator capacity proves insufficient, the design problem shifts from feeding one chip to choosing which communication boundary the workload can tolerate. Practitioners encounter these boundaries in production even when most optimization work remains inside a single accelerator.
Multi-chip scaling approaches
Large AI systems scale beyond individual accelerators by moving the communication boundary outward, and each boundary changes the dominant trade-off. The sequence begins inside the package. Chiplet-based architectures partition large designs into smaller, modular dies interconnected within one package, bypassing manufacturing limits of monolithic chips while preserving relatively low communication latency. The next boundary is the node: multi-accelerator servers connect several chips through board- or server-level interconnects. Each accelerator has dedicated memory and compute resources, so workloads split through data parallelism (each accelerator processes different batches) or model parallelism (different accelerators handle different network layers). High-bandwidth intra-node interconnects can enable efficient gradient synchronization, though realized performance depends on topology and collective communication efficiency.
Beyond the node, the boundary expands into the cluster. Purpose-built data center fabrics coordinate hundreds of accelerators, making topology and collective communication algorithms central determinants of scaling efficiency; near-linear scaling is achievable on some workloads when communication overhead is controlled. Wafer-scale integration is the counter-move: instead of pushing the boundary outward, it collapses more computation back into one large device. Platforms such as Cerebras WSE-class systems integrate extremely large numbers of transistors and cores on a single device, reducing inter-chip communication overhead while introducing their own challenges in thermal dissipation, fault tolerance, and manufacturing yield.
Why scaling introduces new constraints
The transition from single-chip to multi-chip architectures introduces communication overhead and other constraints that reshape optimization. Communication, synchronization, load balance, and parallel work per device can all limit scaling. Amdahl’s Law41 provides one first-order model for exposed work that does not shrink with additional devices. For hundred-billion-parameter models, an unsharded full-gradient payload can occupy hundreds of gigabytes per step, and an AllReduce42 must aggregate it; precision, sharding, compression, and the collective algorithm change the bytes each device transfers.
41 Amdahl’s law (scaling limit): Amdahl's Law and Gustafson's Law formalizes the bound for a fixed nonaccelerated fraction. If 5 percent of a fixed workload remains exposed and unaffected by device count, that model caps speedup at 20\(\times\). Distributed training is more complicated because communication can overlap computation and its cost can itself change with scale, but the example explains why exposed synchronization must be reduced or hidden.
42 AllReduce: A collective operation from MPI that aggregates values across processes (the “reduce”) and distributes the result back to every process (the “all”). In this chapter, the term appears only to identify why accelerator-to-accelerator bandwidth matters for training workloads. At scale, algorithms, topology choices, and runtime protocols determine how costly this synchronization becomes.
The first-order quantity is the gradient payload. For a model with \(P\) parameters, that payload is roughly the parameter count multiplied by the bytes stored for each gradient element before optimizer state, padding, and protocol overhead enter. Scaling therefore improves only when the saved compute time exceeds the time to move this payload through the chosen interconnect and collective algorithm.
This overhead explains why large accelerator counts can show diminishing returns unless the system reduces exchanged data, overlaps communication with useful computation, or chooses a better parallelization pattern. The memory model also changes: separate accelerator memories are commonly managed explicitly, and collective operations establish the required synchronization rather than presenting one fully coherent memory image across the cluster. Coherent links exist in some systems, but their scope and cost are architecture-dependent.
Once computation spans many links, chips, and memory stacks, reliability and energy become part of the same scaling story. Large-scale systems must handle component failures gracefully because the probability of at least one failure rises with system size. For this chapter, the hardware lesson is enough: distributed TPU systems must tolerate component failures at system scale (Jouppi et al. 2023), while Cerebras uses redundant cores and fabric links to replace manufacturing-defective cores and restore the wafer’s logical mesh (Lie 2021). Data movement also grows more expensive with distance, transforming distributed training into a careful balance between computation parallelism and communication efficiency.
Jouppi, Norm, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, et al. 2023. “TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings.” Proceedings of the 50th Annual International Symposium on Computer Architecture, 1–14. https://doi.org/10.1145/3579371.3589350.
Lie, Sean. 2021. Thinking Outside the Die: Architecting the ML Accelerator of the Future.
Data center scaling and edge deployment represent opposite ends of a deployment spectrum, yet they share the same core principles. Data center scaling coordinates many high-throughput accelerators, while edge scaling fits useful AI into a few constrained watts. Both cases require matching workload characteristics to hardware capabilities while minimizing data movement. The principles of compute specialization, memory hierarchy optimization, and workload mapping apply at both scales; only the constraints differ. Data centers optimize for aggregate throughput within power budgets measured in megawatts; edge devices optimize for responsiveness within tight battery and thermal envelopes. The same vision model that runs comfortably in a data center may need a radically different mapping strategy on a smartphone or microcontroller.
Self-Check: Question
Why have accelerator architectures increasingly shifted from monolithic single-die designs toward Multi-Chip Module (MCM) and chiplet architectures?
- Chiplets eliminate all need for semiconductor fabrication foundries
- Monolithic dies are constrained by the physical lithography reticle limit (\(\approx 858\text{ mm}^2\)) and suffer exponential yield loss as die size increases
- Monolithic dies cannot support high-bandwidth memory (HBM) interfaces
- Chiplets allow electrical signals to travel faster than the speed of light
Explain how the non-uniform memory access (NUMA) effect and interconnect bandwidth degradation impact performance when scaling a neural network workload across multiple chiplets or accelerator chips.
Describe the architectural rationale behind Cerebras’s Wafer-Scale Engine and explain how fabricating an accelerator across an entire uncut silicon wafer overcomes traditional multi-chip scaling bottlenecks.
What is the primary bottleneck introduced by inter-node scaling (scaling out across separate servers over Ethernet or InfiniBand) compared to intra-node NVLink scaling?
- Inter-node network adapters cannot transmit FP16 or BF16 floating-point values
- Inter-node communication requires GPUs to switch to single-threaded CPU emulation mode
- Inter-node network bandwidth (e.g., \(400\text{ Gbps} \approx 50\text{ GB/s}\)) is roughly an order of magnitude lower than intra-node NVLink bandwidth (\(\approx 900\text{ GB/s}\)), increasing collective communication latency
- Inter-node scaling eliminates the need for gradient synchronization in distributed training
Heterogeneous SoC Design
Mobile, automotive, and IoT deployments often face tighter per-device power, thermal, and latency limits than data center hardware. Heterogeneous SoCs integrate CPU cores, GPU shaders, digital signal processors (DSPs), and neural processing units (NPUs) under shared memory and system budgets. The mapping problem remains familiar, but it now includes operator support, transfer overhead, real-time deadlines, and thermal state.
Mobile SoC architecture evolution
Modern mobile AI engines exemplify heterogeneous computing by coordinating CPU cores, GPU shaders, DSPs, and dedicated NPUs43 across a shared memory hierarchy. Workload distribution lets computer vision kernels execute on GPU or NPU paths, audio processing use DSP arithmetic units, and matrix-heavy neural-network layers use NPU-optimized engines when the operator set is supported. This coordination requires careful scheduling to meet real-time constraints while managing thermal throttling and battery life.
43 NPU (neural processing unit): The NPU’s specialized matrix engines are optimized for dense tensor operations, providing the hardware basis for the workload distribution described. This specialization creates a critical constraint for the scheduler: any AI operator not mapped to the NPU’s supported data paths must “fall back” to a GPU or CPU. This fallback can erase the NPU’s energy-efficiency advantage, complicate real-time latency budgets, and contribute to thermal pressure on mobile devices.
Example 1.5: Heterogeneous microcontrollers
Scenario: An embedded smart doorbell must run periodic person detection within a tight memory, energy, and latency budget.
Diagnosis: On a supported deployment, a general-purpose Cortex-M path misses the measured energy or latency target. A micro-NPU such as Arm Ethos-U can offload compatible convolution operators, while the CPU retains control flow and unsupported operations.
Systems lesson: Micro-NPU specialization can improve the energy and latency of supported operators, but feasibility must be measured for the whole pipeline, including fallbacks and sensor processing.
Some mobile SoC designs emphasize diverse processor specialization, while vertically integrated strategies highlight how tight hardware-software co-design can enable tightly coordinated heterogeneous execution. Unified memory architectures can reduce explicit data copying overhead, and different compute blocks can be scheduled for different operator types (for example, matrix-heavy layers on an NPU, convolutional operators on a GPU, and control flow on the CPU). This coordination supports interactive on-device experiences, though realized latency depends on the full pipeline and device thermal conditions.
Beyond vertically integrated solutions, IP licensing models allow SoC designers to customize processor combinations based on target applications, mixing CPU, GPU, DSP, and NPU blocks. This modular flexibility allows automotive SoCs to emphasize deterministic real-time processing while smartphone SoCs optimize for interactive performance and battery efficiency.
Strategies for dynamic workload distribution
With multiple specialized processors available on heterogeneous SoCs, the critical challenge becomes intelligently distributing neural network operations across these resources to maximize performance while respecting power and latency constraints. Consider a concrete example: an engineer deploying a real-time object detection pipeline on a mobile SoC with a CPU, GPU, and NPU. The pipeline has three stages: a MobileNet backbone for feature extraction, nonmaximum suppression (NMS) for postprocessing, and a display overlay for rendering bounding boxes. The backbone consists of depthwise separable convolutions with regular, predictable access patterns and low compute cost, making it a good fit for an NPU when the operator set is supported, even though depthwise layers are often memory-bound rather than high-arithmetic-intensity kernels. NMS, by contrast, involves conditional branching over variable-length candidate lists, with irregular memory access that maps poorly to the NPU’s fixed dataflow. The CPU handles NMS more efficiently because its branch predictor and large caches accommodate the unpredictable control flow. Finally, the display overlay involves pixel-level compositing across the entire frame, a massively parallel but arithmetically simple workload that maps naturally to the GPU’s shader cores. This three-way split, NPU for the backbone, CPU for NMS, GPU for the overlay, achieves lower latency and lower power than running the entire pipeline on any single processor.
The example shows how supported operators, measured transfer cost, and the current system budget determine pipeline partitioning. Regular convolutions often map well to GPU or NPU paths, while irregular control flow may favor a CPU. Attention performance depends on shape, precision, kernel availability, and setup overhead; sequence length alone does not determine the best processor.
The best supported assignment can also change with operating mode. A system may reduce frame rate, select a smaller model, lower accelerator frequency, or choose another validated backend as battery, thermal state, and concurrent load change. Moving work is not inherently more efficient because transfers and fallback kernels can outweigh the gain. Safety-critical applications further prioritize bounded latency and validated behavior over peak throughput.
Compounding the processor selection challenge, shared memory architectures require arbitration when multiple processors access LPDDR simultaneously. Mobile memory controllers may prioritize real-time camera or display paths over background AI tasks, forcing neural-network runtimes to adapt their execution patterns to available bandwidth. This arbitration becomes critical during memory-intensive operations like large language model inference, where parameter streaming from DRAM must be carefully coordinated across processors.
Power and thermal management
Mobile AI workloads must maintain high performance while operating within strict power budgets and thermal envelopes. These constraints require tight coordination across heterogeneous processors.
Heterogeneous SoCs use dynamic voltage and frequency scaling (DVFS) across power domains to manage performance within a shared envelope. Raising one domain’s operating point can leave less thermal or electrical headroom for others. Control policies react to utilization, deadlines, temperature, and platform-specific power limits; transitions themselves have latency and stability costs.
When DVFS alone cannot maintain the power envelope, mobile SoCs implement thermal throttling through a mixture of frequency reduction, model adaptation, and task migration. When the NPU approaches thermal limits during intensive neural network processing, the runtime can shift selected operators to another supported processor, lower inference frequency, or choose a smaller model profile. This approach preserves service availability during thermal events, though it requires detailed workload characterization to predict execution time and power consumption across different processors.
Mobile AI systems may also adapt quality, model choice, or inference frequency to battery and charging state. The most energy-efficient processor is workload- and implementation-dependent, so a lower-power operating mode must be selected from measured, validated alternatives rather than inferred from processor labels.
Automotive heterogeneous AI systems
Automotive applications combine constrained power and cooling with deadlines and functional-safety requirements. The required guarantees depend on the function’s safety classification, so not every in-vehicle ML workload is hard real time.
Automotive SoCs use combinations of redundancy, monitoring, partitioning, and bounded scheduling to support safety objectives while accelerators execute perception and other compute-intensive functions. Shared bandwidth is one concrete risk: a convenience workload can interfere with a perception pipeline that has a validated deadline. Hardware partitions, bandwidth controls, or time-triggered schedules can isolate the relevant resources, but the mechanism and guarantee are platform-specific. Multisensor pipelines likewise need bounded end-to-end timing rather than high average throughput alone.
Vehicles may distribute sensing and computation across several controllers or consolidate functions onto central platforms. Either design must timestamp, transport, and align sensor data within the assumptions of the fusion pipeline. The resulting synchronization and communication costs belong in the same latency budget as accelerator execution.
External vehicle-to-everything (V2X) inputs add another source with different latency and trust properties. A safe design cannot assume those messages arrive with the timing or reliability of local sensors; any use in a safety-related function must be bounded and validated accordingly.
Software stack challenges
The architectural sophistication of heterogeneous SoCs turns software into the coordination layer for power, thermal state, determinism, and operator fallback. OpenCL targets heterogeneous compute devices, while Vulkan exposes portable GPU compute; neither erases device-specific tuning. Mobile inference runtimes use delegates or backends for CPUs, GPUs, DSPs, and NPUs, but unsupported operators may fall back to another processor and add transfers. Deployment therefore requires inspecting the realized partition, not only the framework graph.
Shared memory architectures compound the coordination problem. Memory management must account for processor-specific caching behavior, memory access patterns, and coherency requirements. CPU caches may interfere with GPU memory access patterns, while NPU direct memory access (DMA) operations must be synchronized with CPU cache operations to maintain data consistency.
Heterogeneous SoCs address this complexity with profiled cost models, rules, control policies, and sometimes learned predictors. Telemetry on latency, utilization, temperature, and power can guide selection among validated execution plans; it does not remove the need to characterize operator support and worst-case behavior.
No single processor architecture can optimally handle the diverse computational patterns in AI applications, so heterogeneous acceleration becomes a coordination problem rather than a hardware inventory. Efficient mobile AI systems deliver high performance only when processor assignment, memory coherence, thermal limits, and latency constraints are managed together.
The same coordination problem also has an energy consequence. If hardware selection determines how much data moves, how often accelerators stall, and how efficiently arithmetic maps to silicon, then it also determines how much energy the deployment consumes for each useful prediction.
Self-Check: Question
On a modern mobile heterogeneous System-on-Chip (SoC) featuring big.LITTLE CPUs, a mobile GPU, and a dedicated NPU, which compute engine is most energy-efficient for running continuous, low-latency 8-bit quantized convolutional inference?
- The high-performance ‘big’ CPU core running single-threaded FP32 instructions
- The out-of-order system memory controller
- The dedicated Neural Processing Unit (NPU) optimized for quantized INT8 matrix operations
- The host operating system virtualization hypervisor
In automotive autonomous driving SoCs, explain why deterministic worst-case execution time and lockstep redundancy are required, even if they reduce peak average-case throughput.
True or False: In mobile SoCs with unified system memory (LPDDR), sharing physical RAM between the CPU, GPU, and NPU eliminates all data movement overhead between heterogeneous processors.
Describe how Dynamic Voltage and Frequency Scaling (DVFS) and thermal throttling constrain sustained AI inference performance on edge and mobile devices.
Hardware Sustainability
At fleet scale, energy per useful inference becomes an important hardware-selection criterion. Operational emissions depend on workload throughput, measured system power, utilization, cooling overhead, and the electricity mix; embodied emissions add another lifecycle cost. Peak performance per watt is therefore a screening metric, not a substitute for end-to-end energy per request. The following scenario shows how assumptions compound when comparing a generic-CPU fleet with specialized accelerators.
Napkin Math 1.10: An illustrative operational-energy comparison
Problem: Under a stated throughput-equivalence assumption, how does operational energy change when a CPU fleet is replaced by specialized NPUs (Neural Processing Units)?
Assumptions: This stylized comparison treats the listed peak rates as useful workload throughput, assumes both fleets run continuously at the listed power, and excludes host power, cooling, embodied carbon, and utilization differences. It is a sensitivity calculation, not a measured procurement result.
- CPU inference: 100 W for 1 TFLOP/s (efficiency = 0.01 TFLOP/s/W).
- NPU inference: 5 W for 10 TFLOP/s (efficiency = 2 TFLOP/s/W).
- The assumed peak-efficiency gap: 200×.
Math:
- Workload: 1 billion inferences per day.
- CPU fleet energy: 1,000 CPU servers \(\times\) 100 W \(\times\) 24 h \(\approx\) 2,400 kWh/day.
- NPU fleet energy: 100 NPU chips \(\times\) 5 W \(\times\) 24 h \(\approx\) 12 kWh/day.
- Carbon savings: At 0.429 kg/kWh, switching to NPUs saves ~373.9 t of CO2 per year.
Systems insight: Specialized accelerators can reduce operational energy when the workload uses their efficient path and fewer devices deliver the required service throughput. Replace these assumptions with measured requests per second and wall power before using the result for procurement or carbon accounting.
The sustainability perspective reinforces a theme that has recurred throughout this chapter: hardware selection is not determined by peak speed alone. Energy per useful result, carbon intensity, embodied impact, and total cost of ownership belong beside latency, throughput, and memory capacity. The remaining step is to identify the misconceptions that cause teams to choose the wrong hardware path.
Self-Check: Question
In the lifecycle carbon assessment of advanced deep learning accelerators, what constitutes ‘embodied carbon’?
- The electrical energy consumed by the GPU during model forward and backward passes
- The carbon emitted by datacenter air conditioning units during peak summer load
- The carbon credits purchased by cloud providers to offset datacenter energy usage
- The greenhouse gas emissions generated during raw material extraction, semiconductor silicon manufacturing, packaging, and hardware transportation
Explain why maximizing accelerator utilization (e.g., via multi-tenant sharing or continuous pipeline saturation) reduces the amortized carbon footprint per trained model.
True or False: In a datacenter with a Power Usage Effectiveness (PUE) of 1.1, the cooling and electrical distribution infrastructure consumes more power than the actual computing IT equipment (servers and accelerators).
Fallacies and Pitfalls
Hardware acceleration has counterintuitive performance characteristics because impressive specifications can mask workload-specific bottlenecks. The fallacies and pitfalls here capture selection and optimization errors that leave expensive resources underused.
Fallacy: More specialized hardware always provides better performance than general-purpose alternatives.
Engineers assume specialized accelerators automatically outperform general-purpose processors for all AI workloads. In reality, specialized hardware achieves peak performance only when workloads match architectural assumptions, the core of algorithm-machine co-design. As demonstrated in section 1.5, operations must exceed the accelerator’s ridge point to be compute bound; an A100 GPU has a ridge point of 153 FLOP/byte, meaning operations with arithmetic intensity below this threshold are memory bound regardless of the accelerator’s 312 TFLOP/s peak compute. A transformer attention softmax with AI = 2 FLOP/byte–5 FLOP/byte achieves only 4.1 TFLOP/s–10.2 TFLOP/s (3.3 percent utilization) on an A100. CPUs with ridge points around 10 FLOP/byte–20 FLOP/byte still treat this kernel as memory-bound, but the same AI range corresponds to about 10 percent–50 percent of a CPU’s lower peak. Models with irregular memory access, small batch sizes, or dynamic computation graphs may perform better on flexible processors. Effective hardware selection requires matching workload arithmetic intensity to architectural ridge points, not assuming specialization always wins.
Pitfall: Ignoring memory bandwidth limitations when selecting acceleration strategies.
Practitioners focus on peak TFLOP/s without analyzing whether their workloads can achieve compute-bound performance. As quantified in section 1.4.1, the energy model used here assigns about 640 pJ to a DRAM access versus 0.5 pJ for an on-chip L1 SRAM access, creating orders-of-magnitude energy penalties. An accelerator advertising 300 TFLOP/s with 2 TB/s bandwidth has a ridge point of 150 FLOP/byte; LayerNorm operations with AI = 1.5 FLOP/byte achieve only 3 TFLOP/s (1 percent utilization) in this worked example. Organizations can deploy expensive high-compute accelerators for memory-bound workloads and still see low utilization if bandwidth, not compute, is the bottleneck. Teams must calculate workload arithmetic intensity and compare against hardware ridge points before purchasing accelerators.
Fallacy: Hardware acceleration benefits scale linearly with additional accelerators.
Teams expect eight GPUs to train 8\(\times\) faster than one GPU. Multi-accelerator scaling introduces communication overhead that violates linear scaling assumptions. As noted in section 1.10, AllReduce operations for gradient synchronization can require exchanging large gradient payloads for large models. In an ideal ring AllReduce, each rank transfers \(2(N-1)/N\) payloads, or 1.75× payloads for eight GPUs. With NVLink delivering 600 GB/s bidirectional (half that per direction), synchronizing a 1 GB gradient therefore requires 5.83 ms; relative to a 50 ms compute step, this is 11.7 percent before protocol overhead. Without compute-communication overlap, this worked eight-GPU scenario achieves about 7.2× speedup (89.6 percent efficiency) before load imbalance, synchronization barriers, and insufficient parallel work reduce scaling further.
Pitfall: Planning accelerator capacity from peak FLOP/s specifications.
Peak FLOP/s is one accelerator capability, but delivered throughput also depends on arithmetic intensity, shape, precision, kernel efficiency, and system overhead. The Roofline Model (section 1.5) bounds the compute and bandwidth portions of that gap. In the representative budgeting scenarios here, A100 transformer training sustains 120 TFLOP/s–180 TFLOP/s, while a sparse recommender workload sustains 10 TFLOP/s–30 TFLOP/s. These ranges are planning assumptions, not universal measurements. Capacity plans should use representative measured throughput where possible and roofline estimates when measurements are unavailable.
Fallacy: Any FLOP/s rating can estimate a low-precision workload.
Accelerators have separate datapaths for different precisions, and the peak throughput varies dramatically across them. An H100 delivers roughly 989 TFLOP/s in FP16 tensor operations but only about 67 TFLOP/s in FP32 CUDA-core operations: a roughly 15× gap within the same chip (Choquette 2023). Estimating training time with the FP32 number when the workload actually uses BF16 produces utilization figures that look catastrophic for no reason, and matching against the wrong roofline misclassifies kernels as compute-bound when they are memory-bound (or vice versa). Always match the peak constant to the precision the workload actually issues, and quote precision explicitly when reporting model FLOPs utilization.
Pitfall: Deploying small-batch inference workloads on high-compute accelerators.
Small-batch inference can underuse a high-compute accelerator. In this dense-layer traffic model, \(M=N=2048\) gives AI = 1 FLOP/byte at batch 1 and AI = 204.8 FLOP/byte at batch 256. At batch 1, the memory-bound roofline ceilings are about 2.04 TFLOP/s on A100 and 0.3 TFLOP/s on T4. The T4’s FP16 Tensor Core peak is 65 TFLOP/s, with a ridge point of 203.1 FLOP/byte. A lower-cost accelerator can therefore be more economical for this regime, but latency, memory capacity, concurrency, software support, and current pricing must be measured. Match the deployment to its actual batch and service objective rather than assigning accelerator classes by name.
Fallacy: Vendor-specific optimizations have no long-term portability cost.
Organizations optimize exclusively for specific vendors to maximize performance without considering system flexibility. As discussed in section 1.8, deep integration with vendor-specific libraries (CUDA, TensorRT, XLA) and custom kernels creates lock-in. A codebase with many hand-written accelerator kernels can require substantial engineering effort to port to a different vendor, delaying hardware upgrades and preventing multi-vendor deployments. Vendor-specific optimizations should therefore be isolated behind hardware abstraction layers. Maintaining portable code paths enables vendor competition, hardware flexibility, and faster adoption of emerging accelerators while still capturing most performance benefits through framework-level optimizations.
Checkpoint 1.4: Feasibility assessment: Can you run it?
Before procuring hardware, validate all three hard constraints. The fallacies and pitfalls in this section reduce to a concrete procurement test:
This checklist synthesizes the principles developed throughout this chapter, translating theoretical understanding into practical engineering decisions. Together, these fallacies reduce the chapter’s machinery to a diagnostic habit of starting from the workload, choosing the bottleneck metric, matching the hardware path, and budgeting for its portability, scaling, and energy consequences.
Self-Check: Question
A team prunes \(70\%\) of the weights in a large language model using unstructured magnitude pruning, setting those weights to zero. However, when executing the pruned model on standard GPU dense Tensor Cores, inference latency is identical to or slower than the unpruned baseline. What is the primary cause of this pitfall?
- Standard dense hardware cannot skip individual zero elements without structured patterns (e.g., 2:4) or specialized sparse matrix indexing, so dense matrix units still execute all multiplications while sparse formats add indexing overhead
- Floating-point units automatically convert zero values into infinite loops
- Unstructured pruning forces the GPU driver to downclock memory bandwidth to prevent overheating
- The operating system kernel intercepts every zero multiplication and raises a hardware page fault
Describe the pitfall of ‘micro-offloading’ small tensor operations from CPU to GPU, and explain why a sequence of scalar operations can run slower on an accelerator than on the host CPU.
Place the following diagnostic steps in the recommended sequence when troubleshooting an underperforming neural network training workload on an accelerator cluster:
- Roofline & Hardware Counter Analysis: Determine whether individual kernels are compute-bound, memory-bandwidth-bound, or latency-bound
- Amdahl & Host-Side Profiling: Identify serial bottlenecks, data loading stalls, and CPU-GPU synchronization delays
- Kernel-Level Optimization: Apply operator fusion, hierarchical tiling, or precision reduction targeted to the identified bottleneck
- Multi-Device Communication Profiling: Check for gradient all-reduce synchronization delays and interconnect saturation
Which of the following statements represents a classic fallacy regarding peak accelerator specifications?
- High-bandwidth memory reduces the latency of memory-bound operations compared to standard DDR5
- An accelerator with \(2\times\) higher peak theoretical TFLOP/s will automatically deliver a \(2\times\) speedup on any neural network workload
- Kernel fusion can improve arithmetic intensity by reducing global memory traffic
- Warp divergence reduces the execution efficiency of SIMT processor lanes
Summary
Hardware acceleration co-designs compute engines, memory hierarchies, numerical formats, and dataflows. For a chosen memory interface, the iron law compares compute time \(O/(R_{\text{peak}}\eta_{\text{hw}})\) with movement time \(D_{\text{vol}}/\text{BW}\); the roofline restates that competition as \(\min(R_{\text{peak}}, \text{BW}I)\). End-to-end latency also includes launch, synchronization, communication, and fixed overheads. Accelerators improve efficiency only when supported paths capture reuse locally. Hardware selection therefore begins with the workload’s measured or modeled bottleneck, not a vendor peak.
Key Takeaways: Moving data costs more than computing it
- The Roofline Model identifies bottlenecks: Compare a kernel’s measured intensity with the target’s precision-specific ridge point (153 FLOP/byte here). Below it, reduce memory traffic; above it, improve compute utilization.
- Memory bandwidth constrains performance: GPU compute capacity has grown faster than memory bandwidth. Many inference phases are memory bound, especially low-batch decoding and embedding lookup, while large-batch GEMMs can remain compute bound.
- Hardware-software co-design compounds performance: Matching algorithm patterns to architectural capabilities, such as dense GEMM on systolic or tensor units and supported sparsity on sparse paths, can produce large improvements.
- Tensor Cores require a compatible path: Precision, dimensions, layouts, and library support must match the architecture. Batch and shape affect reuse and utilization, but no single batch threshold guarantees peak performance.
- End-to-end speedup stops at the next bottleneck: Accelerating one kernel leaves launch, transfer, synchronization, communication, and unsupported work unchanged. Use Amdahl’s law and profiling to decide whether further silicon, better mapping, or less movement will improve the complete workload.
An accelerator is not a uniformly faster computer; it provides efficient paths for particular operations and dataflows. Tiling, fusion, hierarchy-aware scheduling, and systolic execution reduce modeled-interface traffic only when the workload, numerical format, and software path can use them.
The Roofline Model provides a first diagnostic by using arithmetic intensity and the ridge point to narrow the likely bottleneck; profiling then accounts for latency, occupancy, communication, and implementation overhead. The result is a testable hypothesis, not a prediction.
What’s Next: From optimization to validation
The ML optimization stack rests on three core efficiency pillars: data selection (Data Selection) minimizes training sample requirements, model compression (Model Compression) reduces algorithmic parameter and precision complexity, and hardware acceleration (developed in this chapter) maximizes physical machine throughput. Optimization without empirical measurement, however, remains incomplete. Benchmarking moves from theoretical FLOP bounds to measured wall-clock latency, applying standardized benchmark suites and statistical protocols to test optimization claims against operational reality.
Self-Check: Question
What is the central architectural insight of the hardware acceleration chapter regarding the interaction between model architecture and accelerator efficiency?
- Accelerators will soon eliminate all memory hierarchies in favor of infinite register files
- Hardware efficiency depends entirely on maximizing clock frequency regardless of memory bandwidth
- High delivered hardware efficiency requires co-design across computational primitives, dataflow reuse strategies, memory hierarchies, and compiler-runtime systems
- General-purpose out-of-order CPUs remain superior to specialized TPUs for all deep learning workloads
Summarize how the Roofline model serves as a unified diagnostic bridge connecting high-level neural network operations to low-level hardware architecture choices.
Which combination correctly summarizes the primary function of each layer in the modern AI hardware acceleration software stack?
- Framework: Silicon manufacturing; Compiler: Host PCIe routing; Runtime: Floating-point unit logic
- Framework: Direct transistor clocking; Compiler: Operating system page fault handling; Runtime: Mathematical differentiation
- Compiler: Real-time sensor power regulation; Runtime: Neural network gradient backpropagation; Hardware: Python interpreter dispatch
- Framework: Graph definition and automatic differentiation; Compiler: Graph optimization, operator fusion, and tiling; Runtime: Memory pooling, stream scheduling, and kernel dispatch; Hardware: Parallel matrix/vector execution
Self-Check Answers
Self-Check: Answer
What primary physical limitation brought about the end of Dennard scaling in the mid-2000s, necessitating the transition from increasing CPU clock frequencies to domain-specific hardware accelerators?
- Lithography light diffraction preventing further reduction of transistor gate length below \(1\,\mu\text{m}\)
- Inability to lower operating voltage proportionally with transistor size, leading to unsustainable power density and heat dissipation limits
- Quantum tunneling in copper interconnect lines preventing data transmission between arithmetic units
- Depletion of global silicon substrate supplies requiring migration to gallium nitride semiconductors
Answer: The correct answer is B. Dennard scaling postulated that as transistors shrank, operating voltage could drop in proportion, keeping chip power density constant while operating frequencies increased. When threshold voltage and leakage currents prevented further voltage scaling around 2005, increasing clock frequency resulted in prohibitive power density and thermal dissipation limits (the power wall and ‘dark silicon’), forcing architects to seek energy efficiency through architectural specialization and parallelism. The lithography choice misidentifies lithography limits as the cause of the mid-2000s frequency stall. The quantum tunneling choice confuses gate oxide leakage with bulk copper transmission. The semiconductor depletion choice is physically fabricated.
Learning Objective: Explain the physical breakdown of Dennard scaling that led to domain-specific acceleration.
Contrast the architectural trade-offs of software-managed scratchpad memory (such as Google TPUv1’s Unified Buffer) with hardware-managed cache hierarchies when executing large-scale tensor workloads.
Answer: Software-managed scratchpads eliminate the hardware overhead of tag arrays, cache line state machines, coherence snooping, and dynamic replacement policies, allowing more silicon area and power to be dedicated to storage capacity and dense arithmetic units. When compilers can statically determine regular tensor access patterns, they can deterministically tile and stage data. However, scratchpads increase compiler complexity and software burden, and they perform poorly on irregular, dynamically indexed memory patterns that hardware caches manage automatically.
Learning Objective: Compare software-managed scratchpad memory with hardware-managed caches for deep learning workloads.
**Place the following computing milestones in chronological order (from earliest to most recent) as hardware evolved toward modern AI accelerators:
- Introduction of dedicated Tensor Cores and TPUs for deep learning matrix operations
- Emergence of fixed-function media codecs and network processors for video/packet streaming
- Integration of floating-point units (FPUs) and digital signal processors (DSPs) as discrete coprocessors
- General-purpose programmable GPUs and SIMD instruction set extensions for 3D graphics and multimedia**
Answer: The correct order is (3) Integration of floating-point units (FPUs) and digital signal processors (DSPs) as discrete coprocessors -> (4) General-purpose programmable GPUs and SIMD instruction set extensions for 3D graphics and multimedia -> (2) Emergence of fixed-function media codecs and network processors for video/packet streaming -> (1) Introduction of dedicated Tensor Cores and TPUs for deep learning matrix operations. During the 1980s, discrete FPUs and DSPs accelerated arithmetic. The 1990s introduced SIMD extensions and early 3D GPUs. The 2000s expanded fixed-function media engines and network processors. The 2010s ushered in deep learning DSAs like Google TPUs (2015/2017) and NVIDIA Volta Tensor Cores (2017).
Learning Objective: Explain the historical evolution of hardware specialization from FPUs to modern AI accelerators.
True or False: Google’s TPUv1 achieved substantial performance-per-watt improvements over contemporary general-purpose CPUs primarily by operating at significantly higher clock frequencies.
Answer: False. TPUv1 operated at a modest clock frequency (approximately \(700\text{ MHz}\)) and achieved \(30\times\text{--}80\times\) higher performance-per-watt than contemporary CPUs by omitting complex general-purpose features (such as out-of-order execution, branch prediction, and multi-level cache coherence) and dedicating silicon area to a massive \(256{\times}256\) INT8 systolic matrix multiplier and a large software-managed Unified Buffer.
Learning Objective: Evaluate the architectural source of energy efficiency in domain-specific accelerators like TPUv1.
In the context of hardware scaling, what phenomenon does the ‘Systems Gap’ describe since the 2012 deep learning breakthrough?
- The difference in memory bandwidth between high-end datacenter GPUs and consumer-grade mobile SoCs
- The latency discrepancy between on-chip SRAM access times and host DRAM access times over PCIe
- The exponential divergence between model compute demand (growing \(\approx 6\times\)/year) and single-device hardware supply (growing \(\approx 1.7\times\)/year)
- The mismatch between Python framework dispatch overhead and raw GPU kernel execution duration
Answer: The correct answer is C. The Systems Gap refers to the widening divergence between the computational demand of frontier AI models (growing at roughly \(6\times\) per year according to scaling law trends) and single-accelerator hardware performance growth (growing at roughly \(1.7\times\) per year under Huang’s Law / architectural scaling). Closing this massive exponential gap requires distributed parallelism, system co-design, and algorithmic optimizations. The mobile vs datacenter memory bandwidth choice describes edge heterogeneity rather than macro compute trends. The SRAM vs DRAM latency choice describes the physical memory hierarchy gap. The Python dispatch vs kernel duration choice describes host runtime overhead.
Learning Objective: Analyze the Systems Gap and its implications for AI systems co-design.
Self-Check: Answer
When executing a Transformer layer containing attention projection (\(Q = X W_q\)), Softmax (\(\text{Softmax}(S)\)), Layer Normalization (\(\text{LayerNorm}(X)\)), and GeLU activation (\(\text{GeLU}(Z)\)), which execution unit is specifically responsible for computing transcendental functions (exponential and error function approximations)?
- Systolic 2D Matrix Multiply Units
- Dense Tensor Cores
- Vector Load-Store Memory Controllers
- Special Function Units (SFUs)
Answer: The correct answer is D. Special Function Units (SFUs) are specialized hardware pipelines designed to evaluate transcendental and nonlinear mathematical functions—such as exponentials in Softmax, reciprocal square roots in LayerNorm, and Gaussian error approximations in GeLU—using hardware lookup tables and polynomial approximations. Matrix units and Tensor Cores are dedicated to high-throughput matrix multiply-accumulate operations, not transcendental approximations. Vector Load-Store Memory Controllers handle memory address generation and data movement across caches and DRAM.
Learning Objective: Classify deep learning mathematical operations to their corresponding hardware execution units.
Explain how the \(\text{im2col}\) (image-to-column) transformation allows standard 2D convolution operations to execute on high-throughput matrix multiplication hardware (GEMM engines), and describe the primary memory overhead associated with this approach.
Answer: The \(\text{im2col}\) transformation extracts overlapping local receptive field patches from an input tensor and flattens each patch into a column (or row) of a 2D matrix, while flattening the convolution filters into a weight matrix. This converts the sliding-window convolution into a standard GEMM (\(C = A \cdot B\)) that executes efficiently on Tensor Cores or systolic arrays. However, because receptive fields overlap, input pixel values are duplicated across multiple patches, expanding memory footprint by up to roughly the filter area (\(K_h \times K_w\)) unless implemented via implicit-GEMM address calculation without physical materialization.
Learning Objective: Explain the mechanism and memory trade-offs of the im2col transformation for accelerating convolutions.
True or False: In modern AI accelerators, element-wise vector operations such as residual additions (\(Y = X_1 + X_2\)) achieve higher arithmetic intensity than large matrix-matrix multiplications (\(C = A \cdot B\)).
Answer: False. Element-wise vector operations perform only one arithmetic operation per two operand reads and one write (arithmetic intensity \(\approx 1/12 \text{ FLOP/byte}\) in FP32), making them heavily memory bandwidth bound. In contrast, an \(N \times N\) matrix-matrix multiplication performs \(\mathcal{O}(N^3)\) operations on \(\mathcal{O}(N^2)\) data elements, allowing extensive data reuse in on-chip registers and shared memory to achieve high arithmetic intensity (\(\mathcal{O}(N) \text{ FLOP/byte}\)).
Learning Objective: Compare the arithmetic intensity of element-wise vector operations with matrix multiplications.
The hardware transformation technique that enables convolutional layers to execute as matrix multiplications without physically duplicating overlapping patch data in memory is known as ____ GEMM.
Answer: implicit. Implicit GEMM computes input tensor memory addresses on-the-fly inside kernel index calculations during tile loading, avoiding the large \(\mathcal{O}(K_h \cdot K_w)\) memory expansion of explicit im2col materialization while retaining matrix-unit execution efficiency.
Learning Objective: Explain how implicit GEMM eliminates memory duplication overhead in convolution acceleration.
Which of the following operations in a modern deep learning architecture exhibits the highest operational arithmetic reuse, making it most suitable for dense 2D systolic arrays and Tensor Cores?
- Batched linear layer matrix multiplication (\(Y = X W\))
- Element-wise ReLU activation (\(\max(0, x)\))
- Channel-wise Batch Normalization mean computation
- Token-wise embedding table lookup
Answer: The correct answer is A. Batched linear layers multiply an activation matrix by a weight matrix, enabling each weight element to be reused across all batch elements and each activation element to be reused across multiple output features, yielding high arithmetic intensity suitable for 2D matrix units. Element-wise ReLU performs only one comparison per element load. Batch Normalization mean computation performs reduction passes over data with minimal arithmetic reuse per byte transferred. Embedding table lookups are gather operations with zero arithmetic operations per byte fetched.
Learning Objective: Classify neural network layers based on operational reuse and hardware execution affinity.
Self-Check: Answer
In NVIDIA’s Ampere and Hopper architectures, how does 2:4 structured sparsity achieve an up to \(2\times\) theoretical speedup in Tensor Core matrix multiplication?
- Exactly two non-zero values are preserved in every contiguous four-element block, allowing weights to be stored in half the memory with 2-bit index metadata while sparse Tensor Cores perform math only on non-zeros
- Every alternate row of the weight matrix is dropped completely, allowing the GPU to halve the grid launch dimensions
- Four separate threads simultaneously execute one scalar multiply-accumulate instruction in a single clock cycle
- Floating-point numbers are converted to 2-bit integers, quadrupling register file capacity
Answer: The correct answer is A. The 2:4 structured sparsity pattern mandates that exactly two out of every four contiguous values in a weight tensor are non-zero. The compressed weight matrix stores only the two non-zero values per 4-element block along with 2-bit index metadata (4 bits total per 4 values), halving memory storage and bandwidth requirements, while Sparse Tensor Cores use the metadata to select matching activations and compute matrix multiply-accumulate operations at twice the throughput of dense Tensor Cores. The row-dropping choice describes coarse block pruning rather than fine-grained structured sparsity. The four-thread SIMT scalar choice confuses thread scheduling with tensor hardware sparsity mechanics. The 2-bit integer conversion choice describes extreme quantization rather than sparsity.
Learning Objective: Explain the mechanics and hardware benefits of 2:4 structured sparsity in Tensor Cores.
Contrast the data movement mechanics and primary use cases of Weight-Stationary (WS) and Output-Stationary (OS) systolic array dataflows.
Answer: In a Weight-Stationary dataflow, model weights are loaded into local PE registers and held stationary while input activations and partial sums stream through the array, maximizing weight reuse and making it optimal for CNNs where small filter weights are reused extensively across spatial dimensions. In an Output-Stationary dataflow, partial sums remain stationary in PE accumulators while weights and inputs stream through, eliminating intermediate memory traffic for partial sum write-backs and making it optimal for large-batch matrix multiplications with high accumulation depth.
Learning Objective: Compare Weight-Stationary and Output-Stationary systolic array dataflow strategies.
**Place the following steps in the correct execution sequence for processing a matrix multiplication on an accelerator with Sparse Tensor Cores using 2:4 structured sparsity:
- Fine-tune or prune the weight matrix to ensure exactly two non-zero values exist in every four-element contiguous group
- Sparse Tensor Cores load compressed weights and decode metadata to gather matching input activation elements
- Multiply non-zero weights by gathered activations and accumulate into output partial sums at \(2\times\) dense throughput
- Compress the sparse weight matrix by storing only the non-zero values alongside 2-bit per-value selection metadata**
Answer: The correct order is (1) Fine-tune or prune the weight matrix to ensure exactly two non-zero values exist in every four-element contiguous group -> (4) Compress the sparse weight matrix by storing only the non-zero values alongside 2-bit per-value selection metadata -> (2) Sparse Tensor Cores load compressed weights and decode metadata to gather matching input activation elements -> (3) Multiply non-zero weights by gathered activations and accumulate into output partial sums at \(2\times\) dense throughput. Structured sparsity begins with pruning/fine-tuning to satisfy the 2:4 constraint, followed by offline compression and metadata creation. At runtime, hardware loads compressed data, uses metadata to multiplex input activations, and computes the sparse GEMM.
Learning Objective: Apply the end-to-end execution workflow of 2:4 structured sparse matrix multiplication.
What is the primary difference between the FP8 E4M3 and FP8 E5M2 numerical formats used in modern AI accelerators (such as NVIDIA Hopper and Ada Lovelace)?
- E4M3 uses 4 sign bits and 3 exponent bits, whereas E5M2 uses 5 sign bits and 2 exponent bits
- E4M3 has 4 exponent bits and 3 mantissa bits providing higher precision for forward-pass activations/weights, whereas E5M2 has 5 exponent bits and 2 mantissa bits providing wider dynamic range for gradients
- E4M3 is exclusively an integer fixed-point format, whereas E5M2 is a standard IEEE floating-point format
- E4M3 requires twice as many memory bytes per element as E5M2
Answer: The correct answer is B. Both formats occupy 8 bits (1 sign bit + exponent + mantissa). FP8 E4M3 allocates 4 exponent bits and 3 mantissa bits, providing higher precision (lower rounding error) suitable for forward-pass weights and activations where values are well-scaled. FP8 E5M2 allocates 5 exponent bits (matching FP16 dynamic range) and 2 mantissa bits, offering a wider dynamic range essential for preventing underflow in backward-pass gradients. The sign bit claim is incorrect as floating-point formats use a single sign bit. The integer format claim is incorrect as both are floating-point representations. The byte size claim is incorrect because both are 8-bit (1-byte) formats.
Learning Objective: Compare the numerical properties and intended use cases of FP8 E4M3 and E5M2 formats.
True or False: In NVIDIA’s SIMT (Single Instruction, Multiple Threads) execution model, when threads within the same 32-thread warp execute divergent branches of an
if-elsecondition, both paths are executed concurrently in parallel at full hardware throughput.Answer: False. When threads within a warp diverge on conditional branches, the warp executes each branch path serially while masking off (disabling) threads that do not take that path. The total execution time becomes the sum of the times of both paths, reducing hardware utilization and throughput.
Learning Objective: Analyze the performance penalty of warp divergence in SIMT architectures.
Describe the Tiling Principle in deep learning hardware mapping and explain why multi-level hierarchical tiling (from global memory down to registers) is necessary for high-throughput GEMM kernels.
Answer: The Tiling Principle partitions large matrix multiplication loops into smaller, block-sized submatrices that fit into each successive level of the hardware memory hierarchy (HBM \(\to\) shared memory/SRAM \(\to\) register files). Hierarchical tiling ensures that data loaded from high-latency, limited-bandwidth memory (such as HBM) is reused dozens of times in high-bandwidth, low-latency on-chip storage before being evicted, preventing arithmetic pipelines from stalling on memory bandwidth.
Learning Objective: Explain how hierarchical tiling maximizes data reuse and sustains peak compute utilization.
Self-Check: Answer
An accelerator delivers \(R_{\text{peak}} = 1{,}000\text{ TFLOP/s}\) (\(10^{15}\text{ FLOP/s}\)) in FP16 and has a High-Bandwidth Memory (HBM) subsystem delivering \(\text{BW} = 2.0\text{ TB/s}\) (\(2\times 10^{12}\text{ bytes/s}\)). What is the hardware balance point (ridge point \(I_{\text{ridge}}\)) of this system?
- \(50\text{ FLOP/byte}\)
- \(200\text{ FLOP/byte}\)
- \(500\text{ FLOP/byte}\)
- \(2{,}000\text{ FLOP/byte}\)
Answer: The correct answer is C. The hardware ridge point is defined as \(I_{\text{ridge}} = \frac{R_{\text{peak}}}{\text{BW}} = \frac{10^{15}\text{ FLOP/s}}{2.0 \times 10^{12}\text{ bytes/s}} = 500\text{ FLOP/byte}\). Any kernel with an arithmetic intensity below \(500\text{ FLOP/byte}\) will be memory-bandwidth bound on this hardware, whereas kernels with arithmetic intensity above \(500\text{ FLOP/byte}\) can potentially achieve peak compute throughput. The other choices result from arithmetic miscalculations (\(50\), \(200\), or \(2{,}000\)).
Learning Objective: Calculate the hardware ridge point given peak compute throughput and memory bandwidth.
When training a 7-billion parameter model using standard mixed-precision (FP16/BF16) with the Adam optimizer, calculate the minimum memory required purely for model states (weights, gradients, and optimizer states) and explain why optimizer states dominate this footprint.
Answer: Model states require 16 bytes per parameter: 2 bytes for FP16 weights, 2 bytes for FP16 gradients, 4 bytes for FP32 master weights, 4 bytes for FP32 first momentum, and 4 bytes for FP32 second momentum (\(2 + 2 + 4 + 4 + 4 = 16\text{ bytes/param}\)). For a 7B model, this requires \(7 \times 10^9 \times 16\text{ bytes} = 112\text{ GB}\). Optimizer states dominate (\(12\text{ bytes/param}\), or \(75\%\) of the total) because maintaining FP32 precision for master weights and running statistical moments is numerically necessary to prevent gradient underflow and truncation errors during updates.
Learning Objective: Calculate and justify the memory footprint of model weights, gradients, and optimizer states during mixed-precision training.
**Arrange the following levels of a modern GPU memory hierarchy in order of access latency, from lowest latency (fastest) to highest latency (slowest):
- High-Bandwidth Memory (HBM3)
- Register File
- Pinned Host System Memory (DDR5 via PCIe)
- Shared Memory / L1 Cache
- On-Chip L2 Cache**
Answer: The correct order is (2) Register File -> (4) Shared Memory / L1 Cache -> (5) On-Chip L2 Cache -> (1) High-Bandwidth Memory (HBM3) -> (3) Pinned Host System Memory (DDR5 via PCIe). Registers are accessible in sub-nanosecond/single-cycle latency, followed by on-chip Shared Memory/L1 (~few cycles), L2 Cache (~tens of cycles), device HBM (~hundreds of cycles), and finally host DDR over the PCIe bus (~microseconds/thousands of cycles).
Learning Objective: Classify memory hierarchy levels by access latency and proximity to execution units.
Why does High-Bandwidth Memory (HBM) achieve significantly higher bandwidth (e.g., \(>2\text{ TB/s}\)) than traditional GDDR6X memory (e.g., \(\approx 760\text{ GB/s}\)) while maintaining comparable or lower power per bit?
- HBM operates at a \(10\times\) higher clock frequency than GDDR6X on standard PCB traces
- HBM uses optical photonic signaling to transmit data across the motherboard
- HBM eliminates all error correction codes and row buffer precharge cycles
- HBM vertically stacks DRAM dies using Through-Silicon Vias (TSVs) and connects to the GPU via a wide 1024-bit per stack silicon interposer bus at lower clock speeds
Answer: The correct answer is D. HBM achieves ultra-high bandwidth by vertically stacking DRAM dies with Through-Silicon Vias (TSVs) and interfacing with the accelerator through a silicon interposer with extremely wide memory buses (1024 bits per stack vs. 32/64 bits for standard GDDR channels). Because physical wire distances over the silicon interposer are short and the bus is wide, HBM can run at lower pin clock frequencies, significantly reducing energy per bit (\(pJ/\text{bit}\)) compared to driving high-frequency GDDR signals over long PCB traces. The higher clock frequency choice is incorrect because GDDR6X actually runs at higher pin clock frequencies than HBM. The optical photonics and error-correction elimination choices are physically false.
Learning Objective: Explain the architectural design differences and physical advantages of HBM over GDDR.
Memory allocated on the host CPU that is locked into physical RAM and prevents operating system paging, enabling direct DMA transfers over PCIe to the GPU, is called ____ host memory.
Answer: pinned (or page-locked). Pinned host memory allows GPU DMA engines to transfer data asynchronously across the PCIe bus without CPU staging copies or page faults.
Learning Objective: Explain the role of pinned host memory in optimizing host-to-accelerator data transfers.
True or False: The AI Memory Wall refers solely to the limited physical capacity (GBs) of GPU DRAM, meaning that if an accelerator has sufficient gigabytes to store model weights, memory bandwidth will never bottleneck execution.
Answer: False. The AI Memory Wall encompasses both capacity and bandwidth disparities. While capacity determines whether a model fits on a chip, memory bandwidth (\(\text{TB/s}\)) dictates how fast data can be fed to compute units. Even if a model fits entirely in DRAM, low-arithmetic-intensity kernels (such as LayerNorm or token-by-token decoding) remain heavily bottlenecked by memory bandwidth, underutilizing peak compute throughput.
Learning Objective: Analyze the dual dimensions (capacity and bandwidth) of the AI memory wall.
Self-Check: Answer
A developer runs a LayerNorm kernel on an accelerator with \(R_{\text{peak}} = 312\text{ TFLOP/s}\) and \(\text{BW} = 1.5\text{ TB/s}\) (\(I_{\text{ridge}} = 208\text{ FLOP/byte}\)). The LayerNorm has an arithmetic intensity of \(I = 4\text{ FLOP/byte}\). What is the maximum attainable performance of this kernel, and what is the binding bottleneck?
- \(6.0\text{ TFLOP/s}\), bound by memory bandwidth
- \(312\text{ TFLOP/s}\), bound by peak compute capacity
- \(78\text{ TFLOP/s}\), bound by warp scheduler instruction issue rate
- \(1.5\text{ TFLOP/s}\), bound by PCIe bus transfer limits
Answer: The correct answer is A. According to the Roofline model, \(\text{Attainable Performance} = \min(R_{\text{peak}}, I \cdot \text{BW}) = \min(312\text{ TFLOP/s}, 4\text{ FLOP/byte} \times 1.5\text{ TB/s}) = \min(312, 6.0) = 6.0\text{ TFLOP/s}\). Because \(I = 4 < I_{\text{ridge}} = 208\text{ FLOP/byte}\), the kernel operates deep within the memory-bound regime, achieving less than \(2\%\) of the chip’s peak arithmetic throughput. The peak compute choice incorrectly assumes compute limits apply regardless of arithmetic intensity. The instruction issue and PCIe bus choices confuse the primary memory bandwidth limit with secondary bottlenecks.
Learning Objective: Calculate attainable performance using the Roofline model and identify the binding bottleneck.
Why has the hardware ridge point (\(I_{\text{ridge}}\)) increased dramatically across successive GPU generations (e.g., from Volta to Ampere to Hopper), and what pressure does this trend place on compiler and kernel developers?
Answer: The ridge point \(I_{\text{ridge}} = R_{\text{peak}}/\text{BW}\) has increased because peak arithmetic throughput (driven by specialized Tensor Cores and lower-precision FP8/INT8 math) has grown much faster than physical HBM bandwidth. This shifts the knee of the roofline curve to the right, meaning kernels require significantly higher arithmetic intensity to achieve compute-bound peak efficiency. Developers and compilers are forced to implement aggressive kernel fusion, multi-level tiling, and activation caching to avoid being trapped in the memory-bound regime.
Learning Objective: Analyze the historical trend of increasing hardware ridge points and its architectural implications.
True or False: When a kernel operates in the memory-bound regime of the Roofline model (\(I < I_{\text{ridge}}\)), doubling the accelerator’s peak tensor compute capability (\(R_{\text{peak}}\)) without changing memory bandwidth will double the kernel’s execution speed.
Answer: False. In the memory-bound regime, performance is capped by \(I \cdot \text{BW}\), which depends strictly on arithmetic intensity and memory bandwidth. Increasing \(R_{\text{peak}}\) only raises the horizontal compute ceiling; the kernel’s execution time is dictated by the slanted bandwidth ceiling and will experience \(0\%\) speedup unless memory bandwidth is increased or the kernel is restructured to improve data reuse.
Learning Objective: Evaluate the effect of hardware upgrades on memory-bound workloads using the Roofline model.
In the Roofline model, the transition point on the horizontal axis where the memory-bandwidth ceiling intersects the peak-compute ceiling is known as the hardware ____ point.
Answer: ridge. The ridge point (\(I_{\text{ridge}} = R_{\text{peak}} / \text{BW}\)) defines the minimum arithmetic intensity required for an algorithm to potentially reach peak hardware computational throughput.
Learning Objective: Explain the significance of the ridge point in roofline performance modeling.
An engineer profiles a transformer inference workload and discovers that the attention Softmax kernel is heavily memory bandwidth bound. Which of the following optimization techniques directly increases arithmetic intensity to move the kernel closer to the compute-bound regime?
- Upgrading host CPU RAM to DDR5 to decrease kernel enqueue latency
- Fusing the scale, mask, Softmax, and dropout operations into a single kernel to keep intermediate activations in registers/SRAM
- Increasing the clock frequency of the GPU Tensor Cores by \(15\%\)
- Disabling warp scheduler out-of-order instruction issue
Answer: The correct answer is B. Fusing scale, mask, Softmax, and dropout into a single kernel eliminates intermediate writes to and reads from DRAM (HBM), keeping intermediate values in fast on-chip registers and shared memory. This drastically reduces the total memory traffic \(D_{\text{vol}}\), increasing the arithmetic intensity \(I = \text{FLOP}/\text{Byte}\) and moving the kernel closer to or into the compute-bound regime. Upgrading host CPU RAM does not affect GPU HBM memory traffic. Increasing Tensor Core clock frequency only raises the peak compute ceiling without improving memory-bound throughput. Disabling warp scheduling hurts instruction issue efficiency.
Learning Objective: Design optimization strategies to shift memory-bound kernels toward the compute-bound regime.
Self-Check: Answer
In neural network hardware mapping, what distinguishes a spatial mapping decision from a temporal mapping decision?
- Spatial mapping refers to compiling graph IR, whereas temporal mapping refers to runtime CUDA kernel launches
- Spatial mapping determines precision formats (FP16 vs INT8), whereas temporal mapping determines memory allocation sizes
- Spatial mapping assigns computational tasks to specific physical execution units (e.g., PEs or SMs) simultaneously in parallel, whereas temporal mapping determines the execution ordering and loop scheduling over time on those units
- Spatial mapping operates only on convolutional layers, whereas temporal mapping operates only on transformer attention layers
Answer: The correct answer is C. Spatial mapping decides how tensor dimensions and parallel operations are partitioned and assigned across physical execution resources (such as array PEs, GPU SMs, or SIMD vector lanes) to execute concurrently in space. Temporal mapping decides the chronological schedule, loop ordering, and time steps in which operations and tile iterations execute on those assigned physical units over time. The compiler IR vs runtime launch distinction confuses compilation phases with mapping dimensions. The precision vs memory allocation choice describes quantization and memory planning. The layer-specific choice is incorrect as both mapping types apply to all neural network operations.
Learning Objective: Compare spatial mapping and temporal mapping dimensions in AI hardware acceleration.
Explain why finding the optimal hardware mapping (tiling sizes, loop orders, and spatial partitioning) for a deep neural network on a target accelerator is a combinatorially hard optimization problem.
Answer: Mapping involves searching a discrete combinatorial space of loop transformations across multi-dimensional tensor operations. For an \(M\)-deep nested loop across \(K\) memory hierarchy levels, there are \(M!\) possible loop orderings at each level, multiplied by all possible tile factor combinations that divide loop bounds, unrolling factors, and spatial allocation strategies across processing elements. Furthermore, memory capacity constraints at each tier create complex non-linear dependencies, making exhaustive evaluation computationally intractable and requiring heuristic or learning-based search spaces.
Learning Objective: Analyze the combinatorial complexity of hardware mapping for neural network compilation.
Explain why reordering loop nests in a tensor contraction (e.g., changing from \(I \to J \to K\) to \(K \to I \to J\)) alters memory bandwidth demands and execution speed without changing the total mathematical operation count.
Answer: Loop reordering preserves the mathematical invariants and total scalar multiply-accumulate count (\(\mathcal{O}(I \cdot J \cdot K)\) operations) but fundamentally changes the data access patterns and residency in cache and scratchpad memory tiers. The inner loop dictates which tensor operand (\(A, B,\) or partial sum \(C\)) is held stationary in registers or fast SRAM while other operands stream through. Selecting a suboptimal loop ordering causes cache thrashing and repeated DRAM round-trips for the streamed operands, increasing memory bandwidth demand and drastically degrading execution performance.
Learning Objective: Explain why loop reordering impacts memory bandwidth demand without altering total arithmetic operations.
When mapping a tensor computation to a multi-level memory hierarchy, what is the primary objective function optimized by spatial and temporal tiling?
- Maximizing the total number of intermediate tensors written to host DDR memory
- Maximizing data reuse in the fastest, closest memory tiers (registers and SRAM) to minimize traffic to slower, energy-expensive DRAM
- Ensuring every warp thread executes different instruction streams simultaneously
- Converting all 2D matrix multiplications into 1D scalar operations
Answer: The correct answer is B. The primary goal of spatial and temporal mapping is to maximize data reuse in the highest levels of the memory hierarchy (registers and on-chip SRAM/shared memory), ensuring that each byte fetched from power-hungry, high-latency DRAM is reused as many times as possible before eviction, thereby minimizing DRAM bandwidth demand and maximizing compute throughput. Maximizing writes to host memory degrades performance. Executing different instructions causes severe warp divergence. Converting matrix operations to scalars eliminates vector and tensor unit acceleration.
Learning Objective: Justify the primary objective of tiling and memory hierarchy mapping.
Self-Check: Answer
Why do modern deep learning libraries (e.g., cuDNN, TensorRT) strongly prefer the Channels-Last (NHWC) tensor layout over Channels-First (NCHW) when executing convolutions on NVIDIA Tensor Cores?
- NHWC eliminates the need for spatial convolutions by flattening images into 1D vectors
- NCHW requires floating-point numbers to be stored in big-endian byte order
- NHWC reduces model parameter count by sharing channel weights across batches
- NHWC places channel values for a given spatial location in contiguous memory, aligning with the packed vector/matrix multiply requirements of Tensor Cores
Answer: The correct answer is D. In NHWC layout, the channel dimension \(C\) is the fastest-varying (innermost) dimension, placing all channel features for a specific spatial coordinate \((n, h, w)\) contiguously in memory. Tensor Cores execute matrix operations on contiguous vectors of channels (e.g., 8 or 16 channels per memory vector load), enabling coalesced memory access and direct loading into matrix unit registers without expensive transpose or gather operations. The 1D flattening, big-endian, and parameter sharing choices are technically incorrect.
Learning Objective: Compare the memory layout efficiency of Channels-Last (NHWC) versus Channels-First (NCHW) for Tensor Cores.
Explain the mechanism of vertical kernel fusion (e.g., fusing
Conv2D\(\to\)BatchNorm\(\to\)ReLU), and identify both its performance benefit and its primary architectural constraint.Answer: Vertical kernel fusion combines a sequence of producer-consumer operations into a single GPU kernel, passing intermediate activation values directly through fast on-chip registers or shared memory instead of writing them out to and re-reading them from global DRAM (HBM). This dramatically reduces memory traffic and kernel launch overhead. Its primary architectural constraint is register pressure: storing intermediate variables and fusing complex pipelines increases register usage per thread, which can reduce warp occupancy and limit available parallelism on the SM.
Learning Objective: Explain the performance benefits and hardware constraints of vertical kernel fusion.
**Place the following loop transformation steps in the logical order applied by an optimizing compiler when targeting a matrix multiplication dataflow to an accelerator:
- Loop Reordering: Permute loop indices to establish a specific stationary dataflow (e.g., Output-Stationary)
- Loop Unrolling: Fully or partially unroll the innermost loop to expose instruction-level parallelism and map to hardware registers
- Loop Tiling (Blocking): Partition global loop iterations into sub-tiles that fit into on-chip shared memory / SRAM
- Spatial Partitioning: Assign outer tile loops to physical hardware compute clusters (e.g., GPU thread blocks / SMs)**
Answer: The correct order is (3) Loop Tiling (Blocking): Partition global loop iterations into sub-tiles that fit into on-chip shared memory / SRAM -> (4) Spatial Partitioning: Assign outer tile loops to physical hardware compute clusters (e.g., GPU thread blocks / SMs) -> (1) Loop Reordering: Permute loop indices to establish a specific stationary dataflow (e.g., Output-Stationary) -> (2) Loop Unrolling: Fully or partially unroll the innermost loop to expose instruction-level parallelism and map to hardware registers. Optimization starts with hierarchical tiling to match memory capacities, partitions tiles spatially across SMs/PEs, reorders the inner loops to minimize data movement (dataflow choice), and finally unrolls inner loops for instruction issue efficiency.
Learning Objective: Apply loop transformation pipelines for optimizing matrix dataflow on accelerators.
In a multi-head self-attention layer where a single input activation tensor is projected across multiple query, key, and value weight matrices (\(W_q, W_k, W_v\)), which stationary dataflow strategy provides the highest data reuse in local scratchpad memory?
- Input-Stationary (Activation-Stationary)
- Output-Stationary
- No Local Reuse (NLR)
- Weight-Stationary
Answer: The correct answer is A. In an Input-Stationary (Activation-Stationary) dataflow, an input activation tile is loaded into fast local memory once and kept stationary while multiple weight matrices (\(W_q, W_k, W_v\)) stream through. This maximizes the reuse of the common activation tensor across multiple matrix multiplications, minimizing activation re-reads. Weight-Stationary would require repeatedly reloading activations for each distinct weight matrix. Output-Stationary minimizes accumulation write-backs but does not exploit cross-projection activation sharing. No Local Reuse streams all operands from memory without local caching.
Learning Objective: Design dataflow strategies for multi-head attention workloads.
True or False: Row-Stationary (RS) dataflow, as implemented in architectures like Eyeriss, keeps only the final output activation stationary in registers while streaming 2D convolutional filter rows and input rows from DRAM on every clock cycle.
Answer: False. Row-Stationary dataflow keeps 1D rows of convolutional weights stationary in PE registers, slides 1D rows of input activations through the PEs, and accumulates 1D rows of partial sums locally across a 2D array of PEs, maximizing 2D convolution spatial reuse across all three tensor components simultaneously.
Learning Objective: Evaluate the operational mechanics of Row-Stationary dataflow in convolution accelerators.
Self-Check: Answer
What is the primary purpose of lifetime analysis in an ML compiler’s static memory planner?
- To calculate the physical degradation and failure rate of HBM memory cells over time
- To determine the precise intervals during which each intermediate activation tensor is needed, allowing disjoint tensors to share the same physical memory buffer
- To predict the number of training epochs required for a neural network to converge
- To prevent the compiler from generating out-of-order instruction streams
Answer: The correct answer is B. Lifetime analysis tracks when each intermediate tensor is created (produced) and last referenced (consumed) during graph execution. By identifying tensors whose lifetimes do not overlap, the compiler’s static memory planner can assign them to the same physical memory addresses (buffer reuse/aliasing), significantly reducing the model’s peak runtime memory footprint and avoiding dynamic memory allocation overhead. The physical degradation choice confuses tensor lifecycle with semiconductor physics. The training convergence choice describes algorithmic optimization rather than compiler memory management. The instruction stream choice describes scheduling.
Learning Objective: Explain the role of tensor lifetime analysis in static memory planning.
Explain how double buffering (software pipelining) implemented by an ML compiler hides memory access latency during loop execution on an accelerator.
Answer: Double buffering allocates two alternating memory buffers in on-chip SRAM/shared memory: while compute units execute arithmetic operations on data in the first buffer (tile \(k\)), asynchronous DMA or copy engines concurrently load subsequent data (tile \(k+1\)) into the second buffer from DRAM. In the next iteration, the roles swap. By overlapping data transfer with arithmetic computation, memory transfer latency is completely hidden as long as the compute time equals or exceeds the transfer time.
Learning Objective: Explain the mechanism of double buffering in hiding memory transfer latency.
**Place the following compilation stages in the correct order as an end-to-end ML compiler (such as TVM or XLA) transforms a high-level deep learning model into executable machine code:
- Target Code Generation: Emit hardware-specific binary (e.g., PTX or machine instructions)
- High-Level Graph Optimization: Perform operator fusion, constant folding, and dead code elimination on the computation graph
- Front-End Ingestion: Parse framework model (e.g., PyTorch/ONNX) into High-Level Graph IR
- Low-Level IR & Auto-Tuning: Lower fused operators to loop-level IR and optimize tile sizes, thread bindings, and unroll factors
- Static Memory Planning: Analyze tensor lifetimes and allocate shared physical buffers**
Answer: The correct order is (3) Front-End Ingestion: Parse framework model (e.g., PyTorch/ONNX) into High-Level Graph IR -> (2) High-Level Graph Optimization: Perform operator fusion, constant folding, and dead code elimination on the computation graph -> (4) Low-Level IR & Auto-Tuning: Lower fused operators to loop-level IR and optimize tile sizes, thread bindings, and unroll factors -> (5) Static Memory Planning: Analyze tensor lifetimes and allocate shared physical buffers -> (1) Target Code Generation: Emit hardware-specific binary (e.g., PTX or machine instructions). Compilation begins with parsing graph IR, performs target-independent graph optimizations, lowers to loop IR for hardware auto-tuning and tiling, plans memory buffers, and generates native device code.
Learning Objective: Apply the multi-stage compilation pipeline of modern ML compilers.
How does an auto-tuning ML compiler (such as TVM/Ansor) differ from a traditional handwritten library approach (such as cuDNN) for kernel selection?
- Auto-tuning compilers execute code only on the host CPU, whereas handwritten libraries run on GPUs
- Handwritten libraries search an infinite combinatorial loop space at runtime, whereas auto-tuning compilers use static heuristics
- Auto-tuning compilers explore large parameterized search spaces of loop transformations and tile sizes using cost models to generate custom kernels, whereas handwritten libraries rely on expert-tuned templates for specific fixed shapes
- Auto-tuning compilers require all tensors to be quantized to 1-bit integers
Answer: The correct answer is C. Auto-tuning ML compilers define parameterized spaces of loop transformations (tiling, unrolling, vectorization, thread mapping) and evaluate configurations using statistical cost models or hardware measurements to generate custom kernels optimized for any arbitrary tensor shape and target architecture. In contrast, vendor libraries rely on human experts who write and optimize specific kernel templates for common fixed shapes and precisions. The host CPU only claim is false. The runtime infinite search claim is reversed. The 1-bit quantization claim is unrelated.
Learning Objective: Compare auto-tuning ML compilers with handwritten vendor libraries.
The optimization where an ML compiler merges two or more independent operators at the same graph depth into a single batched kernel to maximize GPU parallelism is known as ____ fusion.
Answer: horizontal. Horizontal fusion combines independent parallel operators (such as parallel projection layers or multi-head \(Q, K, V\) linear transformations) into a single batched kernel, improving hardware occupancy and amortizing kernel launch overhead.
Learning Objective: Classify horizontal versus vertical kernel fusion techniques.
Self-Check: Answer
Why do deep learning frameworks implement custom caching memory allocators (such as PyTorch’s
caching_allocator) rather than directly invokingcudaMallocandcudaFreefor every intermediate tensor?cudaMallocoperates only in FP32 precision and cannot allocate FP16 memory buffers- Direct OS memory allocation encrypts all tensor data, introducing cryptographic decryption latency
- GPU DRAM can only be allocated once during system boot time
cudaMallocis a synchronous operation that stalls GPU execution and causes expensive driver and OS page-table synchronization overhead
Answer: The correct answer is D.
cudaMallocandcudaFreeare synchronous system calls that require driver synchronization, virtual memory address mapping, and device-wide mutex locking, introducing significant latency (\(\approx 10\text{--}100\,\mu\text{s}\)) that stalls asynchronous execution streams. A caching memory allocator pre-allocates large memory blocks and manages a userspace pool of memory chunks on the device, servicing tensor allocations and deallocations in sub-microsecond time without GPU-host synchronization. Precision is independent of raw memory allocation. OS encryption is not part of standard cudaMalloc. GPU memory can be dynamically allocated at runtime.Learning Objective: Analyze the architectural necessity and performance advantages of caching memory allocators in ML runtimes.
Explain how CUDA Graphs eliminate CPU kernel launch overhead during repeated training or inference iterations on small or low-latency models.
Answer: In standard execution, every GPU kernel requires the CPU to enqueue a launch request via the driver, incurring \(\approx 5\text{--}10\,\mu\text{s}\) of CPU overhead per kernel, which severely bottlenecks short-duration kernels. CUDA Graphs capture the entire directed acyclic graph (DAG) of kernel launches, memory copies, and stream dependencies during an initial trace pass. In subsequent iterations, the entire graph is instantiated and replayed via a single driver call, offloading execution sequencing entirely to the GPU hardware scheduler and eliminating host-side dispatch overhead.
Learning Objective: Explain how CUDA Graphs capture and replay execution workflows to eliminate host launch latency.
True or False: In an asynchronous GPU runtime model, when a Python script executes
y = torch.matmul(a, b), the CPU thread blocks and waits until the GPU hardware finishes computing the matrix multiplication before executing the next line of Python code.Answer: False. GPU runtime calls are asynchronous:
torch.matmulenqueues the kernel execution command onto a CUDA stream queue in the driver and immediately returns control to the CPU thread. The CPU continues executing subsequent Python instructions while the GPU executes the kernel in the background, blocking only when synchronization is explicitly requested (e.g., viatorch.cuda.synchronize()or copying data back to CPU host memory).Learning Objective: Analyze the non-blocking execution model of asynchronous GPU runtime streams.
A sequence of asynchronous GPU operations that execute strictly in FIFO order on an accelerator is called a CUDA ____.
Answer: stream. CUDA streams allow operations enqueued within the same stream to execute sequentially while operations in different streams can execute concurrently in parallel when hardware resources permit.
Learning Objective: Explain the role of CUDA streams in managing concurrent and asynchronous GPU execution.
An inference serving system processes dynamic batch sizes ranging from 1 to 32 tokens per request. Why does dynamic batching pose a significant challenge to runtime kernel selection and hardware efficiency?
- Optimal tile sizes, thread block configurations, and memory bandwidth requirements change across batch sizes, making a single static kernel inefficient across all shapes
- Dynamic batching forces the GPU to switch from FP16 to FP64 precision for even batch sizes
- Tensor Cores cannot execute matrix multiplication when batch dimensions are not powers of two
- The GPU memory controller must physically power down DRAM banks when batch size decreases
Answer: The correct answer is A. For small batch sizes (e.g., \(B=1\)), operations are memory-bandwidth bound and require kernels tuned for low latency and high memory throughput, whereas for large batch sizes (e.g., \(B=32\)), operations become compute bound and require larger tile sizes and high-occupancy thread block configurations to saturate Tensor Cores. A single static kernel configuration cannot achieve peak efficiency across disparate tensor shapes, requiring runtime kernel dispatch or multi-version code generation. Precision switching, power-of-two Tensor Core limitations, and DRAM bank power-downs are incorrect.
Learning Objective: Analyze the performance challenges of dynamic shapes on runtime kernel selection.
Self-Check: Answer
Why have accelerator architectures increasingly shifted from monolithic single-die designs toward Multi-Chip Module (MCM) and chiplet architectures?
- Chiplets eliminate all need for semiconductor fabrication foundries
- Monolithic dies are constrained by the physical lithography reticle limit (\(\approx 858\text{ mm}^2\)) and suffer exponential yield loss as die size increases
- Monolithic dies cannot support high-bandwidth memory (HBM) interfaces
- Chiplets allow electrical signals to travel faster than the speed of light
Answer: The correct answer is B. Monolithic silicon dies are physically bounded by optical lithography reticle limits (typically \(\approx 858\text{ mm}^2\)). Furthermore, manufacturing defect density causes wafer yield to drop exponentially as die area approaches the reticle limit, making large monolithic chips prohibitively expensive. Chiplet/MCM architectures break the system into smaller, high-yield dies interconnected via high-density silicon bridges or interposers, enabling much larger aggregate compute and memory capacity. The elimination of foundries is absurd; HBM is supported on monolithic dies (e.g., A100/H100); physical constants cannot be exceeded.
Learning Objective: Justify the transition from monolithic dies to chiplet-based accelerator architectures.
Explain how the non-uniform memory access (NUMA) effect and interconnect bandwidth degradation impact performance when scaling a neural network workload across multiple chiplets or accelerator chips.
Answer: While on-chip or intra-die memory access delivers high bandwidth (\(>2\text{--}3\text{ TB/s}\)) and low latency, crossing chiplet boundaries via inter-die bridges or board interconnects (e.g., NVLink or PCIe) suffers an order-of-magnitude reduction in bandwidth and increased latency. If a tensor computation frequently requires operands from remote chiplets without local caching, execution stalls on inter-chip communication, creating a NUMA bottleneck that degrades parallel scaling efficiency unless data is carefully partitioned.
Learning Objective: Analyze the NUMA and interconnect bandwidth constraints in multi-chip scaling.
Describe the architectural rationale behind Cerebras’s Wafer-Scale Engine and explain how fabricating an accelerator across an entire uncut silicon wafer overcomes traditional multi-chip scaling bottlenecks.
Answer: Standard multi-chip scaling suffers significant latency and bandwidth penalties when signals cross chip package boundaries and PCB traces. Cerebras fabricates hundreds of thousands of cores across an entire uncut \(300\text{ mm}\) silicon wafer, utilizing on-wafer routing to connect adjacent reticle fields. This provides uniform, ultra-high-bandwidth, single-cycle communication and massive distributed on-chip SRAM across the entire wafer, eliminating discrete chip packaging, external SerDes transceivers, and NUMA interconnect bottlenecks.
Learning Objective: Explain the architectural rationale and communication advantages of wafer-scale computing systems.
What is the primary bottleneck introduced by inter-node scaling (scaling out across separate servers over Ethernet or InfiniBand) compared to intra-node NVLink scaling?
- Inter-node network adapters cannot transmit FP16 or BF16 floating-point values
- Inter-node communication requires GPUs to switch to single-threaded CPU emulation mode
- Inter-node network bandwidth (e.g., \(400\text{ Gbps} \approx 50\text{ GB/s}\)) is roughly an order of magnitude lower than intra-node NVLink bandwidth (\(\approx 900\text{ GB/s}\)), increasing collective communication latency
- Inter-node scaling eliminates the need for gradient synchronization in distributed training
Answer: The correct answer is C. Intra-node GPU interconnects like NVLink 4 deliver up to \(900\text{ GB/s}\) of bidirectional bandwidth per GPU, whereas inter-node network interfaces (e.g., \(400\text{ Gbps}\) InfiniBand or RoCE NICs) deliver \(\approx 50\text{ GB/s}\) per port. This sharp bandwidth drop across the node boundary makes inter-node collective communication (such as All-Reduce or All-to-All) a dominant bottleneck in distributed scaling, requiring hierarchical communication algorithms. The network adapter precision and CPU emulation choices are false. The gradient synchronization elimination choice is incorrect because distributed data parallelism requires gradient synchronization.
Learning Objective: Compare the bandwidth constraints of intra-node interconnects with inter-node scale-out networks.
Self-Check: Answer
On a modern mobile heterogeneous System-on-Chip (SoC) featuring big.LITTLE CPUs, a mobile GPU, and a dedicated NPU, which compute engine is most energy-efficient for running continuous, low-latency 8-bit quantized convolutional inference?
- The high-performance ‘big’ CPU core running single-threaded FP32 instructions
- The out-of-order system memory controller
- The dedicated Neural Processing Unit (NPU) optimized for quantized INT8 matrix operations
- The host operating system virtualization hypervisor
Answer: The correct answer is C. The Neural Processing Unit (NPU) is a domain-specific accelerator with fixed-point MAC arrays, specialized activation units, and local scratchpad SRAM designed specifically for INT8/INT4 neural network operations. It delivers an order-of-magnitude higher energy efficiency (TOPS/Watt) than general-purpose CPU cores or high-power GPUs by avoiding instruction fetch/decode overhead and general register file transfers. The big CPU core consumes substantially more power per operation. Memory controllers and hypervisors do not execute tensor math.
Learning Objective: Classify mobile SoC compute engines based on workload efficiency and architectural specialization.
In automotive autonomous driving SoCs, explain why deterministic worst-case execution time and lockstep redundancy are required, even if they reduce peak average-case throughput.
Answer: Automotive systems are subject to strict functional safety standards (such as ISO 26262 ASIL-D), where missing a perception or control deadline in real-time sensor processing can lead to catastrophic physical collisions. Dual-core lockstep execution runs identical computations redundantly across replicated hardware cores to detect transient hardware faults and bit flips immediately. While these safety mechanisms and strict scheduling guarantees introduce hardware overhead and reduce peak average throughput, they guarantee deterministic worst-case latency and fault tolerance necessary for life-critical autonomy.
Learning Objective: Explain the architectural trade-offs between safety-critical determinism and peak throughput in automotive AI SoCs.
True or False: In mobile SoCs with unified system memory (LPDDR), sharing physical RAM between the CPU, GPU, and NPU eliminates all data movement overhead between heterogeneous processors.
Answer: False. While unified memory eliminates physical PCIe bus transfers and duplicate memory copies across discrete devices, data must still be moved across the shared on-chip interconnect and through different cache hierarchy domains. Furthermore, cache coherency protocols, differing tensor layout requirements (e.g., NCHW vs NHWC), and memory bus contention between competing SoC engines still incur significant latency and energy overhead.
Learning Objective: Evaluate the memory movement and cache coherency trade-offs of unified SoC memory.
Describe how Dynamic Voltage and Frequency Scaling (DVFS) and thermal throttling constrain sustained AI inference performance on edge and mobile devices.
Answer: Mobile SoCs operate within a tight thermal design power (TDP) envelope (typically \(3\text{--}5\text{ W}\)) with passive cooling. When continuous AI workloads generate sustained heat, the thermal management system triggers DVFS to lower operating voltage and clock frequencies, preventing device overheating. This thermal throttling causes execution time to degrade over time, meaning peak burst performance cannot be sustained for continuous video or audio streaming workloads without proactive power-budget scheduling.
Learning Objective: Analyze the impact of thermal throttling and DVFS on sustained edge AI inference.
Self-Check: Answer
In the lifecycle carbon assessment of advanced deep learning accelerators, what constitutes ‘embodied carbon’?
- The electrical energy consumed by the GPU during model forward and backward passes
- The carbon emitted by datacenter air conditioning units during peak summer load
- The carbon credits purchased by cloud providers to offset datacenter energy usage
- The greenhouse gas emissions generated during raw material extraction, semiconductor silicon manufacturing, packaging, and hardware transportation
Answer: The correct answer is D. Embodied carbon refers to the total greenhouse gas emissions generated throughout the supply chain and manufacturing lifecycle of the hardware before it ever runs a workload, including silicon ingot purification, advanced extreme ultraviolet (EUV) lithography, cleanroom fabrication, multi-chip packaging, assembly, and transportation. Operational carbon refers to emissions resulting from electricity consumed during active operation and cooling of the hardware. Carbon credits are financial offsets, not physical emissions.
Learning Objective: Compare embodied carbon and operational carbon in AI hardware lifecycle analysis.
Explain why maximizing accelerator utilization (e.g., via multi-tenant sharing or continuous pipeline saturation) reduces the amortized carbon footprint per trained model.
Answer: The total carbon footprint of a machine learning model includes both operational carbon (energy consumed per training step) and a fractional share of the hardware’s embodied carbon amortized over its operational lifespan. When accelerator utilization is low (e.g., GPUs idling on data stalls), the embodied carbon is wasted on idle time. Maximizing utilization ensures that more useful compute operations are extracted over the device’s operational lifetime, minimizing the embodied carbon cost allocated to each trained model.
Learning Objective: Explain how hardware utilization impacts the amortized carbon footprint of AI workloads.
True or False: In a datacenter with a Power Usage Effectiveness (PUE) of 1.1, the cooling and electrical distribution infrastructure consumes more power than the actual computing IT equipment (servers and accelerators).
Answer: False. \(\text{PUE} = \frac{\text{Total Facility Power}}{\text{IT Equipment Power}}\). A PUE of 1.1 means that for every \(1.0\text{ Watt}\) consumed by the computing IT equipment, only \(0.1\text{ Watts}\) (roughly \(9\%\) of total power) is consumed by cooling, lighting, and power distribution overhead, indicating a highly efficient facility where computing equipment dominates power consumption.
Learning Objective: Calculate and interpret Power Usage Effectiveness (PUE) for AI datacenters.
Self-Check: Answer
A team prunes \(70\%\) of the weights in a large language model using unstructured magnitude pruning, setting those weights to zero. However, when executing the pruned model on standard GPU dense Tensor Cores, inference latency is identical to or slower than the unpruned baseline. What is the primary cause of this pitfall?
- Standard dense hardware cannot skip individual zero elements without structured patterns (e.g., 2:4) or specialized sparse matrix indexing, so dense matrix units still execute all multiplications while sparse formats add indexing overhead
- Floating-point units automatically convert zero values into infinite loops
- Unstructured pruning forces the GPU driver to downclock memory bandwidth to prevent overheating
- The operating system kernel intercepts every zero multiplication and raises a hardware page fault
Answer: The correct answer is A. Dense Tensor Cores and SIMT pipelines execute instructions in lockstep on dense contiguous matrices. Unstructured zero values do not change the dense matrix dimensions, so standard hardware still loads and computes all values. If converted to general sparse formats (e.g., CSR), irregular non-coalesced memory access and index metadata decoding overhead negate any FLOP reduction on GPUs unless sparsity is extremely high (\(>90\text{--}95\%\)) or structured (such as 2:4). The infinite loop, driver downclocking, and OS page fault choices are physically incorrect.
Learning Objective: Explain why unstructured sparsity fails to accelerate execution on dense hardware architectures.
Describe the pitfall of ‘micro-offloading’ small tensor operations from CPU to GPU, and explain why a sequence of scalar operations can run slower on an accelerator than on the host CPU.
Answer: Offloading small tensor operations incurs PCIe host-to-device data transfer latency, host driver scheduling overhead, and kernel launch latency (\(\approx 5\text{--}10\,\mu\text{s}\) per kernel). When a tensor contains only a few dozen or hundred elements, computation time is sub-microsecond, meaning fixed transfer and launch overhead completely dominates execution time. Furthermore, small tensors cannot provide enough parallel work to saturate thousands of GPU cores, making scalar or vector execution on low-latency CPU caches much faster.
Learning Objective: Analyze the performance pitfalls of kernel launch and data transfer overhead in micro-offloading.
**Place the following diagnostic steps in the recommended sequence when troubleshooting an underperforming neural network training workload on an accelerator cluster:
- Roofline & Hardware Counter Analysis: Determine whether individual kernels are compute-bound, memory-bandwidth-bound, or latency-bound
- Amdahl & Host-Side Profiling: Identify serial bottlenecks, data loading stalls, and CPU-GPU synchronization delays
- Kernel-Level Optimization: Apply operator fusion, hierarchical tiling, or precision reduction targeted to the identified bottleneck
- Multi-Device Communication Profiling: Check for gradient all-reduce synchronization delays and interconnect saturation**
Answer: The correct order is (2) Amdahl & Host-Side Profiling: Identify serial bottlenecks, data loading stalls, and CPU-GPU synchronization delays -> (1) Roofline & Hardware Counter Analysis: Determine whether individual kernels are compute-bound, memory-bandwidth-bound, or latency-bound -> (3) Kernel-Level Optimization: Apply operator fusion, hierarchical tiling, or precision reduction targeted to the identified bottleneck -> (4) Multi-Device Communication Profiling: Check for gradient all-reduce synchronization delays and interconnect saturation. Diagnostics begin at the macro level (Amdahl unaccelerated fractions and host bottlenecks), zoom in to single-device kernel roofline profiling, optimize the binding kernel bottlenecks, and finally scale up to inter-device communication analysis.
Learning Objective: Apply a systematic diagnostic sequence to identify and resolve accelerator performance bottlenecks.
Which of the following statements represents a classic fallacy regarding peak accelerator specifications?
- High-bandwidth memory reduces the latency of memory-bound operations compared to standard DDR5
- An accelerator with \(2\times\) higher peak theoretical TFLOP/s will automatically deliver a \(2\times\) speedup on any neural network workload
- Kernel fusion can improve arithmetic intensity by reducing global memory traffic
- Warp divergence reduces the execution efficiency of SIMT processor lanes
Answer: The correct answer is B. Assuming that doubling peak theoretical TFLOP/s automatically doubles real-world workload performance is a classic fallacy. If the workload is memory-bandwidth bound (e.g., LayerNorm, embedding lookup, or autoregressive decoding), bound by host communication (Amdahl’s law), or limited by kernel launch overhead, increasing peak compute capacity produces negligible or zero speedup. The other choices represent well-established architectural facts.
Learning Objective: Evaluate common fallacies regarding peak accelerator hardware specifications.
Self-Check: Answer
What is the central architectural insight of the hardware acceleration chapter regarding the interaction between model architecture and accelerator efficiency?
- Accelerators will soon eliminate all memory hierarchies in favor of infinite register files
- Hardware efficiency depends entirely on maximizing clock frequency regardless of memory bandwidth
- High delivered hardware efficiency requires co-design across computational primitives, dataflow reuse strategies, memory hierarchies, and compiler-runtime systems
- General-purpose out-of-order CPUs remain superior to specialized TPUs for all deep learning workloads
Answer: The correct answer is C. The core thesis of AI hardware acceleration is that raw compute scaling alone cannot sustain performance gains. Achieving high delivered efficiency (\(\eta_{\text{hw}}\)) requires comprehensive co-design: aligning neural network mathematical primitives (matrix, vector, transcendental) with specialized execution units (Tensor Cores, systolic arrays, SFUs), selecting dataflow strategies that maximize on-chip data reuse, and leveraging compilers and runtimes to optimize memory allocation, kernel fusion, and asynchronous execution. The other choices contradict the chapter’s fundamental principles.
Learning Objective: Evaluate the principles of hardware-software co-design across the AI acceleration stack.
Summarize how the Roofline model serves as a unified diagnostic bridge connecting high-level neural network operations to low-level hardware architecture choices.
Answer: The Roofline model characterizes any neural network operation by its arithmetic intensity (\(I = \text{FLOP}/\text{byte}\)) and maps it against the hardware’s peak compute capacity (\(R_{\text{peak}}\)) and memory bandwidth (\(\text{BW}\)). By comparing \(I\) to the hardware ridge point (\(I_{\text{ridge}} = R_{\text{peak}}/\text{BW}\)), the model immediately diagnoses whether an operator is memory-bound or compute-bound, directing engineers to the appropriate optimization: kernel fusion and tiling to increase data reuse for memory-bound operators, or Tensor Core utilization and parallelism tuning for compute-bound operators.
Learning Objective: Explain how the Roofline model bridges neural network algorithms and accelerator hardware design.
Which combination correctly summarizes the primary function of each layer in the modern AI hardware acceleration software stack?
- Framework: Silicon manufacturing; Compiler: Host PCIe routing; Runtime: Floating-point unit logic
- Framework: Direct transistor clocking; Compiler: Operating system page fault handling; Runtime: Mathematical differentiation
- Compiler: Real-time sensor power regulation; Runtime: Neural network gradient backpropagation; Hardware: Python interpreter dispatch
- Framework: Graph definition and automatic differentiation; Compiler: Graph optimization, operator fusion, and tiling; Runtime: Memory pooling, stream scheduling, and kernel dispatch; Hardware: Parallel matrix/vector execution
Answer: The correct answer is D. In the acceleration stack, the framework (e.g., PyTorch) defines the neural network computation graph and manages autograd; the compiler (e.g., TVM, XLA, TensorRT) performs graph-level optimizations, operator fusion, memory planning, and loop tiling; the runtime (e.g., CUDA runtime) manages device memory pools, asynchronous stream queues, and kernel launches; and the hardware executes instructions across specialized matrix units, vector lanes, and memory hierarchies. The other options misattribute responsibilities across the stack.
Learning Objective: Classify the functional responsibilities of frameworks, compilers, runtimes, and hardware accelerators.