System Assumptions

A napkin calculation is only as trustworthy as the inputs behind it, and those inputs are easy to lose track of: an accelerator’s peak rate, a model’s parameter count, an electricity price, an energy cost per operation. This appendix gathers those canonical values and unit conventions into a single reference sheet, allowing any quantitative estimate in the book to be audited, updated with newer figures, or checked across chapters. It also fixes the unit conventions those estimates depend on. The chapters can then argue from numbers without restating where each one came from.

How to Use This Appendix

When a chapter says an H100 delivers a certain ridge point, or that training a 7B model needs a certain amount of memory, the underlying bandwidths, capacities, and model sizes are listed here. A local estimate can use the relevant Value and Unit entries. The tables are grouped by topic: accelerators, reference models, energy per access, interconnect bandwidth, economics, production-scale anchors, and unit conventions. Assumptions come from vendor datasheet peaks, published studies or industry reports, illustrative market or grid statistics, and book conventions; section 1.1 lists the provenance in one place.

The calculations below expose which assumptions dominate each estimate, so a changed hardware specification or price can propagate without changing the method.

The constants in this appendix support quick, reproducible calculations. The three examples below apply them to performance, capacity, and cost.

Napkin Math 1.1: Classify a roofline regime

Problem: For the stated H100 and workload intensities, which operations are compute bound and which are memory bound?

Variables: For an H100 at FP16/BF16 peak, use 989 TFLOP/s peak compute and 3.35 TB/s memory bandwidth.

Math: Divide peak FLOP/s by memory bandwidth to get the roofline ridge point: 989 TFLOP/s/3.35 TB/s \(\approx\) 295.2 FLOP/byte. A large general matrix multiply (GEMM) with \(n=\) 4,096 has intensity \(n/3 \approx\) 1,365.3 FLOP/byte.

Result: Operations above 295.2 FLOP/byte are compute bound; operations below it are memory bound. The GEMM example is compute bound, while a single-token autoregressive decode with intensity \(\approx 1\) FLOP/byte is deeply memory bound.

Systems insight: The same accelerator can be compute rich and memory constrained. Arithmetic intensity determines which resource the workload actually consumes.

Throughput is only one constraint. Training also requires enough capacity for weights, gradients, and optimizer state.

Napkin Math 1.2: Estimate training-state memory

Problem: How much model-state memory is required to train a 7B model with mixed-precision Adaptive Moment Estimation (Adam)?

Variables: Mixed-precision Adam stores BF16 weights, gradients, FP32 master weights, momentum, and variance. The model-state budget is 16 bytes per parameter.

Math: For 7B parameters, model state is 7 \(\times 10^9 \times\) 16 bytes = 112 GB.

Result: An H100 has 80 GB of high-bandwidth memory (HBM), so the model state alone exceeds a single accelerator before accounting for activations.

Systems insight: Optimizer state, not weights alone, sets the floor for training memory. Capacity planning that starts from parameter bytes underestimates the real requirement.

Fitting the model in memory establishes feasibility, but not operating cost. Power and elapsed time provide the remaining quantities for a first electricity estimate.

Napkin Math 1.3: Estimate accelerator electricity cost

Problem: What is the accelerator-only electricity cost of the editorial GPT-3-scale scenario?

Variables: Use 1,024 A100s, 400 W per accelerator, ~25 days wall-clock, and $0.12/kWh electricity.

Math: The run requires roughly \(3.14 \times 10^{23}\) FLOPs. The A100-equivalent electricity-cost estimate is ~25 days wall-clock \(\times\) 1,024 A100s \(\times\) 24 h/day \(\times\) 0.4 kW \(\times\) $0.12/kWh = ~$29,491.2.

Result: The accelerator-only electricity cost is approximately ~$29,491.2. This accelerator thermal design power (TDP)-only estimate excludes host CPUs, networking, storage, cooling, and facility overhead. The original GPT-3 run used V100-era infrastructure; this is an editorial A100-equivalent scenario, not a reported configuration or a duration derived from peak FLOP/s alone.

Systems insight: Energy is measurable from power and time, but full cost accounting must also include capital utilization, networking, storage, staffing, and failed or repeated runs.

Accelerator Specifications

These are the accelerator assumptions used in the book’s roofline, training-memory, energy, and cost examples: peak throughput, memory bandwidth, memory capacity, and TDP for each generation the chapters cite. Values are vendor datasheet peaks—ceilings for napkin math, not sustained utilization—drawn from NVIDIA product documentation and IEEE Micro architecture articles (NVIDIA Corporation 2017, 2020b, 2020a, 2021, 2018, 2024; Choquette et al. 2021; Choquette 2023), AMD MI300X documentation (AMD 2023), Google Tensor Processing Unit (TPU) publications (Jouppi et al. 2023), and Google Cloud’s current TPU v6e specification page (Google Cloud 2026). Capacity rows preserve official decimal-GB nameplates. Tables progress from table 2 (Volta) and table 3 (Turing) through current and forward-looking generations.

Master accelerator spec matrix

The master accelerator matrix in table 1 synthesizes specifications across key GPU, TPU, Wafer-Scale, and domain-specific AI accelerators.

Table 1: Master Accelerator Specification Matrix: Comprehensive reference matrix comparing production AI hardware across architectures, process nodes, TDP, multi-precision compute throughputs (FP64 to INT8), memory capacity and bandwidth, on-chip SRAM capacity, and FP16 Roofline Ridge Point (\(I_{\text{ridge}} = \text{Peak FP16 TFLOP/s}/\text{Memory Bandwidth (TB/s)}\)).

Architecture/Chip	Node (nm)	TDP (W)	Compute Peak TFLOPs/TOPS	Memory Capacity & Bandwidth	On-Chip SRAM	\(I_{\text{ridge}}\) (FLOP/B)
NVIDIA V100 (SXM2 32 GB)	12nm (TSMC)	300	FP64: 7.8, FP32: 15.7, FP16: 125 (TC), INT8: 125 (TC)	32 GB HBM2 @ 0.90 TB/s	16 MB	138.9
NVIDIA A100 (80 GB SXM4)	7nm N7 (TSMC)	400	FP64: 9.7/19.5, FP32: 19.5, TF32: 156, FP16/BF16: 312, INT8: 624	80 GB HBM2e @ 2.04 TB/s	40 MB	153.0
NVIDIA H100 (80 GB SXM5)	4nm 4N (TSMC)	700	FP64: 34/67, FP32: 67, TF32: 494, FP16/BF16: 989, FP8: 1,978, INT8: 1,978	80 GB HBM3 @ 3.35 TB/s	50 MB	295.2
NVIDIA H200 (141 GB SXM5)	4nm 4N (TSMC)	700	FP64: 34/67, FP32: 67, TF32: 494, FP16/BF16: 989, FP8: 1,978, INT8: 1,978	141 GB HBM3e @ 4.80 TB/s	50 MB	206.0
NVIDIA B200 (192 GB SXM)	4nm 4NP (TSMC)	1000	FP64: 45/90, FP32: 90, TF32: 1,250, FP16/BF16: 2,250, FP8: 4,500, INT8: 4,500	192 GB HBM3e @ 8.00 TB/s	128 MB	281.3
NVIDIA L40S (48 GB PCIe)	4nm 4N (TSMC)	350	FP64: 1.4, FP32: 91.6, TF32: 183, FP16/BF16: 366, FP8: 733, INT8: 733	48 GB GDDR6 @ 0.86 TB/s	96 MB	423.6
Google TPU v1	28nm (TSMC)	75	INT8: 92 TOPS (FP formats N/A)	8 GB DDR3 @ 0.034 TB/s	28 MB	N/A (2,705.9 INT8)
Google TPU v2	16nm (TSMC)	280 (board)	BF16/FP16: 45 TFLOPs	16 GB HBM @ 0.60 TB/s	32 MB	75.0
Google TPU v3	16nm (TSMC)	450 (board)	BF16/FP16: 123 TFLOPs	32 GB HBM2 @ 0.90 TB/s	32 MB	136.7
Google TPU v4	7nm N7 (TSMC)	170	BF16/FP16: 275, FP32: 275, INT8: 275	32 GB HBM2 @ 1.20 TB/s	48 MB	229.2
Google TPU v5e	7nm N7 (TSMC)	140	BF16/FP16: 197, FP8: 394, INT8: 394	16 GB HBM2e @ 0.82 TB/s	32 MB	240.5
Google TPU v5p	4nm 4N (TSMC)	450	BF16/FP16: 459, FP8: 918, INT8: 918	95 GB HBM3 @ 2.76 TB/s	96 MB	166.3
Google TPU v6e (Trillium)	4nm 4N (TSMC)	400	BF16/FP16: 926, FP8: 1,852, INT8: 1,852	32 GB HBM3 @ 1.64 TB/s	64 MB	564.6
Cerebras WSE-3	5nm N5 (TSMC)	23,000	FP16/BF16: 125,000, FP8: 250,000, INT8: 250,000	44,000 GB SRAM @ 21,000 TB/s	44,000 MB	5.95
AMD Instinct MI300X	5nm+6nm 3D	750	FP64: 163.4, FP32: 163.4, TF32: 653.7, FP16/BF16: 1,307.4, FP8: 2,614.9	192 GB HBM3 @ 5.30 TB/s	256 MB	246.7
Intel Habana Gaudi 2	7nm N7 (TSMC)	600	FP32: 38.5, TF32: 154, FP16/BF16: 307, FP8: 614, INT8: 614	96 GB HBM2e @ 2.45 TB/s	48 MB	125.3
Intel Habana Gaudi 3	5nm N5 (TSMC)	900	FP32: 115, TF32: 460, FP16/BF16: 1,835, FP8: 3,670, INT8: 3,670	128 GB HBM3e @ 3.70 TB/s	96 MB	495.9

NVIDIA V100

Table 2: NVIDIA V100 (Volta): Peak specs from NVIDIA’s architecture whitepaper and product datasheet (NVIDIA Corporation 2017, 2020b). The V100 introduced Tensor Cores for mixed-precision training and anchors baseline-generation comparisons.

NVIDIA Corporation. 2020b. NVIDIA V100 Tensor Core GPU Datasheet. NVIDIA product documentation.

Assumption	Value	Unit
Peak FP16 tensor throughput (V100)	125	TFLOP/s
Peak FP32 throughput (V100)	15.7	TFLOP/s
HBM bandwidth (V100)	900	GB/s
HBM capacity (V100)	32	GB
TDP (V100)	300	W

NVIDIA T4

Table 3: NVIDIA T4 (Turing): Peak specs from NVIDIA Tesla T4 product documentation (NVIDIA Corporation 2018). The 70 W TDP makes this low-power inference accelerator a useful cost-per-inference anchor.

Assumption	Value	Unit
Peak FP16 tensor throughput (T4)	65	TFLOP/s
Peak INT8 throughput (T4)	130	TOPS
Memory bandwidth (T4)	320	GB/s
TDP (T4)	70	W

NVIDIA A100

The Ampere-generation specs (table 4) anchor most training examples in the book.

Table 4: NVIDIA A100 (Ampere): Peak specs from NVIDIA’s architecture whitepaper, 80 GB datasheet, and architecture article (NVIDIA Corporation 2020a, 2021; Choquette et al. 2021). The A100 is the most commonly cited accelerator in training examples; 80 GB HBM2e and TF32 Tensor Cores set memory-capacity and compute-intensity anchors.

NVIDIA Corporation. 2021. NVIDIA A100 Tensor Core GPU Datasheet. NVIDIA product documentation.

Assumption	Value	Unit
Peak FP16 tensor throughput (A100)	312	TFLOP/s
Peak FP32 throughput (A100)	19.5	TFLOP/s
Peak INT8 throughput (A100)	624	TOPS
Peak TF32 throughput (A100)	156	TFLOP/s
HBM bandwidth (A100)	2039	GB/s
HBM capacity (A100)	80	GB
TDP (A100)	400	W

NVIDIA H100

For Hopper-era estimates, the reference values include FP8 Tensor Cores and the Transformer Engine (table 5).

Table 5: NVIDIA H100 (Hopper): Peak specs from the Hopper architecture overview (Choquette 2023). FP8 Tensor Cores and the Transformer Engine drive Hopper-generation training and serving estimates.

Assumption	Value	Unit
Peak FP16 tensor throughput (H100)	989	TFLOP/s
Peak FP32 CUDA throughput (H100)	67	TFLOP/s
Peak FP8 tensor throughput (H100)	1979	TFLOP/s
Peak INT8 throughput (H100)	1979	TOPS
Peak TF32 throughput (H100)	494	TFLOP/s
HBM bandwidth (H100)	3.35	TB/s
HBM capacity (H100)	80	GB
TDP (H100)	700	W

NVIDIA B200

Forward-looking capacity-planning examples rely on Blackwell-generation specs (table 6).

Table 6: NVIDIA B200 (Blackwell): Peak specs from NVIDIA Blackwell product documentation (NVIDIA Corporation 2024). Forward-looking capacity-planning examples use its HBM3e bandwidth and FP8 throughput.

Assumption	Value	Unit
Peak FP16 tensor throughput (B200)	2250	TFLOP/s
Peak FP8 tensor throughput (B200)	4500	TFLOP/s
Peak FP4 throughput (B200)	9000	TFLOP/s
HBM bandwidth (B200)	8	TB/s
HBM capacity (B200)	180	GB
TDP (B200)	1000	W

AMD Instinct MI300X

To compare with H100-class hardware used elsewhere, MI300X capacity, bandwidth, throughput, and power specifications provide a cross-vendor baseline (table 7).

Table 7: AMD Instinct MI300X: Peak specs from AMD Instinct MI300X product documentation (AMD 2023). High HBM capacity (192 GB) and bandwidth make it a common cross-vendor baseline for memory-bound workloads.

Assumption	Value	Unit
Peak FP16 tensor throughput (MI300X)	1307	TFLOP/s
HBM bandwidth (MI300X)	5.3	TB/s
HBM capacity (MI300X)	192	GB
TDP (MI300X)	750	W

Google TPU v4 and v6e

When comparing training economics across accelerator families, TPU figures offer an application-specific integrated circuit (ASIC)-based alternative (table 8).

Table 8: Google TPU v4 and v6e (Trillium): TPU v4 values come from Google TPU publications (Jouppi et al. 2023), and TPU v6e values come from Google Cloud’s current Trillium specification page (Google Cloud 2026).

Assumption	Value	Unit
Peak BF16 throughput (TPU v4)	275	TFLOP/s
Memory bandwidth (TPU v4)	1200	GB/s
Peak BF16 throughput (TPU v6e)	918	TFLOP/s
Memory bandwidth (TPU v6e)	1638	GB/s

CPU and mobile/edge processors

Edge and mobile ML examples rely on a different performance baseline (table 9), contrasting starkly with data center throughput to illustrate why deployment target shapes every design decision.

Table 9: CPU, Mobile NPU, and Edge Device Specs: Illustrative reference points for edge and mobile examples (not vendor peaks for a single SKU). The contrast with data-center accelerators shows why deployment target shapes every design decision.

Assumption	Value	Unit
Peak FP32 throughput (reference CPU)	1	TFLOP/s
DRAM bandwidth (reference server)	50	GB/s
Reference NPU throughput (iPhone 15 Pro)	35	TOPS
Memory bandwidth (iPhone 15 Pro)	51.2	GB/s
TDP (mobile device, reference)	5	W
Power (edge object detector, reference)	2	W
Battery capacity (phone, reference)	15	Wh

Model Specifications

These are the reference model assumptions behind training-cost, memory-footprint, and inference-workload examples: parameter counts, per-inference FLOP budgets, and published training-scale anchors. Published model sizes follow primary papers, model reports, and official model documentation (Devlin et al. 2019; Radford et al. 2019; Brown et al. 2020; Dubey et al. 2024; He et al. 2016; Sandler et al. 2018; Ultralytics 2023). The inference FLOP budgets are book-derived values: they use sequence length 128 for BERT-Base, 224 \(\times\) 224 images for ResNet-50 and MobileNetV2, and 640 \(\times\) 640 images for YOLOv8-Nano; the vision models count two FLOPs per multiply-add. GPT-3 training FLOPs follow (Brown et al. 2020); the 25-day, 1,024-A100 pairing is editorial. The GPT-4 parameter count and training GPU-days are public third-party mixture of experts (MoE) estimates (Patel and Wong 2023) because the GPT-4 technical report does not disclose architecture size. When a chapter estimates GPT-3-scale training time, it uses table 10.

Table 10: Reference Model Specifications: Parameter counts and FLOP budgets for worked examples. BERT, GPT-2, GPT-3, Llama, ResNet-50, MobileNetV2, and YOLOv8 rows use primary papers, model reports, or official model documentation (Devlin et al. 2019; Radford et al. 2019; Brown et al. 2020; Dubey et al. 2024; He et al. 2016; Sandler et al. 2018; Ultralytics 2023). GPT-4 rows cite (OpenAI et al. 2023) for the official model family and (Patel and Wong 2023) for the public MoE parameter and GPU-day estimates used in this edition.

Assumption	Value	Unit
Inference FLOPs (BERT-Base)	2.2e+10	flop
Parameters (BERT-Base)	1.1e+08	param
Parameters (Llama 3 8B)	8.03e+09	param
Hidden dimension (GPT-2)	1600	-
Layers (GPT-2)	48	-
Parameters (GPT-2)	1.5e+09	param
Parameters (GPT-3)	1.75e+11	param
Scenario duration (GPT-3 scale)	25	d
Reference training FLOPs (GPT-3)	3.14e+23	flop
Parameters (GPT-4, public MoE estimate)	1.76e+12	param
Reference training GPU-days (GPT-4)	2.5e+06	GPU-days
Inference FLOPs (ResNet-50)	8.2e+09	flop
Parameters (ResNet-50)	2.56e+07	param
Inference FLOPs (MobileNetV2)	6e+08	flop
Parameters (MobileNetV2)	3.50487e+06	param
Inference FLOPs (YOLOv8-Nano)	8.7e+09	flop

Training Memory Conventions

Training-memory napkin math assumes the mixed-precision Adam storage model used in the napkin-math callout and several training chapters: BF16 weights, BF16 gradients, and FP32 master weights plus Adam first- and second-moment buffers (16 bytes per parameter in total). This is a book convention, not a measured hardware constant. The BF16 data path follows established BFLOAT16 mixed-precision practice (Kalamkar et al. 2019); the FP32-master pattern follows earlier FP16 work (Micikevicius et al. 2017; NVIDIA 2017), and Adam’s first- and second-moment buffers follow (Kingma and Ba 2015). Table 11 lists per-component byte widths; multiplying bytes per parameter (mixed-precision Adam) by the parameter count gives the nonactivation training-state footprint.

Table 11: Training Memory Conventions: Per-parameter storage for mixed-precision Adam (2 + 2 + 12 = 16 bytes before activations). Book convention for napkin math; see (NVIDIA 2017) for mixed-precision training context.

Assumption	Value	Unit
Weight/gradient width (BF16)	2	bytes
Master weight width (FP32)	4	bytes
Master + Adam states per parameter (FP32)	12	bytes
Bytes per parameter (mixed-precision Adam)	16	bytes

Hardware and model assumptions fix what runs where; energy assumptions fix whether the design is thermally and economically viable at the operation and memory-access level.

Energy Constants

Horowitz’s 45 nm arithmetic and 32-bit DRAM estimates (Horowitz 2014), together with illustrative book anchors for register and SRAM access, MobileNetV2 inference, and 5G transfer, underpin the book’s efficiency comparisons but are not process-independent. Table 12 lists the access-energy hierarchy from registers through DRAM. That hierarchy quantifies why data reuse dominates kernel design.

Table 12: Energy per Operation and Access: Arithmetic and 32-bit DRAM estimates use Horowitz’s 45 nm model (Horowitz 2014), and the per-byte DRAM row is derived from that estimate. Register, SRAM, MobileNetV2, and 5G rows are illustrative book anchors. The roughly 6,400× gap between a 0.1 pJ register access and a 640 pJ DRAM access explains why data reuse dominates ML kernel optimization.

Assumption	Value	Unit
Register access energy	0.1	pJ
L1 SRAM access energy	0.5	pJ
L2 SRAM access energy	2	pJ
DRAM access energy (32-bit)	640	pJ
DRAM access energy (per byte)	160	pJ/byte
FP16 multiply energy (45 nm)	1.1	pJ/multiply
FP32 multiply energy (45 nm)	3.7	pJ/multiply
INT8 multiply energy (45 nm)	0.2	pJ/multiply
MobileNetV2 inference energy (reference)	0.1	mJ
5G transfer energy per MB	100	mJ/MB

Precision & arithmetic energy hierarchy

The arithmetic energy hierarchy in table 13 compares the energy required per operation (\(E_{\text{op}}\) in pJ/op) for additions and multiplications across floating-point, tensor, and integer precisions on modern 7nm, 5nm, and 3nm silicon process nodes.

Table 13: Precision & Arithmetic Energy Hierarchy: Energy per operation (\(E_{\text{op}}\) in pJ/op) for Addition (Add) and Multiplication (Mul) across data formats (FP64, FP32, TF32, FP16, BF16, FP8 E4M3/E5M2, INT32, INT8, INT4) and semiconductor process nodes (7nm, 5nm, 3nm). Energy scaling illustrates the quadratic \(O(b^2)\) multiplier vs linear \(O(b)\) adder bitwidth dependence (Horowitz 2014; Dally et al. 2021).

Precision/Data Format	7nm Node (\(E_{\text{op}}\) pJ)	5nm Node (\(E_{\text{op}}\) pJ)	3nm Node (\(E_{\text{op}}\) pJ)	Energy Ratio vs FP32 (3nm)
FP64 (64-bit Float)	Add: 0.90/Mul: 2.50	Add: 0.58/Mul: 1.62	Add: 0.38/Mul: 1.05	2.23\(\times\)
FP32 (32-bit Float)	Add: 0.40/Mul: 1.10	Add: 0.26/Mul: 0.72	Add: 0.17/Mul: 0.47	1.00\(\times\) (Ref)
TF32 (19-bit TensorFloat)	Add: 0.28/Mul: 0.65	Add: 0.18/Mul: 0.42	Add: 0.12/Mul: 0.27	0.57\(\times\)
FP16 (16-bit IEEE Float)	Add: 0.20/Mul: 0.45	Add: 0.13/Mul: 0.29	Add: 0.08/Mul: 0.19	0.40\(\times\)
BF16 (16-bit Bfloat16)	Add: 0.18/Mul: 0.40	Add: 0.12/Mul: 0.26	Add: 0.08/Mul: 0.17	0.36\(\times\)
FP8 (E4M3/E5M2)	Add: 0.09/Mul: 0.20	Add: 0.06/Mul: 0.13	Add: 0.04/Mul: 0.08	0.17\(\times\)
INT32 (32-bit Integer)	Add: 0.10/Mul: 0.80	Add: 0.06/Mul: 0.52	Add: 0.04/Mul: 0.34	0.72\(\times\)
INT8 (8-bit Integer)	Add: 0.03/Mul: 0.20	Add: 0.02/Mul: 0.13	Add: 0.013/Mul: 0.085	0.18\(\times\)
INT4 (4-bit Integer)	Add: 0.010/Mul: 0.060	Add: 0.006/Mul: 0.039	Add: 0.004/Mul: 0.025	0.053\(\times\)

Energy costs operate at the chip level, but real ML systems also move data across interconnects—between accelerators, across racks, and over wide-area networks. The next section lists the bandwidth assumptions used when chapters estimate communication overhead.

Interconnect and Network Bandwidth

These bandwidth assumptions apply when chapters reason about gradient synchronization, pipeline bubbles, checkpoint I/O, or cross–data center latency. NVLink and PCIe rates follow accelerator product documentation (NVIDIA Corporation 2017, 2020a; Choquette 2023); the InfiniBand architecture specification anchors the protocol family (InfiniBand Trade Association 2000), while current high-speed product families and Ethernet roadmaps anchor modern link-rate examples (NVIDIA 2026; Ethernet Alliance 2025); the speed-of-light-in-fiber floor is a physics identity. Table 14 lists NVLink, InfiniBand, PCIe, Non-Volatile Memory Express (NVMe), and Ethernet rates together with the propagation speed of light in fiber.

Table 14: Interconnect and Network Bandwidth: Link-family bandwidths use vendor specs, the InfiniBand architecture specification, and current product/roadmap documentation (InfiniBand Trade Association 2000; NVIDIA 2026; Ethernet Alliance 2025). NVLink rates are aggregate bidirectional; other link rates are nominal per direction. Speed of light in fiber sets the cross–data-center latency floor.

Assumption	Value	Unit
NVLink bandwidth (V100)	300	GB/s
NVLink bandwidth (A100)	600	GB/s
NVLink bandwidth (H100)	900	GB/s
InfiniBand HDR link rate	200	Gb/s
InfiniBand NDR link rate	400	Gb/s
InfiniBand XDR link rate	800	Gb/s
PCIe Gen4 x16 rate	32	GB/s
PCIe Gen5 x16 rate	64	GB/s
NVMe sequential read bandwidth	7	GB/s
10 GbE link rate	10	Gb/s
100 GbE link rate	100	Gb/s
Speed of light in fiber	200000	km/s

Interconnect & network fabric physical spec sheet

The physical specification sheet in table 15 details bandwidth, latency, framing, direct memory access (DMA) support, and energy per bit across host buses, accelerator interconnects, network fabrics, and cache-coherent interfaces.

Table 15: Interconnect & Network Fabric Physical Spec Sheet: Comprehensive physical spec sheet for host buses, GPU interconnects, switches, network fabrics, and coherent interfaces (PCIe Gen3–Gen7, NVLink 1–5, NVSwitch 1–4, InfiniBand EDR–XDR, RoCE v2, CXL 1.1–3.1). Columns enumerate directional and aggregate bandwidth, PHY/link latency, line encoding/framing, DMA and GPUDirect capability, and physical layer energy cost (\(E_{\text{link}}\) in pJ/bit).

Bus/Interconnect Standard	Directional & Aggregate Bandwidth	Latency	Encoding/Framing	DMA/GPUDirect Support	Energy Cost (\(E_{\text{link}}\))
PCIe Gen3 (x16)	15.75 GB/s dir/31.5 GB/s agg	400–500 ns	128b/130b NRZ	PCIe P2P, GPUDirect RDMA	14.0 pJ/bit
PCIe Gen4 (x16)	31.50 GB/s dir/63.0 GB/s agg	250–400 ns	128b/130b NRZ	PCIe P2P, GPUDirect RDMA	10.0 pJ/bit
PCIe Gen5 (x16)	63.00 GB/s dir/126.0 GB/s agg	150–250 ns	128b/130b NRZ	PCIe P2P, GPUDirect RDMA, CXL 1.1/2.0	8.0 pJ/bit
PCIe Gen6 (x16)	126.00 GB/s dir/252.0 GB/s agg	100–150 ns	PAM4 (256B Flit)	PCIe P2P, GPUDirect RDMA, CXL 3.0/3.1	6.0 pJ/bit
PCIe Gen7 (x16)	252.00 GB/s dir/504.0 GB/s agg	<100 ns	PAM4 (512B Flit)	PCIe P2P, GPUDirect RDMA, CXL 3.1+	4.5 pJ/bit
NVLink 1 (P100)	20 GB/s dir/40 GB/s agg (link)	200–300 ns	NRZ (20 Gb/s)	GPUDirect P2P/RDMA	10.0 pJ/bit
NVLink 2 (V100)	25 GB/s dir/50 GB/s agg (link)	150–200 ns	NRZ (25.78 Gb/s)	GPUDirect P2P/RDMA	8.0 pJ/bit
NVLink 3 (A100)	25 GB/s dir/50 GB/s agg (link)	100–150 ns	NRZ (50 Gb/s)	GPUDirect P2P/RDMA	6.5 pJ/bit
NVLink 4 (H100)	25 GB/s dir/50 GB/s agg (link)	80–100 ns	PAM4 (100 Gb/s)	GPUDirect P2P, SHARP Aggregation	4.5 pJ/bit
NVLink 5 (B200)	50 GB/s dir/100 GB/s agg (link)	60–80 ns	PAM4 (200 Gb/s)	GPUDirect P2P, NVLink Network Offload	3.5 pJ/bit
NVSwitch 1 (Volta)	900 GB/s aggregate per chip	~100 ns	NRZ	Hardware P2P Crossbar Routing	8.5 pJ/bit
NVSwitch 2 (Ampere)	2.4 TB/s aggregate per chip	~90 ns	NRZ	Hardware P2P Crossbar, SHARP v2	6.5 pJ/bit
NVSwitch 3 (Hopper)	3.2 TB/s aggregate per chip	~70 ns	PAM4	SHARP v3 In-Network Compute	4.5 pJ/bit
NVSwitch 4 (Blackwell)	14.4 TB/s aggregate per chip	~50 ns	PAM4	SHARP v4, FP8 Reduction Engine	3.2 pJ/bit
InfiniBand EDR	100 Gbps (12.5 GB/s dir)	~0.50 \(\mu\text{s}\) (500 ns)	64b/66b NRZ	GPUDirect RDMA (Verbs)	15.0 pJ/bit
InfiniBand HDR	200 Gbps (25.0 GB/s dir)	~0.60 \(\mu\text{s}\) (600 ns)	64b/66b PAM4/NRZ	GPUDirect RDMA, SHARP v2	10.0 pJ/bit
InfiniBand NDR	400 Gbps (50.0 GB/s dir)	~0.50 \(\mu\text{s}\) (500 ns)	256b/257b PAM4	GPUDirect RDMA, SHARP v3	7.0 pJ/bit
InfiniBand XDR	800 Gbps (100.0 GB/s dir)	~0.40 \(\mu\text{s}\) (400 ns)	PAM4 (200G/lane Flit)	GPUDirect RDMA, SHARP v4	5.0 pJ/bit
RoCE v2 Ethernet	100–800 Gbps (12.5–100 GB/s dir)	1.0–2.5 \(\mu\text{s}\)	NRZ/PAM4 (802.3)	GPUDirect RDMA (RoCEv2 UDP/IP)	8.0–12.0 pJ/bit
CXL 1.1	63 GB/s dir/126 GB/s agg (x16)	~200 ns	128b/130b NRZ	cxl.io, cxl.cache, cxl.mem	8.0 pJ/bit
CXL 2.0	63 GB/s dir/126 GB/s agg (x16)	~180 ns	128b/130b NRZ	SLD/MLD Memory Pooling	7.5 pJ/bit
CXL 3.0	126 GB/s dir/252 GB/s agg (x16)	~120 ns	256B Flit PAM4	P2P Memory Pooling & Fabric Switch	5.5 pJ/bit
CXL 3.1	126 GB/s dir/252 GB/s agg (x16)	~110 ns	256B Flit PAM4	Global Integrated Memory, TEE	5.0 pJ/bit

Economic Constants

These pricing assumptions underpin total cost of ownership (TCO) and energy-cost napkin math in table 16. They are illustrative hyperscaler-order rates for ratio analysis (similar in spirit to carbon accounting examples in (Patterson et al. 2021)), not quotes for a specific region or contract. Local values should replace these anchors when absolute price dominates.

Table 16: Economic Assumptions: Illustrative cloud electricity and egress rates for TCO napkin math (2024–2025 order of magnitude). On-premise cost structures differ; relative magnitudes guide design trade-offs.

Assumption	Value	Unit
Cloud electricity price	0.12	dollar/kWh
Cloud egress price per GB	0.09	dollar/GB

Economic constants set the price per unit of compute and data transfer, but they mean little without a sense of the volumes involved. Production ML systems handle millions to billions of requests per day—numbers large enough to be difficult to internalize without concrete reference points.

Scale References

These scale assumptions in table 17 anchor “how big is big?” for capacity-planning examples, providing illustrative order-of-magnitude baselines for email and search volume, autonomous-vehicle sensor streams, and standard 1080p and 4K video formats. They are magnitude anchors, not audited statistics for a specific year.

Table 17: Production Scale and Data Rate References: Illustrative workload and sensor-rate anchors, plus standard 1080p and UHD 4K video parameters.

Assumption	Value	Unit
Gmail emails per day	1.21e+11	-
Google searches per day	8.5e+09	-
Waymo sensor data rate (low)	1	TB/h
Waymo sensor data rate (high)	19	TB/h
1080p frame width	1920	-
1080p frame height	1080	-
4K frame width	3840	-
4K frame height	2160	-
Bytes per RGB pixel	3	bytes
Video frame rate (standard)	30	Hz

The assumptions above use shared unit conventions: decimal data prefixes, distinct FLOPs and FLOP/s quantities, and the aliases defined in the next section.

Unit Conventions

Table 18 fixes the unit conventions used in every quantitative example in this book. Each row gives the multiplier \(k\) in 1 alias \(=\) \(k\) base. Data prefixes use decimal SI (\(\mathrm{KB} = 10^3\) bytes, not 1024). Binary IEC storage prefixes appear in a few storage-specific discussions but are omitted here because most fleet-scale estimates in the book use decimal KB/GB/TB. Throughput quantities (FLOP/s, GB/s) combine these aliases with time; adding incompatible dimensions (bytes to FLOP/s) is a category error in napkin math, not a unit conversion.

Hardware capacity is the one deliberate exception. Vendors label memory with decimal symbols but ship binary quantities. An accelerator sold as an 80 GB device provides \(80 \times 2^{30}\) bytes, and a microcontroller sold with 512 KB of SRAM provides \(512 \times 2^{10}\) bytes. This book keeps both the vendor’s number and the vendor’s label, so a capacity printed as 80 GB is the nameplate figure rather than a decimal conversion of it. Every other data quantity, including model footprints, activation sizes, and transfer volumes, uses the decimal prefixes above. Where a worked example subtracts a computed footprint from a device capacity, both terms are carried in decimal so that the arithmetic printed on the page is the arithmetic a reader can reproduce.

Table 18: Unit Conventions: Decimal SI aliases and scale factors (book convention for napkin math). Throughput forms such as FLOP/s and GB/s divide work or data by time using the same conventions.

Alias	Multiplier	Base unit
byte	1	byte
KB	1000	byte
MB	1e+06	byte
GB	1e+09	byte
TB	1e+12	byte
PB	1e+15	byte
flop	1	flop
GFLOPs	1e+09	flop
TFLOPs	1e+12	flop
ZFLOPs	1e+21	flop
param	1	param
Mparam	1e+06	param
Gbps	1e+09	bit/s
NS	\(10^{-9}\)	second
US	\(10^{-6}\)	second
MS	\(10^{-3}\)	second
second	1	second
hour	3600	second
day	86400	second
joule	1	joule
watt	1	watt
meter	1	meter

Each assumption also has a provenance record that lets readers trace the number back to its primary source.

Assumption Provenance

Table 19 groups the appendix’s quantitative inputs by evidence class and identifies the primary references behind each group. Vendor specifications and standards define published limits, peer-reviewed measurements provide empirical anchors, and explicitly marked editorial scenarios supply illustrative values where no durable universal constant exists. This distinction lets readers judge the evidentiary weight of an input, update time-sensitive values, and reproduce an estimate with an appropriate local substitute.

Table 19: Assumption Provenance Catalog: Quick map from appendix section to source class and bibliography.

Appendix section	Source type	Primary references
Accelerator specifications	Vendor datasheet peaks; editorial CPU/mobile anchors; Master Spec Matrix (table 1)	(NVIDIA Corporation 2017, 2018, 2020a, 2024; Choquette et al. 2021; Choquette 2023; AMD 2023; Jouppi et al. 2023; Google Cloud 2026); editorial for CPU/mobile rows
Model specifications	Published papers, model reports, and official docs; editorial GPT-3 scenario; GPT-4 size from public analysis	(Devlin et al. 2019; Radford et al. 2019; Brown et al. 2020; Dubey et al. 2024; He et al. 2016; Sandler et al. 2018; Ultralytics 2023; OpenAI et al. 2023; Patel and Wong 2023)
Training memory conventions	Book convention (mixed-precision Adam layout)	(Kingma and Ba 2015; Kalamkar et al. 2019; Micikevicius et al. 2017; NVIDIA 2017)
Energy constants	Published 45 nm arithmetic/DRAM estimates; VLSI synthesis node scaling; Precision Energy Hierarchy (table 13)	(Horowitz 2014; Dally et al. 2021) for arithmetic/DRAM; editorial for register/SRAM, MobileNetV2, and 5G rows
Interconnect bandwidth	Vendor specs; InfiniBand standard; PCIe/NVLink/CXL standards; Physical Spec Sheet (table 15)	(NVIDIA Corporation 2017, 2020a; Choquette 2023; InfiniBand Trade Association 2000; NVIDIA 2026; Ethernet Alliance 2025)
Economic assumptions	Illustrative cloud/utility rates	(Patterson et al. 2021) (methodology context)
Scale references	Illustrative workload rates; standard video conventions	Editorial for workload rates; standard format definitions for video
Unit conventions	Decimal SI; book notation	Editorial

NVIDIA Corporation. 2017. NVIDIA Tesla V100 GPU Architecture. NVIDIA Whitepaper.

NVIDIA Corporation. 2018. NVIDIA Tesla T4 Tensor Core GPU. NVIDIA product documentation.

NVIDIA Corporation. 2020a. NVIDIA A100 Tensor Core GPU Architecture. NVIDIA Whitepaper, V1.0.

NVIDIA Corporation. 2024. NVIDIA Blackwell Architecture. NVIDIA product documentation.

Choquette, Jack, Wishwesh Gandhi, Olivier Giroux, Nick Stam, and Ronny Krashinsky. 2021. “NVIDIA A100 Tensor Core GPU: Performance and Innovation.” IEEE Micro 41 (2): 29–35. https://doi.org/10.1109/mm.2021.3061394.

Choquette, Jack. 2023. “NVIDIA Hopper H100 GPU: Scaling Performance.” IEEE Micro 43 (3): 9–17. https://doi.org/10.1109/mm.2023.3256796.

AMD. 2023. AMD Instinct MI300X Accelerators. AMD product documentation.

Jouppi, Norm, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, et al. 2023. “TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings.” Proceedings of the 50th Annual International Symposium on Computer Architecture, 1–14. https://doi.org/10.1145/3579371.3589350.

Google Cloud. 2026. TPU v6e.

Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 4171–86. https://doi.org/10.18653/v1/n19-1423.

Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models Are Unsupervised Multitask Learners. OpenAI.

Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. 2020. “Language Models Are Few-Shot Learners.” Advances in Neural Information Processing Systems 33: 1877–901. https://doi.org/10.48550/arxiv.2005.14165.

Dubey, Abhimanyu, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783.

He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. “Deep Residual Learning for Image Recognition.” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–78. https://doi.org/10.1109/cvpr.2016.90.

Sandler, Mark, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. “MobileNetV2: Inverted Residuals and Linear Bottlenecks.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4510–20. https://doi.org/10.1109/cvpr.2018.00474.

Ultralytics. 2023. YOLOv8.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, et al. 2023. “GPT-4 Technical Report.” arXiv Preprint arXiv:2303.08774, ahead of print. https://doi.org/10.48550/arXiv.2303.08774.

Patel, Dylan, and Gerald Wong. 2023. GPT-4 Architecture, Infrastructure, Training Dataset, Costs, Vision, MoE. SemiAnalysis Blog.

Kingma, Diederik P., and Jimmy Ba. 2015. “Adam: A Method for Stochastic Optimization.” 3rd International Conference on Learning Representations (ICLR).

Kalamkar, Dhiraj, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, et al. 2019. A Study of BFLOAT16 for Deep Learning Training.

Micikevicius, Paulius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, et al. 2017. “Mixed Precision Training.” arXiv Preprint arXiv:1710.03740.

NVIDIA. 2017. Training with Mixed Precision.

Horowitz, Mark. 2014. “1.1 Computing’s Energy Problem (and What We Can Do about It).” 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), 10–14. https://doi.org/10.1109/isscc.2014.6757323.

Dally, William J., Stephen W. Keckler, and David B. Kirk. 2021. “Evolution of the Graphics Processing Unit (GPU).” IEEE Micro 41 (6): 42–51. https://doi.org/10.1109/mm.2021.3113475.

InfiniBand Trade Association. 2000. InfiniBand Architecture Specification Volume 1. InfiniBand Trade Association.

NVIDIA. 2026. NVIDIA Quantum-X800 InfiniBand Platform. NVIDIA product documentation.

Ethernet Alliance. 2025. 2025 Ethernet Roadmap. Ethernet Alliance roadmap.

Patterson, David, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2021. “Carbon Emissions and Large Neural Network Training.” arXiv Preprint arXiv:2104.10350.

Summary

This appendix is the reference catalog for the quantitative inputs used throughout the book. It gathers accelerator and model specifications, training-memory conventions, energy constants, interconnect bandwidths, economic and scale references, unit conventions, and provenance. The values come from vendor peaks, published measurements, standards, and editorial scenarios, so they carry different evidentiary weight and age at different rates.

Use these tables as starting inputs, not universal constants. For each estimate, record the table row, unit convention, and provenance; replace time-sensitive peaks, prices, and rates with local values; distinguish a physical ceiling from a measured operating point and a scenario assumption; and report which assumptions dominate the result. This keeps napkin math reproducible and makes clear which input controls the conclusion.