Data Foundations
Data problems rarely present as data problems; they surface as a training run that will not speed up, an accelerator sitting idle, or accuracy eroding while dashboards stay green. To keep chapter arguments focused on system design, this appendix gathers the back-of-the-envelope calculations and statistical tools that trace such symptoms back to measurable data bottlenecks: transfer times, storage-layout costs, drift diagnostics, and numerical-stability tools. The tools here lean on basic probability and on the data term of the iron law.
How to Use This Appendix
This appendix is designed as a reference. Reach for it when debugging “slow training,” “low accelerator utilization,” or “production accuracy drift” calls for the quickest path from symptom to measurement.
Conventions used here follow the book-wide notation (for example, \(B\) is reserved for batch size and \(\text{BW}\) denotes bandwidth).
- When data will not move: Start with table 1 and the transfer-time equation in section 1.1.1.
- When the accelerator is starving: Use table 2 and the layout discussion in section 1.1.3.
- When pipelines explode in cost: Use the primitives in section 1.1.4, especially join-induced shuffles.
- When “average looks fine” but users complain: Use section 1.2.1 and session-level tail probability.
- When accuracy drifts silently: Use section 1.2.2 to compare full distributions.
The data landscape can feel like a zoo of formats, encodings, and edge cases, but this appendix focuses on the structural constants that constrain every ML system.
With those reference points in place, the appendix begins with the physical constraints of data engineering and then moves to the statistical monitoring that keeps ML systems healthy. From storage formats that determine I/O throughput to drift metrics that detect silent failures, these foundations connect directly to the data pipelines in Data Pipeline Architecture and the operational monitoring in Model and infrastructure monitoring.
Data Engineering Foundations
Understanding hardware constraints is only half the battle; we must also shape our data to fit them. Data engineering applies the principles of the memory hierarchy to storage formats and pipeline design, ensuring that the accelerator never starves. This process begins with recognizing that data is physical—it has volume, it takes time to move, and it requires energy to parse.
Napkin math: The physics of data gravity
Data gravity1 is a metaphor grounded in transfer-time calculations. Unlike compute, which gets faster every year, the speed of light is fixed and network bandwidth is a finite resource. When datasets grow large enough, their practical inertia grows—moving them can cost more time and energy than moving computation to where the data already lives.
1 Data gravity: Coined by Dave McCrory in 2010 to describe how large datasets attract services and applications toward them, much as massive bodies attract smaller ones in physics (McCrory 2010). The analogy is apt: the “escape velocity” required to move a petabyte-scale dataset is often measured in weeks.
The ideal line-rate transfer time is \(T = D_{\text{vol}}/\text{BW}\) after converting data volume and bandwidth to compatible units (the calculations below convert decimal bytes and bits per second explicitly). For the large volumes here, latency is negligible; the full equation appears in Bandwidth vs. latency. Table 1 gives lower bounds; protocol overhead and endpoint limits increase them.
| Data Volume | 1 Gbps (Standard WAN) | 10 Gbps (High-End WAN) | 100 Gbps (Direct Connect) | Truck (Illustrative) |
|---|---|---|---|---|
| 1 TB | 2.2 hours | 13.3 minutes | 80 seconds | N/A |
| 100 TB | 9 days | 22.2 hours | 2.2 hours | N/A |
| 1 PB | 3 months | 9 days | 22.2 hours | 2 days |
The cost of serialization
Even after data arrives at the machine, the serialization tax remains a final hurdle.2 As table 2 shows, many engineers meticulously optimize their accelerator kernels while ignoring the CPU overhead of decoding data. Parsing text-based formats like JavaScript Object Notation (JSON) or CSV is extremely CPU-intensive, often leaving the accelerator idling while the CPU struggles to convert strings into floating-point numbers.
2 Serialization: From Latin serialis (forming a series). The process of converting in-memory data structures into a byte stream for storage or transmission, and the reverse (deserialization). In ML pipelines, the choice of serialization format can dominate end-to-end training time; Data ingestion covers pipeline design strategies that minimize this overhead.
| Format | Decoding Speed (MB/s) | Relative CPU Decode Cost | Suitability |
|---|---|---|---|
| CSV/JSON | ~100 MB/s | High | Inspection/interchange |
| Protobuf | ~300 MB/s | Medium | RPC/Messages |
| Parquet | ~1,000 MB/s | Low | Columnar storage |
Row vs. columnar formats
The choice of file format determines the “physics” of how the data is read. Row-oriented formats such as CSV and JSON store data record-by-record, so reading the age field still requires parsing each row. This suits appended logs but not analytics on selected features. Parquet stores compressed column chunks on disk, while Arrow provides a columnar in-memory and IPC format. Parquet readers can use projection pushdown to fetch selected column chunks, and both formats support vectorized processing. Compare the two arrangements in figure 1 to see why columnar access avoids scanning unnecessary bytes.
\begin{tikzpicture}[font=\small\sffamily]
\tikzset{
DataBlock/.style={draw=GrayLine, line width=0.8pt, node distance=-0.8pt,
minimum height=8mm, minimum width=24mm, align=center, font=\small\sffamily},
Label/.style={text=TextBlack, font=\small\bfseries\sffamily}
}
\tikzset{mycylinder/.style={cylinder, shape border rotate=90, aspect=1.3, draw, fill=white,
minimum width=25mm,minimum height=11mm,line width=\Linewidth,node distance=-0.15},
pics/data/.style = {
code = {
\pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=STREAMING,scale=\scalefac, every node/.append style={transform shape}]
\node[mycylinder,fill=\filllcolor!50] (A) {};
\node[mycylinder, above=of A,fill=\filllcolor!30] (B) {};
\node[mycylinder, above=of B,fill=\filllcolor!10] (C) {};
\end{scope}
}
}
}
\tikzset{pics/lupa/.style = {
code = {
\pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=BRAIN,scale=\scalefac, every node/.append style={transform shape}]
\coordinate(PO)at(-0.1,0.2);
\node[circle,draw=white,line width=1pt,fill=\filllcirclecolor,minimum size=5mm,inner sep=0pt](LV)at(PO){};
\node[draw=none,rotate=40,rounded corners=2pt,rectangle,minimum width=1.2mm,inner sep=1pt,
fill=\filllcirclecolor,minimum height=6mm,anchor=north]at(PO){};
\node[circle,draw=none,fill=white,minimum size=3.0mm,inner sep=0pt](LM)at(PO){};
\node[font=\tiny\bfseries]at(LM){...};
\end{scope}
}
}
}
\pgfkeys{
/channel/.cd,
Depth/.store in=\Depth,
Height/.store in=\Height,
Width/.store in=\Width,
filllcirclecolor/.store in=\filllcirclecolor,
filllcolor/.store in=\filllcolor,
drawcolor/.store in=\drawcolor,
drawcircle/.store in=\drawcircle,
scalefac/.store in=\scalefac,
Linewidth/.store in=\Linewidth,
picname/.store in=\picname,
filllcolor=BrownLine,
filllcirclecolor=violet!20,
drawcolor=black,
drawcircle=violet,
scalefac=1,
Linewidth=0.5pt,
Depth=1.3,
Height=0.8,
Width=1.1,
picname=C
}
% Row Store
\node[DataBlock, fill=RedFill!60] (r1a) at (0, 2.8) {ID: 1};
\node[DataBlock, fill=cyan!20, right=of r1a] (r1b) {Age: 25};
\node[DataBlock, fill=green!80!black!20, right=of r1b] (r1c) {Loc: US};
%
\node[DataBlock, fill=RedFill!60, right=7mm of r1c] (r2a) {ID: 2};
\node[DataBlock, fill=cyan!20, right=of r2a] (r2b) {Age: 30};
\node[DataBlock, fill=green!80!black!20, right=of r2b] (r2c) {Loc: UK};
%%fitting arrow
\node[draw=none,fill=none, inner ysep=3mm, inner xsep=6mm,fit=(r1a)(r2c),xshift=-3mm](A){};
\coordinate(AL)at($($(A.north west)!0.5!(A.south west)$)+(0.6,0)$);
\coordinate(AD)at($($(A.north east)!0.5!(A.south east)$)+(0.6,0)$);
\scoped[on background layer]
\draw[draw=none,fill=black!10](A.north west)--(A.north east)--(AD)--(A.south east)--(A.south west)--(AL)--cycle;
\node[Label,anchor=south,font=\small\sffamily\bfseries] at (A.north) {Row-Oriented (CSV)};
\node[anchor=north east]at(A.south east) {Scan reads \textbf{all fields}};
% Column Store
\node[DataBlock, fill=RedFill!60,below=1.25 of A] (c1a) {ID: 1};
\node[DataBlock, fill=cyan!20, below=of c1a] (c2a) {Age: 25};
\node[DataBlock, fill=green!80!black!20, below=of c2a] (c3a) {Loc: US};
\node[DataBlock, fill=RedFill!60, right=of c1a] (c1b) {ID: 2};
\node[DataBlock, fill=cyan!20, below=of c1b] (c2b) {Age: 30};
\node[DataBlock, fill=green!80!black!20, below=of c2b] (c3b) {Loc: UK};
%%fitting arrow
\node[draw=none,fill=none, inner ysep=0mm, inner xsep=10mm,fit=(c2a)(c2b),xshift=-3mm](B){};
\coordinate(BL)at($($(B.north west)!0.5!(B.south west)$)+(0.4,0)$);
\coordinate(BD)at($($(B.north east)!0.5!(B.south east)$)+(0.4,0)$);
\scoped[on background layer]
\draw[draw=none,fill=black!10](B.north west)--(B.north east)--(BD)--(B.south east)--(B.south west)--(BL)--cycle;
\node[Label,anchor=south,above=2pt of c1a.north east,font=\small\sffamily\bfseries] {Column-Oriented (Parquet)};
\node[anchor=north west]at(B.south east) {Scan reads \textbf{\color{cyan}ONLY Age}};
%icons
\pic[shift={(-1.3,-0.5)}] at (AL){data={scalefac=0.4,picname=1,filllcolor=VioletLine, Linewidth=0.6pt}};
\pic[shift={(-0.8,-0.05)}] at (BL){lupa={scalefac=1,picname=1,filllcirclecolor=VioletLine!60, Linewidth=0.5pt}};
\end{tikzpicture}The layouts in figure 1 differ in how many irrelevant bytes enter the training path.
Systems Perspective 1.1: The accelerator starvation problem
The algebra of data
Feature engineering turns raw records into the columns a model can learn from, and that transformation is usually a dataflow problem before it is a modeling problem. A pipeline first narrows the population, then chooses the features, then attaches context from other tables. Those three moves correspond to three Structured Query Language (SQL) primitives, and their computational cost determines whether the feature job stays local, scans unnecessary bytes, or turns into a network shuffle.
- Selection (\(\sigma\)): Filtering rows (for example,
WHERE age > 30).- Cost: \(\mathcal{O}(\log N + K)\) if a B-tree index lookup returns \(K\) rows; \(\mathcal{O}(N)\) for a full scan.
- Projection (\(\pi\)): Selecting columns (for example,
SELECT age).- Cost: Selected chunks plus decoding and metadata in columnar formats. This row-format example reads 1 KB for a 4-byte integer, wasting 99.6 percent of the read.
- Join (\(\bowtie\)): Combining tables. The most expensive operation.
- Shuffle Join: Both tables are partitioned by key and exchanged over the network.
- Cost: Reshuffling both 1 TB tables moves about ~2 TB; partitioning and locality can reduce this.
- Broadcast Join: One small table is sent to all workers.
- Cost: \(S(W-1)\) bytes; the \(S\)-byte table must fit on each of \(W\) workers.
- Shuffle Join: Both tables are partitioned by key and exchanged over the network.
Understanding data formats, serialization costs, and algebraic primitives tells us how to move data efficiently. Even a perfectly engineered pipeline, however, can silently fail if the data it carries changes character over time. Detecting that change—and quantifying how much it matters—requires a different set of tools: probability and statistics.
Probability and Statistics
Once data is flowing through our pipelines, we need mathematical tools to ensure its quality and consistency. Probability and statistics provide the language for monitoring system health, detecting the silent failures of data drift, and managing uncertainty.
Distributions and the long tail
In systems, the mean is often misleading. Latency distributions are often long-tailed, and user-visible services are commonly governed by high-percentile behavior rather than mean response time (Dean and Barroso 2013). A “P99” (99th percentile) latency of 500 ms means at most 1 percent of requests take longer than 500 ms; with many requests per session, the fraction of users who see at least one slow request can be much higher. At scale (1M users), even a 1 percent affected-user rate would affect 10,000 users.
Napkin Math 1.1: The median experience
Systems insight: Under this model, a 100-request session is more likely than not to include an event beyond P99. Correlated slowdowns can change this probability (Dean and Barroso 2013).
Measuring drift (divergence)
Because distributions have long tails where the most dangerous failures hide, simple metrics like “mean shift” are insufficient. We need tools that compare the entire shape of the distribution. Detecting drift between the serving distribution (\(P_t\)) and training distribution (\(P_0\)) requires measuring the “distance” between distributions.
For discrete outcomes \(x\), equation 1 expresses KL divergence3 as the expected extra coding cost of approximating \(P_t\) with \(P_0\) (Kullback and Leibler 1951; Cover and Thomas 2006). The displayed form is finite only when \(P_0(x) > 0\) wherever \(P_t(x) > 0\); production implementations therefore need explicit smoothing or zero handling. \[ \mathcal{D}_{\text{KL}}(P_t \lVert P_0) = \sum_x P_t(x) \log \frac{P_t(x)}{P_0(x)} \tag{1}\]
3 KL divergence: Named after Solomon Kullback and Richard Leibler, who introduced it in 1951. Also called relative entropy, it quantifies the expected extra information needed to encode samples from one distribution using a code optimized for another; for drift monitoring in this volume, the canonical direction is \(\mathcal{D}_{\text{KL}}(P_t \lVert P_0)\). With natural logarithms, as in the example above, the unit is nats; with base-2 logarithms, the unit is bits. In production ML, KL divergence is the theoretical backbone of many drift-detection metrics.
Napkin Math 1.2: Worked example: KL divergence for drift detection
Math:
Training distribution \(P_0\): [0.60, 0.30, 0.10]. Serving distribution \(P_t\): [0.45, 0.40, 0.15].
\[\begin{gather*} \mathcal{D}_{\text{KL}}(P_t \lVert P_0) = 0.45 \log\frac{0.45}{0.60} + 0.40 \log\frac{0.40}{0.30} + 0.15 \log\frac{0.15}{0.10}\end{gather*}\] \[\begin{gather*} = 0.45 \times (-0.2877) + 0.40 \times (0.2877) + 0.15 \times (0.4055) \\[-1pt] = -0.1295 + 0.1151 + 0.0608 = 0.0464\text{ nats}\end{gather*}\] \[\begin{gather*} \mathcal{D}_{\text{KL}}(P_0 \lVert P_t) = 0.60 \log\frac{0.60}{0.45} + 0.30 \log\frac{0.30}{0.40} + 0.10 \log\frac{0.10}{0.15}\end{gather*}\] \[\begin{gather*} = 0.60 \times 0.2877 + 0.30 \times (-0.2877) + 0.10 \times (-0.4055) \\[-1pt] = 0.1726 + (-0.0863) + (-0.0405) = 0.0458\text{ nats}\end{gather*}\]
Notice the asymmetry: \(\mathcal{D}_{\text{KL}}(P_0 \lVert P_t) \neq \mathcal{D}_{\text{KL}}(P_t \lVert P_0)\). The population stability index (PSI) symmetrizes this; the calculation below uses natural logarithms:
\[\begin{gather*} \text{PSI} = \sum (P_{0,i} - P_{t,i}) \log\frac{P_{0,i}}{P_{t,i}} = (0.15)(0.2877) + (-0.10)(-0.2877) + (-0.05)(-0.4055) \\[-1pt] = 0.04315 + 0.02877 + 0.02027 = 0.0922 \end{gather*}\]
Systems insight: Since PSI = 0.0922 < 0.2, this drift does not yet cross the illustrative manual-review threshold used here. The right response is to keep the feature on the monitoring dashboard and look for corroborating drift signals rather than retraining from this statistic alone.
The threshold remains illustrative: its meaning depends on sample size, binning, smoothing, and zero handling, and retraining requires model-quality evidence.
Both KL divergence and PSI are grounded in a deeper framework—information theory—which provides the units and bounds that make these metrics principled rather than ad hoc.
Information theory for systems
A training run can consume more examples, run longer, and still stop improving if the added data carries too little useful signal. At that point, the bottleneck is not only compute or bandwidth; it is the amount of information the data pipeline delivers to the learner. Information roofline (the destination) treats that data quality limit as a diagnostic constraint, and information theory provides the units for reasoning about it.
Entropy (\(H\))4 is the average uncertainty in a distribution, defined as \(H(X) = -\sum p(x) \log p(x)\). The log base sets the unit: natural logs give nats, while \(\log_2\) gives bits. A uniform distribution has maximum entropy only over a fixed finite support. This connects directly to KL divergence above: \(\mathcal{D}_{\text{KL}}\) measures the excess coding cost of using the wrong distribution.
4 Entropy: Shannon adopted the term in 1948 to quantify information content (Shannon 1948). In systems terms, Shannon entropy is a lower bound on average lossless code length. Practical prefix codes approach it, and block codes can approach it asymptotically. With \(\log_2\), the unit is bits; natural logs give nats. These bounds inform compression ratios and data-pipeline sizing.
Information density is used here as a book-level heuristic for useful learning signal per unit of storage, not as a standard information-theory quantity.
Signal-to-noise ratio (SNR) is the ratio of signal power to noise power. In ML, the book uses low SNR qualitatively for data in which task-relevant structure is weak relative to noise; adding compute cannot compensate for missing task-relevant signal.
Logits and numerical stability
Neural-network classifiers commonly output a vector of logits5 \(\mathbf{z}\) (unnormalized scores). For class index \(i\), softmax normalizes \(z_i\) over all class indices \(j\) to produce a probability:
5 Logit: Joseph Berkson coined the term in 1944 by analogy with probit. The logit function is the inverse of the logistic (sigmoid) function: \(\text{logit}(p) = \log(p/(1-p))\). In deep learning, “logits” refers more loosely to the raw, unnormalized output of the final linear layer before any activation function is applied.
\[ \text{Softmax}(z_i) = \frac{e^{z_i}}{\sum e^{z_j}} \]
The problem is that if \(z_i\) is large (for example, 100), the exponential \(e^{z_i}\) overflows common training and inference formats such as FP32, BF16, and FP16. FP64 can represent this specific value, but production ML kernels rarely use FP64 for softmax. The solution is to compute in log-space: the “Log-Sum-Exp” trick allows us to compute \(\log\left(\sum e^{z_j}\right)\) without ever calculating the massive exponentials directly, preserving numerical precision.6
6 Log-sum-exp: Implemented as torch.logsumexp in PyTorch and scipy.special.logsumexp in SciPy. It relies on the identity \(\log\left(\sum e^{x_i}\right) = a + \log\left(\sum e^{x_i - a}\right)\), where \(a = \max(x_i)\). Shifted values \(x_i - a\) are \(\le 0\), ensuring exponentials never overflow.
A small worked example shows why this trick matters in practice, even for large but realistic logit values:
Napkin Math 1.3: Log-sum-exp in action
Without the trick (naive softmax): \[\text{ $\exp(100) \approx 2.7 \times 10^{43}$, $\exp(101) \approx 7.3 \times 10^{43}$, $\exp(102) \approx 2.0 \times 10^{44}$}\]
These numbers are representable in FP64 but overflow FP32 (max \(\approx 3.4 \times 10^{38}\)). With FP16 (max \(\approx 65{,}504\)), even \(e^{12}\) overflows. Raw logits rarely reach magnitude 100, but large intermediate values do arise (for example, unscaled attention scores or overconfident outputs), and because FP16 and BF16 are the standard training precisions, a naive softmax risks overflow often enough that the stable form is used by default.
With the trick: Subtract \(a = \max(\mathbf{z}) =\) 102:
Shifted logits: \(\mathbf{z} - a =\) \(\lbrack -2, -1, 0 \rbrack\)
Exponentials: \(\exp(-2) \approx 0.135\), \(\exp(-1) \approx 0.368\), \(\exp(0) = 1.0\)
Sum = 1.503. LogSumExp = 102 \(+ \log(1.503) = 102.408\).
Softmax: \(\lbrack 0.135/1.503,\; 0.368/1.503,\; 1.0/1.503 \rbrack = \lbrack 0.090,\; 0.245,\; 0.665 \rbrack\)
Systems insight: The exponentiated shifted values and the resulting softmax probabilities are in \([0, 1]\)—no overflow risk, even in FP16.
Together, these worked examples connect the full chain, from physical data movement to distribution shift and numerical stability.
Summary
The tools in this appendix—tail-aware metrics, drift divergences, information-theoretic bounds, and numerical stability techniques—support quantitative diagnosis of data-related failures rather than anecdotal explanations.