Data Selection

Purpose

How can a carefully selected subset retain most of the value of a much larger dataset?

Training pays to acquire, label, score, move, and process examples even though their learning value differs. Large datasets contain redundant, noisy, or target-misaligned samples, while scarce regions may lack the cases a model needs. Data selection turns that heterogeneity into a systems optimization. It can remove avoidable work before training, direct labels and training steps toward informative cases as learning proceeds, or create missing examples when scarcity is the binding constraint. It can also invest once in reusable representation learning when that cost can be amortized across downstream tasks. The objective is therefore not the smallest dataset but the lowest end-to-end cost of reaching target quality without losing rare cases, underrepresented groups, or deployment-relevant coverage. Every strategy adds its own overhead: examples must be scored, generated, shuffled, stored, or validated, and those costs can erase the savings they promise. That judgment must be made end to end, because gains in labeling or training can be erased by selection latency, storage access, or distributed coordination. A method earns its place only when the learning it preserves or creates exceeds the work required to apply it. This distinction matters across repeated experiments, where useful reductions compound, and across adaptation, where the value of an example changes with the model and target distribution. Data engineering established that data is the source code of ML systems; data selection asks which evidence is useful now and which is still missing. In D·A·M terms, it is data-algorithm co-design: shape the training signal so each unit of machine work buys more useful learning.

Learning Objectives

Explain data selection as data-algorithm co-design that reduces total operations before training begins
Calculate information-compute ratio to decide whether additional examples improve learning per FLOP
Compare deduplication, coreset selection, and quality pruning for reducing redundant pretraining data
Design curriculum, active learning, or synthetic-data strategies for changing data value during training
Apply the selection inequality to test whether selection overhead beats full-dataset training cost
Evaluate distributed selection pipelines against storage locality, consistency, and GPU utilization constraints
Select data-selection investments using ROI, amortization, and compute-optimal frontier diagnostics

Data Selection Fundamentals

Training pays the iron-law cost for every example it processes, but not every example returns learning signal worth that cost. Data selection gives a clean, well-engineered dataset a systems objective: keep the examples that contribute the most learning per unit of compute. Data engineering makes the dataset reliable through correct labels, consistent schemas, and governed records. Data selection optimizes the dataset’s value by extracting maximum learning from minimum samples, directly shrinking the total operations $(O)$ term in the iron law (principle 3). The distinction matters: quality asks whether data is correct, while value asks whether correct data is worth the compute spent processing it.

Compute supply can outrun high-quality data supply.

¹ Scaling laws: Jared Kaplan and colleagues at Johns Hopkins and OpenAI empirically demonstrated in 2020 that language model loss follows power-law relationships with model size, dataset size, and compute budget over the regimes they studied (Kaplan et al. 2020). Their fitted data exponent was approximately $\alpha = 0.095$ in $\mathcal{L} \propto D^{-\alpha}$. The value is not universal, but the diminishing-return shape makes it possible to reason about when selection becomes more cost-effective than collection.

Kaplan, J., S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. 2020. “Scaling Laws for Neural Language Models.” ArXiv Preprint abs/2001.08361.

² Data wall: Unlike compute (which scales with capital expenditure) or algorithms (which improve through research), the stock of high-quality human-generated text grows slowly; Epoch AI’s updated projections estimated that, under the paper’s modeled consumption assumptions, models could use datasets roughly equal in size to the stock of public human-generated text between 2026 and 2032 (Villalobos et al. 2022). The point for systems design is not a fixed calendar deadline; it is that data can become a supply constraint rather than merely an economic one. This constraint directly affects the total operations $(O)$ term: when quality data becomes scarce, additional compute yields diminishing returns regardless of hardware throughput.

Villalobos, Pablo, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. 2022. “Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data.” arXiv Preprint arXiv:2211.04325.

Increasing the amount of training data has long been a standard way to improve models. Scaling laws¹ (Kaplan et al. 2020; Hoffmann et al. 2022) quantify how model performance can improve with dataset size, and teams have responded by collecting and generating more examples. Accelerator fleets, however, can expand usable compute faster than the supply of novel, high-quality human-generated text and images. Much of the easily accessible public web has already been incorporated into large training corpora, and expert labeling capacity grows slowly. This asymmetry is the data wall,² and it shifts attention from collecting more data to extracting more value from existing data. The sections that follow develop the selection methods and cost models needed to make that choice.

Table 1 uses an illustrative growth scenario in which GPU compute rises 10× every 3 years while accessible high-quality data grows more slowly. These are scenario inputs, not measured universal rates.

Table 1: Scaling Asymmetry in ML Resources: In the illustrated scaling regime, compute grows faster than high-quality data supply, creating a compute-to-data imbalance that makes data selection valuable.

Resource	Growth Rate	Implication
GPU Compute	~10× / 3 years	Hardware throughput can rise quickly in a given era
Training Data (Web)	~2× / 5 years	High-quality web text is finite; much already scraped
Labeled Data	~1.5× / 5 years	Human annotation throughput is inherently bounded
Synthetic Data	Potentially large	Bounded by generator quality (models trained on model-generated data can degrade)

Scaling dataset size eventually encounters the finite stock of high-quality human speech and text. Follow the log-scale trajectory in figure 1, observing how foundation model training requirements approach the estimated upper bound of available public data.

The gap between what compute can process and what accessible, high-quality data can support is therefore a systems variable, not a fixed law. In the regime illustrated by table 1, compute supply grows faster than accessible high-quality data, so accelerator budgets can outrun the corpus and leave the system compute-rich but data-constrained. Maintaining model relevance requires repeated refreshes of the training corpus as the world changes, and intelligent data selection becomes critical when data quality rather than accelerator time is the binding constraint.

\begin{tikzpicture}[font=\sffamily\small]

\pgfplotsset{
  layers/axis lines on top/.define layer set={
    axis background,
    axis grid,
    axis ticks,
    axis tick labels,
    pre main,
    main,
    axis lines,
    axis descriptions,
    axis foreground,
  }{/pgfplots/layers/standard},
}

\begin{axis}[
   /pgf/number format/.cd,
   tick label style={/pgf/number format/assume math mode=true},
   ticklabel style={font=\footnotesize\sffamily},
   1000 sep={}, % uklanja zareze
  yticklabel style={
  /pgf/number format/.cd,
  sci,
  sci generic={mantissa e exponent},
  precision=1
},
    width=15cm,
    height=75mm,
    xmin=2012, xmax=2028.5,
    ymin=1e8, ymax=1.4e15,
    ymode=log,
    log basis y=10,
    xtick={2012,2014,2016,2018,2020,2022,2024,2026,2028},
    ytick={1e8,1e9,1e10,1e11,1e12,1e13,1e14,1e15},
    yticklabels={10\textsuperscript{8},10\textsuperscript{9},
    10\textsuperscript{10},10\textsuperscript{11},
    10\textsuperscript{12},10\textsuperscript{13},
    10\textsuperscript{14},10\textsuperscript{15}},
    xlabel={Year},
    ylabel={Dataset Size (Tokens)},
    axis lines*=left,
    axis line style={gray!70,line width=1pt},
 % axis on top=true,
    set layers=axis lines on top,
    tick style={gray!70},
    tick label style={font=\sffamily\footnotesize},
    ylabel style={font=\small\sffamily,align=center,yshift=-1.2mm},
    xlabel style={font=\small\sffamily},
    grid=both,
    major grid style={gray!12},
    minor grid style={gray!6},
    clip=false,
    major tick style={thin,black},
   tick align=outside,
   major tick length=4pt,
]

%—shaded public text stock band ---
\addplot[
    draw=none,
    fill=myorange!07
] coordinates {
    (2012,1e14)
    (2028,1e14)
    (2028,1e15)
    (2012,1e15)
}; %\closedcycle

\node[
    anchor=center,
    align=center,
    text=myorange!75!black,
    font=\footnotesize\sffamily
] at (axis cs:2015.5,3e14)
{High-Quality Public Text Stock\\(Books, Papers, Code, Web)};

%—scaling line ---
\addplot[
    BlueLine,
    line width=1.5pt,
] coordinates {
    (2017.15,1e9)
    (2026.60,1e15)
};

%—data points ---
\addplot[
    only marks,
    mark=*,
    mark size=2.5,
    BlueLine,
    draw=white,line width=0.95pt
] coordinates {
    (2019.0,1.4e10)   % unlabeled low point
    (2020.0,3.0e11)   % GPT-3
    (2022.0,1.4e12)   % Chinchilla
    (2023.0,2.0e12)   % Llama 2
    (2024.0,1.5e13)   % Llama 3
};

%—point labels ---
\node[anchor=west, text=BlueLine, font=\sffamily\footnotesize]
    at (axis cs:2018.5,3.0e11) {GPT-3};

\node[anchor=west, text=BlueLine, font=\sffamily\footnotesize]
    at (axis cs:2022.0,9.0e11) {Chinchilla};

\node[anchor=west, text=BlueLine, font=\sffamily\footnotesize]
    at (axis cs:2023.1,2.0e12) {Llama 2};

\node[anchor=west, text=BlueLine, font=\sffamily\footnotesize]
    at (axis cs:2024.15,1.5e13) {Llama 3};

%—red curved arrow and label ---
\node[
    anchor=west,
    text=myred,
    font=\sffamily\footnotesize
](PD) at (axis cs:2015.4,1.2e12)
{Projected Data Limit};
\coordinate(PO)at(axis cs:2025.05,1.05e14);
\end{axis}
\draw[myred,line width=0.75pt, <-,>=latex]
(PO)
to[bend right=15]
(PD);
\end{tikzpicture}

Figure 1: Dataset Growth Approaching Limits: Several foundation-model datasets have grown toward estimates of the total stock of high-quality public text. The projection should be read as a dated constraint scenario rather than a fixed forecast: data selection, synthetic generation, multimodal learning, licensing, and filtering policy all change the usable supply.

The compute-data asymmetry can invert the optimization priority. When data is abundant and compute is scarce, algorithmic efficiency can extract more accuracy from limited GPU cycles. When compute is abundant and quality data is scarce, data selection can instead extract more learning from each sample. Data selection operates upstream of model and machine optimizations. By pruning redundancy and selecting high-value samples, the workload is reduced before it enters the model or reaches the hardware, directly shrinking the total operations $(O)$ term in the iron law. That is why the systems perspective treats selection as upstream workload reduction rather than a modeling heuristic. For teams whose accelerator budget exceeds their curated corpus, the bottleneck shifts from GPU access to the quality, legality, and diversity of the training data.

The engineering toolkit for intelligent data selection follows a deliberate optimization ordering: first ask whether a sample is worth processing, then ask how to process the remaining workload efficiently. Data selection puts the “largest return first” principle into practice by looking for avoidable work before simplifying or accelerating the work that remains. Static pruning removes low-value samples before a single gradient is computed. Dynamic selection adapts the data diet during training through curriculum learning and active learning. Synthetic generation creates high-value samples through augmentation, simulation, or teacher-generated examples when real data runs short.

Each stage can increase the information density of the data that reaches the model when measured quality gains justify its overhead, and together they form a complementary toolkit: pruning reduces what the pipeline contains, selection focuses how the pipeline uses it, and synthesis expands what the pipeline can access. Comparing these techniques requires a systems definition of data selection and a measurable account of its effectiveness.

Defining data selection

The three-stage pipeline needs a quantity that lets engineers compare samples before they spend accelerator time on them. A duplicated image, a mislabeled record, and a rare boundary case may all cost the same forward and backward pass, but they do not contribute the same learning signal. Data selection therefore starts by making sample value explicit relative to compute cost. The information-compute ratio measures that value as learning signal gained per unit of training compute, formalized in section 1.1.3.

Definition 1.1: Data selection

Data selection is the process of maximizing the information-compute ratio (ICR) of a training dataset.

Significance: It identifies data worth retaining, prioritizing, or generating for the target objective, reducing the total operations $(O)$ when redundant or noisy samples can be excluded without losing required coverage.
Distinction: Unlike data engineering, which focuses on the cleanliness and consistency of data, data selection focuses on the informativeness and diversity of the samples.
Common pitfall: A frequent misconception is that more data is always better. Additional low-quality data can yield less improvement than a much smaller, carefully selected set, so retained volume must be evaluated together with coverage and target quality.

To make this concrete, consider training an autoregressive language model in the GPT-2/Llama lighthouse family from Lighthouse roster: Model biographies. The scenario uses a 70-billion-parameter model to make the compute-data gap visible at large scale.

The compute budget (10,000 H100 GPUs for 3 months) represents roughly $86.4M at the chapter’s cloud-training price anchor and can process about 73.2T tokens at 40 percent sustained model FLOPs utilization. Estimates of quality- and repetition-adjusted public human-generated text are on the order of 300T tokens, far larger than a single curated web corpus. The practical bottleneck is narrower: how much of that stock is accessible, legally usable, high quality, deduplicated, and useful for the target distribution. If a team has only a 5T filtered corpus ready for use, the compute budget can already process it roughly 14.6× over. At that point, the team faces three options:

Repeat epochs: The team can train on the same data for multiple epochs, but returns can diminish as examples are reused; the useful epoch count is workload-dependent.
Lower quality thresholds: The team can include more data, but lower-quality tokens can degrade model quality.
Invest in data selection: The team can improve filtering, curriculum design, and synthetic augmentation to extract more learning from each token.

Under these assumptions, the decision criterion favors selection over buying more accelerator time.

The opportunity applies across model architectures, though the bottlenecks and safe reduction differ. Unlike our compute-bound ResNet-50 lighthouse, GPT-2/Llama models can be memory-bandwidth-bound during inference and compute-bound during training. When a validated selection method removes examples or tokens without adding compensating steps, fewer samples mean fewer training FLOPs. The appropriate framing is therefore systemic rather than purely statistical.

Systems perspective

The data wall establishes why data selection matters; the systems perspective reveals how to approach it effectively. The conventional ML framing focuses on achieving the same accuracy with fewer samples, centering on statistical sample complexity and generalization theory. While valid, that framing misses the larger picture.

A data-selection systems framing asks instead how to reduce the total cost of achieving target performance across the entire ML lifecycle. The shift moves attention from accuracy curves to resource consumption, as table 2 illustrates.

Table 2: ML vs. Systems Perspectives on Data Selection: The ML framing optimizes sample complexity; the systems framing optimizes total resource cost across the pipeline.

ML Framing	Systems Framing
“Fewer samples for same accuracy”	“Fewer FLOPs for same accuracy”
“Better generalization”	“Lower training cost (time, money, energy)”
“Sample complexity bounds”	“End-to-end resource efficiency”
“Learning theory”	“Cost engineering”

The systems framing reveals optimization opportunities invisible to the ML framing. To see why, consider how data selection interacts with the iron law introduced in Iron Law of ML Systems.

Systems Perspective 1.1: Data selection and the iron law

In the iron law of ML systems $(T = \frac{D_{\text{vol}}}{\text{BW}} + \frac{O}{R_{\text{peak}} \cdot \eta_{\text{hw}}} + L_{\text{lat}})$, data selection reduces the number of examples or tokens that enter the workload. Later model-level optimizations reduce operations per forward/backward pass, and machine-level optimizations increase $R_{\text{peak}}$ (peak throughput) and $\eta_{\text{hw}}$ (utilization). Selection removes work before those downstream optimizations execute it.

This makes data selection multiplicatively valuable in the iron law: when all three optimization layers act on the same bottleneck, a 2× reduction in dataset size with 2× fewer operations per sample and 2× higher effective throughput yields 8× total cost reduction, not 6×.

Consider training cost reduction. With the epoch count, batch size, and per-sample work held fixed, a 50 percent reduction in dataset size halves the number of forward passes, backward passes, and gradient updates. For the $100M scenario used here, this translates to $50M in compute savings. Schedules that add epochs or steps reduce that saving.

Compute savings cascade through the entire infrastructure stack. Large datasets consume petabytes of storage and saturate network bandwidth during distributed training; deduplication and coreset selection reduce storage costs while eliminating I/O bottlenecks that can idle expensive GPU clusters. The savings extend to labeling economics: expert labeling can exceed compute costs, and active learning and semi-supervised methods can substantially reduce labeling budgets in favorable regimes. The environmental implications compound further: for compute-dominated training runs, reducing the number of examples reduces energy consumption in proportion to the work avoided, provided the selected subset preserves accuracy. Smaller curated datasets also enable faster iteration velocity. A team that can iterate in hours rather than days has a compounding advantage in model development.

The cascading benefits illustrate a broader point: the ML researcher usually frames the problem as sample complexity, while the systems engineer frames it as cost-per-accuracy-point across the entire pipeline, from data acquisition through deployment. The systems engineer’s toolkit for that problem includes techniques to minimize total cost, metrics to quantify efficiency gains, and architectural patterns to implement data selection at scale.

Information-compute ratio

The systems framing established in section 1.1.2 calls for a quantitative metric. Data selection creates a frontier between accuracy and cost: keeping every example maximizes coverage but wastes compute on redundancy, while pruning too aggressively saves compute but loses signal. We need a way to measure where a sample sits on that frontier. The metric is the information each sample contributes to the model’s learning per unit of computation. We formalize it as the information-compute ratio.

Figure 2 recasts the D·A·M taxonomy as an optimization map, with data selection playing the role of input optimization: reducing total workload before it enters the model or hardware. The model side asks how much math each example requires. The machine side asks how quickly the hardware can execute that math. The data side asks whether the example should be processed at all. The three edges of the triangle capture the dominant bottlenecks: compute bound describes systems limited by arithmetic throughput, I/O bound describes systems limited by data movement, and sample efficiency describes systems limited by the information content of training data.

\scalebox{0.8}{%
\begin{tikzpicture}[line join=round,font=\sffamily\footnotesize]
\tikzset{%
Box2/.style={align=flush center, inner sep=0pt,draw=none,fill=none,minimum height=8mm },
planet/.style = {circle, draw=none,semithick, fill=yellow!10,line width=1.5pt,
                    font=\sffamily\bfseries,
                    minimum size=27mm, inner sep=1mm,align=flush center},
satellite/.style = {circle, draw=#1, dashed, thick, fill=none,%#1!10,
                    text width=26mm, inner sep=1pt, align=flush center,minimum size=28mm,minimum height=12mm},
TxtC/.style = {font=\footnotesize\sffamily,text width=50mm,align=flush center},
LineA/.style = {violet!60,{Kite[line width=1.1pt,fill=white,round,length=9pt,width=7pt]}-,line width=1.5pt,shorten <=-3pt},
ALineA/.style={{Triangle[width=14pt,length=10pt]}-{Triangle[width=14pt,length=10pt]}, line width=7pt,cyan!40}
}

\tikzset{mycylinder/.style={cylinder, shape border rotate=90, aspect=1.3, draw, fill=white,
minimum width=25mm,minimum height=11mm,line width=\Linewidth,node distance=-0.15},
pics/data/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=STREAMING,scale=\scalefac, every node/.append style={transform shape}]
\node[mycylinder,fill=\filllcolor!50] (A) {};
\node[mycylinder, above=of A,fill=\filllcolor!30] (B) {};
\node[mycylinder, above=of B,fill=\filllcolor!10] (C) {};
\fill[\filllcolor!50!black]($(C.west)!0.12!(C.east)$)circle(3pt);
\fill[\filllcolor!50!black]($(B.west)!0.12!(B.east)$)circle(3pt);
\fill[\filllcolor!50!black]($(A.west)!0.12!(A.east)$)circle(3pt);
 \end{scope}
     }
  }
}

%funnel
\tikzset{%
 pics/funnel/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=FUNNEL,scale=\scalefac, every node/.append style={transform shape}]
\draw[fill=\filllcolor!50,line width=\Linewidth,draw=\drawcolor](-0.12,-0.81)--(-0.19,-0.25)--(-0.7,0.41)--(0.7,0.41)--(0.19,-0.25)--(0.12,-0.81)--cycle;
\draw[fill=\filllcolor!50,line width=\Linewidth,draw=\drawcolor](-0.19,-0.25)--(0.08,-0.25);
\draw[fill=\filllcolor!50,line width=\Linewidth,draw=\drawcolor](0.16,-0.09)--(0.41,0.31);
%
\node[line width=\Linewidth,draw=\drawcolor,fill=\filllcolor!50,inner sep=1pt,
rectangle,rounded corners=0.5pt,minimum width=16mm,minimum height=5pt]at(0,0.5){};
%
\foreach \i in{-0.5,0,0.5}{
\node[single arrow, line width=0.8*\Linewidth,draw=black,fill=\filllcirclecolor, rotate=270,inner sep=1pt,
      minimum width =9pt, single arrow head extend=2pt,
      minimum height=5mm]at(\i,0.9) {}; % length of arrow
   }
\node[single arrow,line width=0.8*\Linewidth,draw=black,fill=\filllcirclecolor, rotate=270,inner sep=1pt,
      minimum width =11pt, single arrow head extend=2pt,
      minimum height=5mm]at(0,-1.1) {}; % length of arrow
 \end{scope}
     }
  }
}
%CPU
\tikzset{%
 pics/cpu/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=FUNNEL,scale=\scalefac, every node/.append style={transform shape}]
\node[fill=\filllcolor,minimum width=66, minimum height=66,
            rounded corners=2,outer sep=2pt] (C1) {};
\node[fill=white,minimum width=54, minimum height=54] (C2) {};
\node[fill=\filllcolor!40,minimum width=44, minimum height=44] (C3) {\large CPU};

\foreach \x/\y in {0.11/1,0.26/2,0.41/3,0.56/4,0.71/5,0.85/6}{
\node[fill=\filllcolor,minimum width=3, minimum height=15,
           inner sep=0pt,anchor=south](GO\y)at($(C1.north west)!\x!(C1.north east)$){};
}
\foreach \x/\y in {0.11/1,0.26/2,0.41/3,0.56/4,0.71/5,0.85/6}{
\node[fill=\filllcolor,minimum width=3, minimum height=15,
           inner sep=0pt,anchor=north](DO\y)at($(C1.south west)!\x!(C1.south east)$){};
}
\foreach \x/\y in {0.11/1,0.26/2,0.41/3,0.56/4,0.71/5,0.85/6}{
\node[fill=\filllcolor,minimum width=15, minimum height=3,
           inner sep=0pt,anchor=east](LE\y)at($(C1.north west)!\x!(C1.south west)$){};
}
\foreach \x/\y in {0.11/1,0.26/2,0.41/3,0.56/4,0.71/5,0.85/6}{
\node[fill=\filllcolor,minimum width=15, minimum height=3,
           inner sep=0pt,anchor=west](DE\y)at($(C1.north east)!\x!(C1.south east)$){};
}
 \end{scope}
     }
  }
}
\tikzset{
pics/algorithm/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
\node[fill=\filllcolor!60,draw=\drawcolor,line width=\Linewidth,rectangle,minimum size=2.5mm,
rounded corners=1pt,inner sep=1pt](B2)at(0,-0.47){};
\node[fill=\filllcolor!30,draw=\drawcolor,line width=\Linewidth,rectangle,minimum size=2.5mm,
rounded corners=1pt,inner sep=1pt](B3)at(-0.6,-0.47){};
\node[fill=\filllcolor!30,draw=\drawcolor,line width=\Linewidth,rectangle,minimum size=2.5mm,
rounded corners=1pt,inner sep=1pt](B1)at(0.6,-0.47){};
%
\node[fill=\filllcolor!99!violet!80,draw=\drawcolor,line width=\Linewidth,rectangle,
minimum width=8.5mm,minimum height=3mm,
rounded corners=2pt,inner sep=1pt](B0)at(0,0.53){};
\draw[draw=\drawcolor,shorten >=4pt,shorten <=4pt,line width=1.5*\Linewidth]
($(B0.north west)!0.33!(B0.south west)$)--($(B0.north east)!0.33!(B0.south east)$);
\draw[draw=\drawcolor,shorten >=4pt,shorten <=4pt,line width=1.5*\Linewidth]
($(B0.north west)!0.66!(B0.south west)$)--($(B0.north east)!0.66!(B0.south east)$);
\draw[draw=\drawcolor,rounded corners](B1)|-(B0);
\draw[draw=\drawcolor,rounded corners](B3)|-(B0);
\draw[draw=\drawcolor,rounded corners](B2)--(B0);
\node[fill=\filllcirclecolor!60,draw=\drawcolor,line width=\Linewidth,rectangle,minimum size=2.5mm,
rounded corners=1pt,inner sep=1pt,rotate=45](R2){};
\node[fill=\filllcirclecolor!30,draw=\drawcolor,line width=\Linewidth,rectangle,minimum size=2.5mm,
rounded corners=1pt,inner sep=1pt,rotate=45](R1) at (-0.6,0){};
\node[fill=\filllcirclecolor!30,draw=\drawcolor,line width=\Linewidth,rectangle,minimum size=2.5mm,
rounded corners=1pt,inner sep=1pt,rotate=45](R3) at (0.6,0){};
\end{scope}
    }
  }
}
%vaga
\tikzset{
pics/vaga/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
\node[rectangle,minimum width=2mm,minimum height=22mm,
draw=none, fill=\filllcolor,line width=\Linewidth](1R) at (0,-0.95){};
\fill[fill=\filllcolor!60!black](230:2.8)arc(230:310:2.8)--cycle;%circle(2.9);
%LT
\node [semicircle, shape border rotate=180,  anchor=chord center,
      minimum size=11mm, draw=none, fill=\filllcirclecolor](LT) at (-2,-0.5) {};
\node [circle,  minimum size=4mm, draw=none, fill=\filllcirclecolor](T1) at (-2,1.25) {};
\draw[draw=\drawcolor,,line width=1.2*\Linewidth,shorten <=3pt,shorten >=3pt](T1)--(LT);
\draw[draw=\drawcolor,,line width=1.2*\Linewidth,shorten <=3pt,shorten >=3pt](T1)--(LT.30);
\draw[draw=\drawcolor,,line width=1.2*\Linewidth,shorten <=3pt,shorten >=3pt](T1)--(LT.150);
%DT
\node [semicircle, shape border rotate=180,  anchor=chord center,
      minimum size=11mm, draw=none, fill=\filllcirclecolor!70!black](DT) at (2,-0.5) {};
\node [circle,  minimum size=4mm, draw=none, fill=\filllcirclecolor!70!black](T2) at (2,1.25) {};
\draw[draw=\drawcolor,line width=1.2*\Linewidth,shorten <=3pt,shorten >=3pt](T2)--(DT);
\draw[draw=\drawcolor,,line width=1.2*\Linewidth,shorten <=3pt,shorten >=3pt](T2)--(DT.30);
\draw[draw=\drawcolor,,line width=1.2*\Linewidth,shorten <=3pt,shorten >=3pt](T2)--(DT.150);
%
\node[draw=none,rectangle,minimum width=32mm,minimum height=1.5mm,inner sep=0pt,
fill=\filllcolor!60!black]at(0,1.25){};
\node[draw=white,fill=\filllcolor,line width=2*\Linewidth,ellipse,minimum width=9mm,  minimum height=15mm](EL)at(0,0.85){};
\node[draw=white,fill=\filllcolor!60!black,line width=2*\Linewidth,,circle,minimum size=10mm](2C)at(0,2.05){};
\end{scope}
    }
  }
}
\pgfkeys{
  /channel/.cd,
   Depth/.store in=\Depth,
  Height/.store in=\Height,
  Width/.store in=\Width,
  filllcirclecolor/.store in=\filllcirclecolor,
  filllcolor/.store in=\filllcolor,
  drawcolor/.store in=\drawcolor,
  drawcircle/.store in=\drawcircle,
  scalefac/.store in=\scalefac,
  Linewidth/.store in=\Linewidth,
  picname/.store in=\picname,
  filllcolor=BrownLine,
  filllcirclecolor=violet!20,
  drawcolor=black,
  drawcircle=violet,
  scalefac=1,
  Linewidth=0.5pt,
  Depth=1.3,
  Height=0.8,
  Width=1.1,
  picname=C
}
\definecolor{Siva}{RGB}{161,152,130}
\def\radius{3.15}
%planet
\node (p)   [planet]    {};
\node[TxtC,below=0pt of p]{ML\\ Performance};
%satellites

\foreach \i/\j/\sho [count=\k from 0] in {
magenta!70!/{Algorithms (Model)}/105pt,
Siva/{Data Selection}/105pt,
cyan/{Machine (Hardware)}/105pt
%violet!75!/{\textbf{Embodiment}\\{\footnotesize No grounding}}/11pt%,
%orange/\textbf{Alignment}\\ {\footnotesize Value loading}/11pt
}
{
\def\startangle{90}
%Satelit
\pgfmathsetmacro{\angle}{\startangle - \k * (360/3)}
\node (s\k) [satellite=\i, font=\footnotesize\sffamily] at (\angle:\radius+\sho) {};
\node[TxtC,below=0pt of s\k]{\j};
%Arrows
%\draw[arr=\i,shorten >=\sho] (p) --coordinate[pos=0.35](AR\k) (s\k);
}

 \pic[shift={(0,0)}] at  (0,0){vaga={scalefac=0.35,picname=1,filllcolor=BlueLine,  Linewidth=1.0pt,filllcirclecolor=orange}};
 %algorithm
 \pic[shift={(0,0)}] at  (s0){algorithm={scalefac=1.2,picname=1,
drawcolor=black,filllcolor=orange!80!, Linewidth=0.75pt,filllcirclecolor=green}};
%data
\begin{scope}[local bounding box=DATA1,shift={($(0,-0.66)+(s1)$)},
scale=0.6, every node/.append style={transform shape}]
\pic[shift={(0,0)}] at  (0,0){data={scalefac=1,picname=1,filllcolor=BlueLine, Linewidth=0.7pt}};
\node[draw=black!70,line width=1pt,yshift=-3mm,fill=white,circle,minimum size=18mm](CIR)at(A.north east){};
\pic[shift={(0,0)}] at  (CIR){funnel={scalefac=0.6,picname=1,Linewidth=0.5pt,
 filllcolor=BrownL,drawcolor=black,filllcirclecolor=green}};
 \end{scope}
  %machine - CPU
 \pic[shift={(0,0)}] at  (s2){cpu={scalefac=0.55,picname=1,filllcolor=BrownLine, Linewidth=0.7pt}};
 %Arrows
 \draw[ALineA] (60:\radius) arc[radius=\radius, start angle=60, end angle= 0];
\coordinate (AR1) at (30:\radius);

\draw[ALineA] (300:\radius) arc[radius=\radius, start angle=300, end angle= 240];
\coordinate (AR2) at (270:\radius);

\draw[ALineA] (180:\radius) arc[radius=\radius, start angle=180, end angle= 120];
\coordinate (AR3) at (150:\radius);
%%%%%%%%%
 \path[LineA](AR1)--++(0:1.0)coordinate(MA);
 \node[Box2,anchor=west](T1)at(MA){Sample Efficiency};
 \draw[violet!50,line width=2pt](T1.north west)to[bend right=25]coordinate(AR11)(T1.south west);
 \draw[LineA](AR1)--(AR11);

 \path[LineA](AR2)--++(270:0.65)coordinate(MA2);
 \node[Box2,anchor=west,rotate=90](T2)at(MA2){};%Sample Efficiency
  \node[anchor=north](T222)at(MA2){I/O Bound};
 \draw[violet!50,line width=2pt](T2.north west)to[bend left=25]coordinate(AR22)(T2.south west);
 \draw[LineA](AR2)--(AR22);

 \path[LineA](AR3)--++(0:-1.0)coordinate(MA1);
 \node[Box2,anchor=east,align=right](T3)at(MA1){Compute Bound};
 \draw[violet!50,line width=2pt](T3.north east)to[bend left=25]coordinate(AR33)(T3.south east);
 \draw[LineA](AR3)--(AR33);
 \end{tikzpicture}}

Figure 2: The D·A·M Taxonomy as an Optimization Map: Machine learning performance is organized around Algorithms (Model), Data Selection, and Machine (Hardware). The triangle’s edges label the compute-bound, I/O-bound, and sample-efficiency relationships, while data selection reduces workload before model or hardware execution.

We can formalize this as the ICR, where $I$ denotes information content: \[\text{ICR} = \frac{\Delta I}{\Delta \text{FLOPs}}\]

A higher ICR means each FLOP of training buys more learning; pushing it up is the goal of every technique in this chapter. The numerator is not directly observable in production, so engineers estimate it through proxies: validation improvement per unit compute, area under the learning curve, loss reduction on held-out data, uncertainty or gradient-based sample scores, and coverage checks on deployment-relevant slices. Those proxies are imperfect, but they make the systems question measurable before the training run spends its full budget.

The ICR frontier: When data becomes a tax

Past the frontier, data becomes a tax: compute climbs, learning stalls.

The information-compute ratio is not constant; it follows a law of diminishing returns. The ICR frontier is the point where the marginal learning signal from additional data drops toward zero.

To illustrate diminishing returns, let $I(D)$ be the information content of a dataset of size $D$ and assume $I(D) \propto \log D$. This is a simplified analytical model rather than a universal law. If compute cost scales linearly with the per-sample operation count $O_{\text{sample}}$, then $C(D) = O_{\text{sample}} \cdot D$, and the resulting ICR follows equation 1: \[\text{ICR}(D) = \frac{\frac{d}{dD} I(D)}{\frac{d}{dD} C(D)} \approx \frac{1/D}{O_{\text{sample}}} = \frac{1}{O_{\text{sample}} \cdot D} \tag{1}\]

Under this illustrative model, the $1/(O_{\text{sample}} \cdot D)$ decay creates the data wall. Beyond a workload-dependent frontier, additional data may provide little learning while still adding compute. In this regime, data becomes a data tax that inflates the $O$ term of the iron law without a commensurate improvement in the accuracy numerator of the RoC (return on compute, see Return on compute (RoC) as an economic lens). The knee is a practical trade-off between target performance and marginal cost, not the mathematical maximum of ICR in this model.

Data selection turns the total operations $(O)$ term in the iron law from a fixed constant into a variable. A 2$\times$ improvement in measured ICR halves the FLOPs required to reach the same measured gain. It produces the same ideal time reduction as a 2$\times$ throughput increase only when the workload is compute-bound and the saved work lies on the critical path. ICR focuses specifically on compute; the cost-modeling framework in section 1.8 extends the same reasoning to acquisition, labeling, and storage costs.

A random batch of raw data can have low ICR when it contains redundant, noisy, or already-mastered examples. High-efficiency data pipelines (figure 3) can improve ICR through three stages: static pruning before training, dynamic selection during training, and synthetic generation on demand. To illustrate, consider computing ICR on a concrete coreset selection task: a deliberately selected subset intended to preserve the full dataset’s learning signal. Section 1.2.2 defines the EL2N and GraNd scoring methods used to build such subsets, and section 1.11 provides the complete measurement framework for evaluating these efficiency gains, including a compute-optimal frontier diagnostic that can suggest whether training is data-limited or compute-limited.

\begin{tikzpicture}[font=\small\sffamily, >=stealth]
\tikzset{
Box/.style={align=center, inner xsep=2pt,draw=GreenLine, line width=1pt,fill=none,
minimum width=24mm, minimum height=25mm,node distance=1.5},
LineA/.style={violet!50,line width=4.0pt,{-{Triangle[width=1.5*6pt,length=2.0*5pt]}},shorten <=1pt,shorten >=1pt},
ALine/.style={black!50, line width=1.1pt,{{Triangle[width=0.9*6pt,length=1.2*6pt]}-}},
Larrow/.style={fill=violet!50, single arrow,  inner sep=2pt, single arrow head extend=3pt,
            single arrow head indent=0pt,minimum height=10mm, minimum width=3pt}
}
\tikzset{%
 pics/inbox/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=INBOX,scale=\scalefac, every node/.append style={transform shape}]
\node[line width=\Linewidth,draw=\drawcolor,fill=\filllcolor!50,
rectangle,rounded corners=3pt,minimum width=14mm,minimum height=10mm]at(0,0.3){};
\node[line width=\Linewidth,draw=\drawcolor,fill=\filllcolor,
rectangle,rounded corners=3pt,minimum width=15mm,minimum height=10mm]at(0,0.1){};
\node[line width=\Linewidth,draw=\drawcolor,fill=\filllcolor!50,
,rectangle,rounded corners=3pt,minimum width=17mm,minimum height=10mm]
at(0,-0.1){};
\draw[line width=\Linewidth,draw=\drawcolor,fill=\filllcirclecolor,,rounded corners=2pt](-0.92,0.05)--
(-0.92,-0.78)--(0.92,-0.78)--(0.92,0.05)--(0.40,0.05)--(0.32,-0.2)--(-0.29,-0.2)--(-0.40,0.05)--cycle;

\node[single arrow, line width=\Linewidth,draw=black,fill=cyan!90!black!30, rotate=270,
      minimum width = 15pt, single arrow head extend=6pt,
      minimum height=10mm]at(0,0.5) {}; % length of arrow
 \end{scope}
     }
  }
}
%target
\tikzset{
pics/target/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
\definecolor{col1}{RGB}{62,100,125}
\definecolor{col2}{RGB}{219,253,166}
\colorlet{col1}{\filllcolor}
\colorlet{col2}{\filllcirclecolor}
\foreach\i/\col [count=\k]in {22mm/col1,17mm/col2,12mm/col1,7mm/col2,2.5mm/col1}{
\node[circle,inner sep=0pt,draw=\drawcolor,fill=\col,minimum size=\i,line width=\Linewidth](C\k){};
}
\draw[thick,fill=brown,xscale=-1](0,0)--++(111:0.13)--++(135:1)--++(225:0.1)--++(315:1)--cycle;
\path[green,xscale=-1](0,0)--(135:0.85)coordinate(XS1);
\draw[thick,fill=yellow,xscale=-1](XS1)--++(80:0.2)--++(135:0.37)--++(260:0.2)--++(190:0.2)--++(315:0.37)--cycle;
\end{scope}
    }
  }
}
%brain
\tikzset{pics/brain/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=BRAIN,scale=\scalefac, every node/.append style={transform shape}]
\fill[fill=\filllcolor!50](0.1,-0.5)to[out=0,in=180](0.33,-0.5)
to[out=0,in=270](0.45,-0.38)to(0.45,-0.18)
to[out=40,in=240](0.57,-0.13)to[out=110,in=310](0.52,-0.05)
to[out=130,in=290](0.44,0.15)to[out=90,in=340,distance=8](0.08,0.69)
to[out=160,in=80](-0.42,-0.15)to (-0.48,-0.7)to(0.07,-0.7)to(0.1,-0.5)
(-0.10,-0.42)to[out=310,in=180](0.1,-0.5);
\draw[draw=\drawcolor,line width=\Linewidth](0.1,-0.5)to[out=0,in=180](0.33,-0.5)
to[out=0,in=270](0.45,-0.38)to(0.45,-0.18)
to[out=40,in=240](0.57,-0.13)to[out=110,in=310](0.52,-0.05)
to[out=130,in=290](0.44,0.15)to[out=90,in=340,distance=8](0.08,0.69)
(-0.42,-0.15)to (-0.48,-0.7)
(0.07,-0.7)to(0.1,-0.5)
(-0.10,-0.42)to[out=310,in=180](0.1,-0.5);
\draw[fill=\filllcolor,line width=\Linewidth](-0.3,-0.10)to(0.08,0.60)
to[out=60,in=50,distance=3](-0.1,0.69)to[out=160,in=80](-0.26,0.59)to[out=170,in=90](-0.46,0.42)
to[out=170,in=110](-0.54,0.25)to[out=210,in=150](-0.54,0.04)
to[out=240,in=130](-0.52,-0.1)to[out=300,in=240]cycle;
\draw[fill=\filllcolor,line width=\Linewidth]
(-0.04,0.64)to[out=120,in=0](-0.1,0.69)(-0.19,0.52)to[out=120,in=330](-0.26,0.59)
(-0.4,0.33)to[out=150,in=280](-0.46,0.42)
%
(-0.44,-0.03)to[bend left=30](-0.34,-0.04)
(-0.33,0.08)to[bend left=40](-0.37,0.2) (-0.37,0.12)to[bend left=40](-0.45,0.14)
(-0.26,0.2)to[bend left=30](-0.24,0.13)
(-0.16,0.32)to[bend right=30](-0.27,0.3)to[bend right=30](-0.29,0.38)
(-0.13,0.49)to[bend left=30](-0.04,0.51);

\draw[rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcirclecolor,length=2.5pt]}](-0.23,0.03)--(-0.15,-0.03)--(-0.19,-0.18)--(-0.04,-0.28);
\draw[rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcirclecolor,length=2.5pt]}](-0.17,0.13)--(-0.04,0.05)--(-0.06,-0.06)--(0.14,-0.11);
\draw[rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcirclecolor,length=2.5pt]}](-0.12,0.23)--(0.31,0.0);
\draw[rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcirclecolor,length=2.5pt]}](-0.07,0.32)--(0.06,0.26)--(0.16,0.33)--(0.34,0.2);
\draw[rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcirclecolor,length=2.5pt]}](-0.01,0.43)--(0.06,0.39)--(0.18,0.51)--(0.31,0.4);
\end{scope}
     }
  }
}
%starS
\tikzset{
 mystar/.style={shape=star,star points=4,inner sep=1pt,
 minimum size=#1,star point ratio=2.1,rounded corners=#1/20},
pics/starS/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
\node[fill=\filllcolor,mystar={13mm}] at (0,0) {};
\node[fill=\filllcolor,mystar={8mm}] at (-0.75,0.6) {};
\node[fill=\filllcolor,mystar={4mm}] at (-0.2,1.0) {};
\node[fill=\filllcolor,mystar={5mm}] at (-0.8,-0.3) {};
\end{scope}
    }
  }
}
%funnel
\tikzset{%
 pics/funnel/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=FUNNEL,scale=\scalefac, every node/.append style={transform shape}]
\draw[fill=\filllcolor,line width=\Linewidth,draw=\drawcolor](-0.12,-0.81)--(-0.19,-0.25)--(-0.7,0.41)--(0.7,0.41)--(0.19,-0.25)--(0.12,-0.81)--cycle;
\draw[fill=\filllcolor,line width=\Linewidth,draw=\drawcolor](-0.19,-0.25)--(0.08,-0.25);
\draw[fill=\filllcolor,line width=\Linewidth,draw=\drawcolor](0.16,-0.09)--(0.41,0.31);
%
\node[line width=\Linewidth,draw=\drawcolor,fill=\filllcolor,inner sep=1pt,
rectangle,rounded corners=2pt,minimum width=16mm,minimum height=5pt]at(0,0.5){};
%
\foreach \i in{-0.5,0,0.5}{
\node[single arrow, line width=0.8*\Linewidth,draw=black,fill=\filllcirclecolor, rotate=270,inner sep=1pt,
      minimum width =9pt, single arrow head extend=2pt,
      minimum height=5mm]at(\i,0.9) {}; % length of arrow
   }
\node[single arrow,line width=0.8*\Linewidth,draw=black,fill=\filllcirclecolor, rotate=270,inner sep=1pt,
      minimum width =11pt, single arrow head extend=2pt,
      minimum height=5mm]at(0,-1.1) {}; % length of arrow
 \end{scope}
     }
  }
}
\pgfkeys{
  /channel/.cd,
   Depth/.store in=\Depth,
  Height/.store in=\Height,
  Width/.store in=\Width,
  filllcirclecolor/.store in=\filllcirclecolor,
  filllcolor/.store in=\filllcolor,
  drawcolor/.store in=\drawcolor,
  drawcircle/.store in=\drawcircle,
  scalefac/.store in=\scalefac,
  Linewidth/.store in=\Linewidth,
  picname/.store in=\picname,
  filllcolor=BrownLine,
  filllcirclecolor=violet!20,
  drawcolor=black,
  drawcircle=violet,
  scalefac=1,
  Linewidth=0.5pt,
  Depth=1.3,
  Height=0.8,
  Width=1.1,
  picname=C
}
%Raw Data
\node[Box](B1){};
\fill[green!07](B1.north west) rectangle ($(B1.north east)!0.6!(B1.south east)$)coordinate(B1DE);
\fill[green!20](B1.south east) rectangle ($(B1.north west)!0.6!(B1.south west)$)coordinate(B1LE);
\node[Box](){};
\tikzset{Text2/.style={font=\sffamily\bfseries\small,align=center}}
\node[Text2]at($(B1.south west)!0.5!(B1DE)$){Raw Data};
\coordinate(Q1)at($(B1.north west)!0.5!(B1DE)$);
\pic[shift={(0,0)}] at  (Q1){inbox={scalefac=0.7,picname=1,Linewidth=1.0pt,
 filllcolor=BrownL,drawcolor=black,filllcirclecolor=orange!70!yellow!80}};
%Static Pruning
\node[Box, right=of B1](B2){};
\fill[cyan!07](B2.north west) rectangle ($(B2.north east)!0.6!(B2.south east)$)coordinate(B2DE);
\fill[cyan!20](B2.south east) rectangle ($(B2.north west)!0.6!(B2.south west)$)coordinate(B2LE);
\node[Box, right=of B1,draw=BlueD](B2){};
\node[Text2]at($(B2.south west)!0.5!(B2DE)$){1. Static\\ Pruning};
\coordinate(Q2)at($(B2.north west)!0.5!(B2DE)$);
\pic[shift={(0,0)}] at  (Q2){funnel={scalefac=0.53,picname=1,Linewidth=0.5pt,
 filllcolor=green!70!orange!70,drawcolor=black,filllcirclecolor=red!70!blue!80}};
%%Dynamic Selection
\node[Box, right=of B2](B3){};
\fill[violet!07](B3.north west) rectangle ($(B3.north east)!0.6!(B3.south east)$)coordinate(B3DE);
\fill[violet!20](B3.south east) rectangle ($(B3.north west)!0.6!(B3.south west)$)coordinate(B3LE);
\node[Box, right=of B2,draw=violet](B3){};
\node[Text2]at($(B3.south west)!0.5!(B3DE)$){2. Dynamic\\ Selection};
\coordinate(Q3)at($(B3.north west)!0.5!(B3DE)$);
 \pic[shift={(0,0)}] at  (Q3){target={scalefac=0.55,picname=1,drawcolor=BlueD,
filllcolor=cyan!90!,Linewidth=0.7pt, filllcirclecolor=cyan!20}};
%Synthetic Gen
\node[Box, right=of B3](B4){};
\fill[orange!07](B4.north west) rectangle ($(B4.north east)!0.6!(B4.south east)$)coordinate(B4DE);
\fill[orange!20](B4.south east) rectangle ($(B4.north west)!0.6!(B4.south west)$)coordinate(B4LE);
\node[Box, right=of B3,draw=OrangeLine](B4){};
\node[Text2]at($(B4.south west)!0.5!(B4DE)$){3. Synthetic\\ Gen};
\coordinate(Q4)at($(B4.north west)!0.5!(B4DE)$);
\pic[shift={(0.15,-0.23)}] at  (Q4){starS={scalefac=0.7,picname=1,filllcolor=violet!90!}};
%Model
\node[Box, right=1.95 of B4](B5){};
\fill[black!05](B5.north west) rectangle ($(B5.north east)!0.6!(B5.south east)$)coordinate(B5DE);
\fill[black!15](B5.south east) rectangle ($(B5.north west)!0.6!(B5.south west)$)coordinate(B5LE);
\node[Box, right=1.95 of B4,draw=black](B5){};
\node[Text2]at($(B5.south west)!0.5!(B5DE)$){Model};
\coordinate(Q5)at($(B5.north west)!0.5!(B5DE)$);
\pic[shift={(0,0)}] at  (Q5){brain={scalefac=0.9,picname=1,filllcolor=orange!30!, Linewidth=0.95pt}};
%arrows
\foreach \i in{1,2,3}{
\pgfmathtruncatemacro{\X}{\i + 1} %
\draw[LineA](B\i)--(B\X);
}
\draw[LineA,font=\footnotesize\sffamily,text=black!70](B4)--node[above]{High ICR}(B5);
\node[below =2pt of B2,text=black!70]{Pretraining};
\node[below =2pt of B3,text=black!70]{During Training};
\node[below =2pt of B4,text=black!70]{On-Demand};
\end{tikzpicture}

Figure 3: The Data Selection Pipeline: A structured approach to increasing data value. Raw data is first pruned (Static Pruning), then selected during training (Dynamic Selection), and finally synthesized (Synthetic Generation). Each stage can increase the Information-Compute Ratio (ICR) when its quality gain justifies its compute cost.

Checkpoint 1.1: Data selection efficiency

The goal of data selection is to maximize the ICR.

Metric checks:

ICR application: given two training runs with identical accuracy gains but different compute budgets, can you determine which had higher ICR?
Data efficiency: under a fixed evaluation protocol, can you explain when a 50 percent smaller dataset with 2$\times$ higher ICR yields the same measured gain for half the training FLOPs?

Pipeline check:

The three stages: can you map static pruning, dynamic selection, and synthetic generation to the training lifecycle?

The practical question is how large the efficiency gap could become on a workload where dataset size, model cost, and selection strategy interact with concrete FLOP budgets. The following ImageNet-scale ResNet-50 scenario uses assumed accuracy gains to illustrate the calculation; it is not a reported EL2N result.

Napkin Math 1.1: Computing ICR: Coresets

Problem: Training the ResNet-50 lighthouse model from Lighthouse roster: Model biographies on ImageNet for one epoch costs 3.15 × 10¹⁶ FLOPs. Under the scenario’s assumed accuracy gains, how would retaining 50 percent of the data change the information-compute ratio? The calculation holds the per-sample work fixed and does not include selection overhead.

Setup:

Dataset: ImageNet (1.28M)
Model: ResNet-50 lighthouse (~8.2 GFLOP per forward pass, roughly 24.6 GFLOP for a forward plus backward training step, depending on implementation)
One epoch: 1.28M $\times$ 24.6 GFLOP = 3.15 × 10¹⁶ FLOPs
Assumed accuracy improvement per epoch (early training): 5 percentage points

Random selection (baseline):

Process all 1.28M samples uniformly
Accuracy gain: 5 percentage points
$\text{ICR}_{\text{random}}$ = 5 percentage points / (3.15 × 10¹⁶ FLOPs) = $1.6 \times 10^{-16}$ per FLOP

EL2N coreset (Error L2-Norm, a training-dynamics score developed in section 1.2.2; 50 percent of data):

Process 640.6K selected samples
Assumed accuracy gain: 4.5 percentage points (90 percent of the full-data gain)
Compute: 640.6K $\times$ 24.6 GFLOP = 1.6 × 10¹⁶ FLOPs
$\text{ICR}_{\text{coreset}}$ = 4.5 percentage points / (1.6 × 10¹⁶ FLOPs) = $2.9 \times 10^{-16}$ per FLOP

Systems insight: Under these assumptions, the selected subset achieves 1.8× higher ICR with a 0.5 percentage points smaller accuracy gain. A real decision requires measured gains, selection overhead, and coverage checks.

The three-stage optimization pipeline (static pruning, dynamic selection, and synthetic generation) provides concrete techniques for improving ICR. Static pruning, the first stage, removes examples before training begins; the safe reduction depends on the dataset, model, selection method, and target metric.

Self-Check: Question

In an ML infrastructure scaling scenario where available GPU compute grows by approximately $10\times$ every 3 years while high-quality web data grows by only $2\times$ every 5 years, what primary systems regime emerges, and what is the appropriate systems response?
1. A compute-rich, data-constrained Data Wall regime where intelligent data selection and curation must maximize the learning signal extracted per token
2. A memory bandwidth-bound regime where model parallel sharding must replace data parallelism across all training clusters
3. An I/O ingestion bottleneck where disk read bandwidth must be quadrupled to keep GPUs saturated
4. A compute-starved regime where synthetic data generation should be eliminated to avoid wasting accelerator cycles
Under the illustrative analytical model where dataset information content scales logarithmically as $I(D) \propto \log D$ and training compute scales linearly with per-sample operations $C(D) = O_{\text{sample}} \cdot D$, how does the marginal Information-Compute Ratio $\text{ICR}(D)$ scale with dataset size $D$?
1. It remains constant at $\mathcal{O}(1)$ because additional compute scales proportionally with dataset size
2. It decays as $\mathcal{O}(1 / (O_{\text{sample}} \cdot D))$, turning additional unselected data into a data tax that consumes compute with minimal learning progress
3. It grows logarithmically as $\mathcal{O}(\log D / O_{\text{sample}})$ due to power-law parameter scaling
4. It decays exponentially as $\mathcal{O}(\exp(-D))$ once the training corpus exceeds accelerator memory capacity
Explain why a dataset where $100\%$ of the sample labels are verifiably correct can still exhibit a very low Information-Compute Ratio (ICR).
True or False: Because deep learning models benefit from large-scale training, collecting and training on twice as much raw, deduplicated web data will always double the total information learned by the model.
Order the three primary stages of the high-efficiency data selection pipeline according to their execution in an ML system lifecycle: (1) Dynamic Selection, (2) Static Pruning, (3) Synthetic Data Generation.

See Answers →

Static Pruning

The first stage of the pipeline acts entirely before training begins, removing low-value samples so that fewer of them ever reach the model. Static pruning and pretraining filtration can reduce total computation without modifying the training loop or model architecture, but their effect on final quality must be measured.

The case for smaller datasets

Data-selection studies show that some datasets can be reduced without lowering measured accuracy. More data can improve performance, but large datasets may also contain substantial redundancy. Empirical studies on coreset selection and data pruning have demonstrated this effect on several standard benchmarks.

On CIFAR-10, Paul et al. (2021) report that EL2N pruning removed half of the training data without reducing accuracy. On ImageNet-1K, Sorscher et al. (2022) report that a self-supervised prototype metric discarded 20 percent of the data without sacrificing performance. The pattern extends to language modeling, where web-scraped corpora like The Pile³ and C4⁴ contain enough duplicate and templated content to make deduplication a systems issue. In datasets studied by Lee et al. (2022), approximate near-duplicate removal affected 3.04 percent of C4 and 13.63 percent of RealNews, and deduplicated training reduced memorization while preserving or improving perplexity.

Sorscher, Ben, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. 2022. “Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning.” Advances in Neural Information Processing Systems (NeurIPS) 35: 19523–36. https://doi.org/10.52202/068431-1419.

³ The Pile: An approximately 886 GB English text corpus (reported as 825 gibibytes by the source) aggregating twenty-two sub-datasets, including PubMed, ArXiv, GitHub, Project Gutenberg, Common Crawl, Stack Exchange, Wikipedia, and USPTO patents (Gao et al. 2020). Its multi-source design makes it a useful example of data diversity: each source family has a different duplication, quality, and domain-coverage profile, so selection pipelines must preserve coverage while removing redundant text.

Gao, Leo, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, et al. 2020. “The Pile: An 800GB Dataset of Diverse Text for Language Modeling.” arXiv Preprint arXiv:2101.00027.

⁴ C4 (Colossal Clean Crawled Corpus): C4 applies filtering to Common Crawl data, including language detection, deduplication of repeated three-sentence spans, and removal of pages that fail heuristic quality checks, producing approximately 750 GB of cleaned English text (Raffel et al. 2020). Filtering has its own compute cost and can remove useful material, so its ROI and quality effects must be measured rather than inferred from corpus size alone.

Raffel, C., N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. 2020. “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.” Journal of Machine Learning Research 21 (140): 1–67.

The reported gains are benchmark-specific. Pruning effectiveness depends on the dataset’s intrinsic redundancy, the selection algorithm, and the model architecture; validate the result on the specific task before deploying aggressive pruning in production.

Individual data points need not provide equal value for training. This heterogeneity follows in part from how classifiers learn decision boundaries. Many samples fall far from a boundary: a picture of a dog in good lighting may become easy once the model has learned its dominant features, while ambiguous and underrepresented cases can remain informative. Label quality also affects data requirements. In a simplified binary symmetric-noise model, a label-flip rate $\rho$ reduces an effective-sample-size factor to $(1 - 2\rho)^2$; at $\rho = 10\%$, matching the clean-data bound requires about $1/(1-2\rho)^2 \approx 1.56\times$ as many samples. The notebook below uses a separate toy learning-curve model to illustrate another data quality multiplier; neither multiplier is universal.

Napkin Math 1.2: The data quality multiplier

Problem: How much more noisy data does it take to reach the same target error as clean data?

Math: In this illustrative model, clean-data error scales as $\mathcal{O}(1/D)$, so $D_{\text{clean}} \propto 1/\epsilon$. Noisy-data error scales as $\mathcal{O}(1/\sqrt{D})$, so $D_{\text{noisy}} \propto 1/\epsilon^2$. For target error $\epsilon$ = 0.01 (1 percent):

$D_{\text{clean}}$ ≈ 100
$D_{\text{noisy}}$ ≈ 10,000

Result: Under these assumptions, noisy data requires 100× more samples at the target error.

Systems insight: Here, cleaning data acts as a 100× compute accelerator; the real multiplier is workload-dependent.

Coreset selection algorithms

The practical question then becomes how to identify which samples to keep. Coreset selection⁵ turns the static pruning decision into a coverage problem: keep the smallest subset that preserves the statistical properties of the entire dataset.

⁵ Coreset (core set): In computational geometry, a small subset can approximate a specified geometric objective within a controlled error factor (Agarwal et al. 2005). Machine learning adapts this idea as a selection principle, but a guarantee for an embedding and distance objective does not by itself guarantee neural-network accuracy. The retained subset must still be validated on the target task.

Agarwal, Pankaj K., Sariel Har-Peled, and Kasturi R. Varadarajan. 2005. “Geometric Approximation via Coresets.” In Combinatorial and Computational Geometry, edited by Jacob E. Goodman, János Pach, and Emo Welzl, vol. 52, 52. MSRI Publications. Cambridge University Press. https://doi.org/10.1017/9781009701259.002.

The systems decision is where to spend the selection budget: on cheap coverage metrics that preserve distributional structure, or on costlier training-dynamics scores that better target the decision boundary. The decision also needs a guardrail before any score is applied: classes, demographic groups, time windows, and rare failure modes that matter at deployment require minimum representation, because the highest-average-ICR subset can still remove the examples that define production risk.

Geometry-based methods select samples that cover a chosen representation without requiring target-model training. The $k$-Center algorithm,⁶ a facility-location-style objective, selects samples that reduce the maximum distance from any point to its nearest selected center under the chosen embedding and metric.

⁶ k-center algorithm: Its greedy strategy iteratively picks the point farthest from the existing centers under a chosen embedding and metric. Sener and Savarese use this core-set framing for convolutional-neural-network active learning (Sener and Savarese 2018). The resulting coverage bound applies to that geometric objective, not directly to downstream accuracy or rare-class preservation.

Sener, Ozan, and Silvio Savarese. 2018. “Active Learning for Convolutional Neural Networks: A Core-Set Approach.” International Conference on Learning Representations (ICLR).

Welling, Max. 2009. “Herding Dynamical Weights to Learn.” Proceedings of the 26th Annual International Conference on Machine Learning, 1121–28. https://doi.org/10.1145/1553374.1553517.

Herding takes a different approach. Welling’s original procedure iteratively constructs representative pseudo-samples whose feature statistics approximate observed moments (Welling 2009). Related moment-matching objectives can guide subset selection from a fixed pool. These methods are computationally attractive because they operate on feature representations, but their scores are not inherently label-aware.

Training-dynamics methods use a proxy model to identify examples that may be important. GraNd (Gradient Normed) and EL2N (error L2-norm)⁷ score samples by gradient magnitude or prediction error early in training (Paul et al. 2021). High scores identify examples the proxy finds difficult, but they can also identify noise or unrepresentative samples and therefore need coverage and quality checks. The same work reports useful score transfer across several architectures, which can enable less expensive proxy-based selection. Forgetting Events⁸ tracks how often a sample is correctly classified and later misclassified during training (Toneva et al. 2019).

⁷ EL2N (error L2-norm) and GraNd (gradient normed): These methods score samples based on their error or gradient norm early in training, identifying samples the model finds most difficult. The practicality of this approach relies on transferability, where scores from a small proxy model can guide data selection for a much larger target model. For instance, a proxy trained for five epochs can generate scores to curate a dataset for a full 90-epoch production training run (Paul et al. 2021).

Paul, Mansheej, Surya Ganguli, and Gintare Karolina Dziugaite. 2021. “Deep Learning on a Data Diet: Finding Important Examples Early in Training.” Advances in Neural Information Processing Systems (NeurIPS) 34: 20596–607.

⁸ Forgetting events: This method identifies valuable examples by tracking when the model “forgets” them—transitioning from a correct to an incorrect classification during training. The central trade-off is the high cost of this analysis, which requires a full training run. However, the resulting importance scores transfer reliably from small proxy models to large target models (for example, ResNet-18 to ResNet-50), which is precisely what makes the “inexpensive proxy-based selection” strategy viable (Toneva et al. 2019).

Toneva, Mariya, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. 2019. “An Empirical Study of Example Forgetting During Deep Neural Network Learning.” International Conference on Learning Representations (ICLR).

Geometry-based and training-dynamics methods optimize different proxies, so their quality ranking depends on the dataset, representation, target model, and scoring budget. The comparison should be read as a selection-budget table: a score is useful only if its target-quality gains justify its scoring cost.

Each algorithm in table 3 occupies a different point in the ICR framework’s compute-versus-information trade-off. Figure 4 illustrates one uncertainty-based strategy rather than every coreset method. Random sampling appears on the left, while the right panel concentrates the selection budget near the illustrated decision boundary. Geometry-based methods may instead prioritize coverage across the embedding space.

Table 3: Coreset Selection Algorithm Comparison: $D =$ dataset size, $K =$ coreset size. Geometry-based and training-dynamics methods optimize different proxies, so their quality ranking depends on the dataset, representation, target model, and scoring budget.

Method	Compute Cost	Requires Training	Best For	Limitation
k-Center	$\mathcal{O}(D^2)$ or $\mathcal{O}(DK)$	No	Coverage, exploration	Ignores label information
Herding	$\mathcal{O}(DK)$	No	Distribution matching	Assumes Gaussian-like
GraNd	$\mathcal{O}(\text{epochs} \times D)$	Yes (few epochs)	Decision boundaries	Requires proxy training
Forgetting	$\mathcal{O}(\text{full training})$	Yes (full)	Hard examples	Expensive to compute
EL2N	$\mathcal{O}(\text{epochs} \times D)$	Yes (few epochs)	Uncertainty sampling	Best with proxy model

\scalebox{0.9}{%
\begin{tikzpicture}[font=\small\sffamily,x=9mm,y=8mm]
% Left plot: Random Sampling
\begin{scope}
    \node[font=\small\sffamily] at (2.15, 5.0) {Random Sampling};

    % Decision boundary
    \draw[thick, dashed, gray] (0, 0) -- (5, 5);

    % Class A points (below line) - circles
    \foreach \x/\y in {0.5/0.2, 1.0/0.5, 0.8/1.2, 1.5/0.8, 2.0/1.0,
                       2.5/1.5, 1.2/0.3, 0.3/0.8, 1.8/1.5, 2.2/0.5,
                       3.0/2.0, 3.5/2.5, 2.8/1.8, 3.2/1.2, 4.0/2.8} {
        \fill[blue!60] (\x, \y) circle (2pt);
    }

    % Class B points (above line) - triangles
    \foreach \x/\y in {0.5/1.5, 1.0/2.0, 0.3/2.5, 1.5/2.5, 2.0/3.0,
                       2.5/3.5, 1.2/3.2, 0.8/3.8, 1.8/3.5, 2.2/4.0,
                       3.0/4.0, 3.5/4.5, 2.8/3.8, 3.2/4.2, 4.0/4.5} {
        \fill[red!60] (\x, \y) circle (2pt);
    }

    % Randomly selected (circled) - some easy, some hard
    \foreach \x/\y in {0.5/0.2, 1.5/2.5, 3.0/2.0, 0.8/3.8, 2.2/0.5} {
        \draw[thick, orange] (\x, \y) circle (5pt);
    }

    % Axis
    \draw[->,>=Latex,thick] (0, 0) -- (5.2, 0) node[right, font=\footnotesize\sffamily] {$x_1$};
    \draw[->,>=Latex,thick] (0, 0) -- (0, 5.2) node[above, font=\footnotesize\sffamily] {$x_2$};

    % Label
    \node[font=\footnotesize\sffamily, orange] at (2.5, -0.5) {Selected (random)};
\end{scope}

% Right plot: Coreset Selection
\begin{scope}[xshift=7cm]
    \node[font=\small\sffamily] at (2.15, 5.0) {Coreset Selection};

    % Decision boundary
    \draw[thick, dashed, gray] (0, 0) -- (5, 5);

    % Uncertainty band near boundary
    \fill[yellow!20] (0, 0) -- (0, 1) -- (4, 5) -- (5, 5) -- (5, 4) -- (1, 0) -- cycle;
        \node[font=\fontsize{7pt}{8}\sffamily, fill=white, inner sep=1pt] at (3.5, 3.25) {High uncertainty};

    % Class A points (below line) - circles
    \foreach \x/\y in {0.5/0.2, 1.0/0.5, 0.8/1.2, 1.5/0.8, 2.0/1.0,
                       2.5/1.5, 1.2/0.3, 0.3/0.8, 1.8/1.5, 2.2/0.5,
                       3.0/2.0, 3.5/2.5, 2.8/1.8, 3.2/1.2, 4.0/2.8} {
        \fill[blue!60] (\x, \y) circle (2pt);
    }

    % Class B points (above line) - triangles
    \foreach \x/\y in {0.5/1.5, 1.0/2.0, 0.3/2.5, 1.5/2.5, 2.0/3.0,
                       2.5/3.5, 1.2/3.2, 0.8/3.8, 1.8/3.5, 2.2/4.0,
                       3.0/4.0, 3.5/4.5, 2.8/3.8, 3.2/4.2, 4.0/4.5} {
        \fill[red!60] (\x, \y) circle (2pt);
    }

    % Coreset selected (near boundary) - circled
    \foreach \x/\y in {0.8/1.2, 2.5/1.5, 3.0/2.0, 1.0/2.0, 2.0/3.0} {
        \draw[thick, green!60!black] (\x, \y) circle (5pt);
    }

    % Axis
    \draw[->,>=Latex,thick] (0, 0) -- (5.2, 0) node[right, font=\footnotesize\sffamily] {$x_1$};
    \draw[->,>=Latex,thick] (0, 0) -- (0, 5.2) node[above, font=\footnotesize\sffamily] {$x_2$};

    % Label
    \node[font=\footnotesize\sffamily, green!60!black] at (2.5, -0.5) {Selected (boundary)};
\end{scope}
\end{tikzpicture}}

Figure 4: Illustrative Uncertainty-Based Selection: Both panels select five samples, isolating allocation rather than budget. Random sampling distributes those choices across the feature space; the illustrative uncertainty strategy concentrates them in the shaded band around the current decision boundary.

EL2N with a small proxy model offers one practical balance between scoring cost and selection quality. The approach trains a lightweight model for a few epochs, computes EL2N scores, and selects a subset subject to coverage and quality checks. The proxy must provide rankings that transfer to the target model, and the investment pays off only when downstream training savings exceed selection overhead. A concrete scenario illustrates this workflow without asserting that its 10 percent subset preserves accuracy.

Example 1.1: Coreset selection in practice

Scenario: An engineering team aims to select a 100,000 coreset (10 percent) from 1 million training images for faster model iteration.

Diagnosis: Naive random subsampling drops rare classes and boundary examples. Training a lightweight proxy model for 5 epochs calculates EL2N error scores, retaining high-uncertainty samples near the decision boundary.

Systems lesson: Proxy-based coreset selection replaces some full-data passes with targeted scoring compute. It accelerates full-model training only when the proxy ranking transfers and coverage checks preserve rare classes and edge cases.

Listing 1 demonstrates how to compute EL2N scores and select a coreset using a lightweight proxy model. The mechanism spends a small amount of probe compute to identify high-uncertainty samples near the decision boundary, then trains the full model on the retained subset rather than on redundant easy examples. The example requires an unshuffled dataloader so each score position remains aligned with its dataset index.

Proxy scoring turns uncertainty into reusable selected indices: compute_el2n_scores measures which samples still confuse a briefly trained model, and select_coreset retains those high-information examples for the full training run.

Listing 1: EL2N-Based Coreset Selection: The compute_el2n_scores function trains a proxy model for a few epochs, then measures prediction error via L2 distance from one-hot labels. High scores identify samples the proxy finds difficult, including potentially noisy examples. The select_coreset function illustrates ranking by this score; production use also requires coverage, noise, and target-quality validation.

def compute_el2n_scores(model, dataloader, num_epochs=5):
    """Compute EL2N scores.

    Returns L2 norm of (prediction - one_hot_label).
    """
    # Train proxy model for a few epochs to get meaningful predictions
    train_proxy(model, dataloader, num_epochs)

    scores = []
    model.eval()
    for x, y in dataloader:
        logits = model(x)
        probs = softmax(logits, dim=1)
        # One-hot encode labels
        one_hot = zeros_like(probs).scatter_(1, y.unsqueeze(1), 1)
        # EL2N score = L2 distance from confident prediction
        el2n = (probs - one_hot).norm(dim=1)  # High = uncertain
        scores.extend(el2n.tolist())
    return scores


def select_coreset(scores, dataset, fraction=0.1):
    """Select top-k highest-scoring (most uncertain) samples."""
    k = int(len(dataset) * fraction)
    # Sort by score descending (highest uncertainty first)
    indices = argsort(scores, descending=True)[:k]
    return Subset(dataset, indices)


# Illustrative 10% subset; validate accuracy and coverage before use
scores = compute_el2n_scores(proxy_model, full_loader)
coreset = select_coreset(scores, full_dataset, fraction=0.1)
train_full_model(model, coreset)

Data deduplication

While coreset selection identifies which samples to keep based on their informativeness, a complementary approach targets duplicate samples that add compute without adding learning signal. Deduplication can provide immediate efficiency gains, especially for exact and near-duplicates, and requires no model training. This makes it one of the most accessible optimizations in data selection, but near-duplicate thresholds must be validated so the pipeline does not remove useful distributional signal.

The simplest form of deduplication (introduced as a data engineering pipeline stage in Systematic Data Processing, and here elevated to an optimization lever) uses hash-based methods for exact matches. By computing a cryptographic hash (MD5 or SHA-256) for each sample and removing those with identical hashes, practitioners can eliminate byte-for-byte duplicates that inevitably accumulate in large web-scraped corpora. This process is computationally cheap, scaling linearly with dataset size, and can be parallelized trivially.

Near-duplicate detection addresses content that differs at the byte level while retaining substantial token- or character-shingle overlap. For text, MinHash⁹ with Locality-Sensitive Hashing¹⁰ (LSH) approximates Jaccard similarity¹¹ efficiently. It can detect lightly edited content with high overlap, but it does not establish semantic paraphrase equivalence.

⁹ MinHash: Invented by Broder (1997) to detect duplicate web pages for AltaVista, the algorithm creates compact signatures using random hash functions such that similar documents produce similar signatures with high probability. Each signature compresses a document to a fixed-size sketch, enabling pairwise similarity estimation in $\mathcal{O}(s)$ time for sketch size $s$.

Broder, A. Z. 1997. “On the Resemblance and Containment of Documents.” Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No.97TB100171), 21–29. https://doi.org/10.1109/sequen.1997.666900.

¹⁰ Locality-sensitive hashing (LSH): LSH hashes MinHash signatures into buckets so that similar signatures are more likely to collide. Candidate generation avoids an exhaustive all-pairs comparison, while runtime and memory depend on signature width, banding, bucket sizes, and the number of candidates that require verification.

¹¹ Jaccard similarity: Defined as $|A \cap B| / |A \cup B|$, ranging from 0 (disjoint) to 1 (identical). For deduplication, the metric compares sets of shingles across documents of different lengths. The threshold must be calibrated for the shingle definition, corpus, and downstream quality because no universal cutoff separates duplicates from legitimately related documents.

¹² CLIP (Contrastive Language-Image Pretraining): Pretrained on 400 million image-text pairs, CLIP maps semantically related images and text into a shared embedding space (Radford et al. 2021). Embeddings can retrieve candidates for review, but semantic proximity is not proof that two samples are duplicates. Generating embeddings also costs substantially more than computing perceptual hashes, with the ratio depending on model and hardware.

Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, et al. 2021. “Learning Transferable Visual Models from Natural Language Supervision.” International Conference on Machine Learning (ICML), 8748–63.

For images, perceptual hashing produces signatures robust to minor transformations like resizing and compression, identifying visually similar candidates stored in different formats. Embedding-based similarity can retrieve semantic near-neighbor candidates by computing dense representations (CLIP¹² (Radford et al. 2021) for images, sentence transformers for text), though proximity does not itself prove duplication and the approach incurs higher computational overhead.

Deduplication can reduce repeated-content memorization and avoid spending FLOPs on exact or near duplicates. It can also alter the corpus’s intentional empirical weighting, so thresholds and downstream quality must be validated rather than assuming that every duplicate is harmful.

Deduplication also changes when retained examples index a stateful representation rather than only a training corpus.

Lighthouse 1.1: DLRM and embedding deduplication

Our DLRM lighthouse model from Lighthouse roster: Model biographies is memory capacity-bound, with embedding tables consuming terabytes of storage for billions of user/item IDs. Much of this capacity is wasted on cold embeddings, IDs that appear rarely in training data.

Data selection for DLRM can include interaction deduplication and embedding pruning for cold IDs. Removing repeated interactions reduces training examples but shrinks an embedding table only when IDs disappear or the allocation, hashing, or sharing policy changes.

Data pruning by quality

Deduplication removes redundant samples, but a third category of problematic data remains: samples that actively harm learning. Quality-based pruning eliminates samples that either contribute no meaningful signal or introduce contradictory information that confuses the optimization process.

Label error detection can be a high-impact form of quality pruning. Tools like Cleanlab identify samples where the assigned label may be incorrect based on model confidence patterns across training. A sample that the model consistently predicts as class A but is labeled class B may be a hard boundary case, a distribution shift, a model blind spot, or an annotation mistake. The score should trigger review rather than automatic deletion; confirmed errors can then be corrected or removed.

Outlier removal addresses a different pathology: samples far from any cluster center in feature space. Such points can indicate noise, annotation errors, or corruption, but they can also be the rare edge cases that define deployment risk. The key is distinguishing informative outliers from invalid samples, with conservative thresholds and slice-level review before removal.

Low-information filtering applies domain-specific heuristics to remove samples that lack sufficient signal for learning. For text corpora, this often means removing high-perplexity garbled text (perplexity is a language model’s measure of how surprising a text is, so high values flag incoherent strings) and, in some pipelines, low-perplexity boilerplate or repetitive text. For image datasets, filtering targets blurry, corrupted, or near-uniform samples that provide little visual information.

Together, these three static pruning techniques—coreset selection, deduplication, and quality filtering—can reduce work before training. With epoch count, batch size, and per-sample work fixed, a 50 percent dataset reduction means 50 percent fewer forward passes, backward passes, and gradient updates. Training schedules that add compensating epochs or steps reduce that saving.

Static pruning answers a question about what to keep, but it treats the answer as fixed. Once the pruned dataset is set, every epoch trains on the same subset. A sample’s usefulness, however, can change as the model learns: examples that challenge an undertrained model may become easy after sufficient gradient updates. Dynamic selection techniques address this limitation by adapting the training data at each stage based on what the model has already mastered.

Self-Check: Question

A team wants to select a coreset of size $K$ from a dataset of size $D$ before training a large production model. They need a method that accounts for model uncertainty near decision boundaries but cannot afford a full target-model training run for scoring. Which method and systems trade-off best fits their requirement?
1. $k$-Center clustering on raw pixel inputs, because it guarantees zero-cost label-aware boundary identification
2. Forgetting Events scoring on the full production model, because it requires no proxy architecture and computes in $\mathcal{O}(1)$ time
3. EL2N (Error L2-Norm) scoring using an inexpensive proxy model trained for a few epochs, leveraging proxy score transferability
4. Herding on Gaussian-distributed features, because it completely avoids computing feature representations
Why is Locality-Sensitive Hashing (LSH) with MinHash preferred over exhaustive pairwise Jaccard similarity comparison for large-scale text deduplication?
1. MinHash LSH guarantees $100\%$ precision in detecting semantic paraphrases across different natural languages
2. MinHash eliminates the need to tokenize or shingle input documents before hashing
3. Exhaustive pairwise Jaccard comparison requires training a deep neural network, whereas LSH is purely rule-based
4. Exhaustive pairwise comparison requires $\mathcal{O}(D^2)$ document comparisons, whereas MinHash LSH hashes compact signatures into sublinear candidate collision buckets
In recommendation systems like DLRM where embedding tables consume terabytes of memory, how does interaction deduplication differ in systems impact from cold embedding pruning?
True or False: Removing near-duplicate documents with MinHash LSH always improves model accuracy because duplicate data has zero statistical value in all training regimes.
The training-dynamics coreset metric that measures the Euclidean norm of the difference between predicted class probabilities and the one-hot target vector early in training is known as ____.

See Answers →

Dynamic Selection

Early in training, a model benefits from broad, easy coverage that builds stable feature representations. Later, those same examples produce little new gradient signal, while harder samples near the decision boundary become more valuable for refinement. Dynamic selection exploits this changing information-compute ratio by adapting the data diet to the model’s current state.

Curriculum learning: Easy to hard

The first dynamic selection technique, curriculum learning¹³ (Bengio et al. 2009; Soviany et al. 2022), structures the order in which data is presented to the model. Instead of random shuffling, it starts with simpler examples and gradually introduces more complex ones, mirroring how humans learn by mastering basics before advancing to harder material.

¹³ Curriculum learning: From Latin currere (“to run”), originally meaning “the course to be run”—a metaphor that maps directly to the technique: training data as a course run in deliberate order, easy stretches first. Curriculum learning can act as a continuation method for nonconvex optimization, with easier examples shaping the early optimization trajectory before harder examples are introduced. From a systems perspective, the ICR of a sample can vary during training, which motivates changing the mix of examples over time.

Bengio, Yoshua, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. “Curriculum Learning.” Proceedings of the 26th Annual International Conference on Machine Learning, 41–48. https://doi.org/10.1145/1553374.1553380.

Soviany, Petru, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. 2022. “Curriculum Learning: A Survey.” International Journal of Computer Vision 130 (6): 1526–65. https://doi.org/10.1007/s11263-022-01611-x.

Curriculum learning can change the optimization trajectory by controlling which gradients dominate at each stage. Easy examples may provide stable early signals, while examples scored as hard can contain useful boundary information, label noise, or outliers. An easy-to-hard schedule can smooth training on some workloads, but its benefit depends on the difficulty score, pacing rule, model, and data; table 5 therefore treats the savings as illustrative rather than guaranteed.

Implementing a curriculum requires two components: a difficulty scorer that ranks samples, and a pacing function that controls how quickly hard samples are introduced. A common choice is linear pacing: \[ \text{samples}_{n_{\text{epoch}}} = \texttt{sort\_by\_difficulty}[:D \cdot \min(1, n_{\text{epoch}}/N_{\text{warmup}})] \] where $n_{\text{epoch}}$ is the current epoch, $D$ is the full dataset size, and $N_{\text{warmup}}$ is the number of warmup epochs before the full dataset becomes available. Early epochs train on the easiest $D \cdot (n_{\text{epoch}}/N_{\text{warmup}})$ fraction; after warmup, training proceeds on the full dataset.

The difficulty scorer is a systems choice because it trades probe-compute overhead against ordering quality, as table 4 shows. Loss and confidence scoring buy better ordering with extra inference, heuristics avoid compute but require domain knowledge, and self-paced scoring moves adaptation into the training loop itself.

Table 4: Difficulty Scoring Strategies for Curriculum Learning: The rows compare each scorer’s signal source with the system prerequisite or trade-off it introduces.

Strategy	Difficulty Score	Best For
Loss-Based	Loss from probe model (low = easy)	General-purpose; requires probe training
Confidence-Based	Teacher model confidence (high = easy)	When teacher available; distillation setups
Domain Heuristics	Sentence length, image complexity	No extra compute; domain knowledge required
Self-Paced	Current model’s loss (updated each epoch)	Adaptive; no probe needed

Table 5 uses hypothetical epoch counts to illustrate how curriculum savings would be computed across workloads. The values are not measurements from the cited curriculum-learning studies.

Table 5: Illustrative Curriculum Epoch Reductions: The table applies one target-accuracy rule to hypothetical baseline and curriculum epoch counts. It demonstrates the arithmetic but does not establish a ranking among datasets or methods.

Dataset	Model	Pacing Strategy	Epochs to Target Acc.	Epoch Reduction
CIFAR-10	ResNet-18	Linear warmup	115 vs. 150 baseline	23.3% fewer epochs
CIFAR-100	ResNet-32	Self-paced	180 vs. 220 baseline	18.2% fewer epochs
ImageNet	ResNet-50	Loss-based	80 vs. 90 baseline	11.1% fewer epochs
ImageNet	ResNet-50	Learned (noisy labels)	70 vs. 90 baseline	22.2% fewer epochs

The scenario illustrates that useful curriculum gains must be measured at a fixed target metric. The ordering is task-dependent: anti-curriculum presents hard examples first, while self-paced learning adjusts difficulty using the model’s current loss. Neither ordering is universally superior.

Active learning: Human-in-the-loop

Curriculum learning optimizes the order in which samples are presented but assumes all samples are already labeled. This assumption breaks down in specialized domains where labeling requires substantial expertise, time, or money. Rather than labeling everything upfront, active learning¹⁴ (Settles 2012; Ren et al. 2021) shifts the optimization target from choosing which labeled samples to train on to choosing which unlabeled samples are worth labeling.

¹⁴ Active learning: The “active” component is the learning algorithm itself selecting which unlabeled samples a human expert should label next. This reframes the problem from a computational optimization (training on given data) to a financial one: maximizing model improvement per dollar spent on expert labeling. Querying the most informative examples can substantially reduce labeling needs compared with random sampling, but the multiplier is task-, model-, and oracle-dependent.

Active learning transforms data acquisition from static collection into an iterative query process. Trace the circular flow path in figure 5, following how model uncertainty metrics select high-value unlabeled samples for expert annotation.

The effectiveness of active learning depends critically on the query strategy used to select samples for annotation (Settles 2009, 2012; Ren et al. 2021). The simplest approach, uncertainty sampling, selects samples where the model is least confident, such as predictions near 0.5 probability for binary classification. It is inexpensive to score with one model but can overconcentrate queries near one ambiguous region. Query-by-committee extends this idea by training multiple models and selecting samples where they disagree most, capturing epistemic uncertainty that a single model might miss.

Settles, Burr. 2009. Active Learning Literature Survey. Computer Sciences Technical Report 1648. University of Wisconsin-Madison.

Settles, Burr. 2012. Active Learning. Synthesis Lectures on Artificial Intelligence and Machine Learning. Springer Cham. https://doi.org/10.1007/978-3-031-01560-1.

Ren, Pengzhen, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B. Gupta, Xiaojiang Chen, and Xin Wang. 2021. “A Survey of Deep Active Learning.” ACM Computing Surveys 54 (9): 1–40. https://doi.org/10.1145/3472291.

For practitioners willing to invest more compute, expected model change selects samples that would cause the largest gradient update if labeled. This approach provides a theoretically grounded but expensive alternative. Diversity sampling complements uncertainty-based methods by selecting samples dissimilar from currently labeled data, improving coverage rather than clustering every query around the same ambiguous region.

Active learning is particularly valuable in domains where labeling requires expertise. In medical imaging, for instance, an AI system diagnosing diseases from X-rays may be confident on common conditions but uncertain about rarer cases. By focusing human annotation on these ambiguous cases, active learning optimizes the use of expensive expert time while accelerating model improvement.

\scalebox{0.9}{\begin{tikzpicture}[font=\small\sffamily]
\tikzset{
Box2/.style={align=flush center, inner sep=2pt,draw=none,fill=black!70,
font=\fontsize{8pt}{8}\sffamily\bfseries,text=white,minimum width=20mm, minimum height=5mm },
Box/.style={align=center, inner xsep=2pt,draw=black!70, line width=1pt,node distance=12mm,
fill=none, minimum width=25mm, minimum height=20mm},
Circle1/.style={circle,  minimum size=33mm, draw=none, fill=BrownLine!20},
LineD/.style={BrownLine!60!black!20,line width=4.0pt,dashed,dash pattern=on 5pt off 2pt,
{-{Triangle[width=1.5*6pt,length=2.0*5pt]}},shorten <=5pt,shorten >=1pt},
LineA/.style={BrownLine!80!black!40,line width=4.0pt,
{-{Triangle[width=1.5*6pt,length=2.0*5pt]}},shorten <=5pt,shorten >=1pt},
ALineA/.style={violet!60,{Circle[line width=1.1pt,fill=white,round,length=5pt,width=5pt]}-,
line width=1.5pt,shorten <=-3pt,shorten >=-6pt}
}

%dataS
\tikzset{%
 pics/dataS/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=FUNNEL,scale=\scalefac, every node/.append style={transform shape}]
%plats
\draw[fill=\filllcolor,line width=\Linewidth,draw=\drawcolor](0,-0.3)--(-0.67,0.03)--(0,0.37)--(0.67,0.03)--cycle;
\draw[fill=\filllcirclecolor,line width=\Linewidth,draw=\drawcolor](0,0)--(-0.67,0.33)--(0,0.67)--(0.67,0.33)--cycle;
\draw[fill=\filllcolor,line width=\Linewidth,draw=\drawcolor](0,0.3)--(-0.67,0.63)--(0,0.97)--(0.67,0.63)--cycle;
%left
\draw[line width=\Linewidth,draw=\drawcolor](-0.39,1.21)--++(210:0.55)--++(270:1.22)--(-0.39,-0.56);
\fill[line width=\Linewidth,fill=\filllcirclecolor!60!violet!,draw=green,draw=\drawcolor](-0.39,1.21)circle(3pt);
\fill[line width=\Linewidth,fill=\filllcolor,draw=green,draw=\drawcolor](-0.39,-0.56)circle(3pt);
%right
\draw[line width=\Linewidth,draw=\drawcolor](0.39,1.21)--++(330:0.55)--++(270:1.22)--(0.39,-0.56);
\fill[line width=\Linewidth,fill=\filllcirclecolor!60!violet!,draw=green,draw=\drawcolor](0.39,1.21)circle(3pt);
\fill[line width=\Linewidth,fill=\filllcolor,draw=green,draw=\drawcolor](0.39,-0.56)circle(3pt);
\end{scope}
     }
  }
}
%graph style
\tikzset{
pics/graph/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=GRAPH,line join=round,scale=1, every node/.append style={transform shape}]
\def\dx{\Width}
\def\dy{\Height}
\def\dz{\Depth}
%
\def\x{0}
\def\y{0.15}
\def\z{0}
% colors
\draw[draw=\filllcirclecolor,line width=1pt](-0.2,0)--(1.3,0);
\draw[draw=\filllcirclecolor,line width=1pt](-0.2,0)--(-0.2,1.2);
\filldraw[fill=\filllcolor!10, draw=\drawcolor] (\x,\y+\dy,\z) -- (\x,\y+\dy,\z+\dz) -- (\x+\dx,\y+\dy,\z+\dz) -- (\x+\dx,\y+\dy,\z) -- cycle; % gornja strana
\filldraw[fill=\filllcolor!50, draw=\drawcolor] (\x+\dx,\y,\z) -- (\x+\dx,\y,\z+\dz) -- (\x+\dx,\y+\dy,\z+\dz) -- (\x+\dx,\y+\dy,\z) -- cycle; % desna strana
\filldraw[fill=\filllcolor!60, draw=\drawcolor] (\x,\y,\z+\dz) -- (\x+\dx,\y,\z+\dz) -- (\x+\dx,\y+\dy,\z+\dz) -- (\x,\y+\dy,\z+\dz) -- cycle; % prednja strana
\end{scope}
    }
  }
}
 %person style
 \tikzset{
 pics/man/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=PERSON,scale=\scalefac, every node/.append style={transform shape}]
     % tie
    \draw[draw=\tiecolor,fill=\tiecolor] (0.0,-1.1)--(0.16,-0.87)--(0.09,-0.46)--(0.13,-0.37)--(0.0,-0.28)
                   --(-0.13,-0.37)--(-0.09,-0.46)--(-0.16,-0.87)--cycle;
    % ears
    \draw[fill=black] (0.74,0.95) to[out=20,in=80](0.86,0.80) to[out=250,in=330](0.65,0.65) to[out=70,in=260] cycle;
    \draw[fill=black] (-0.76,0.96) to[out=170,in=110](-0.85,0.80) to[out=290,in=190](-0.65,0.65) to[out=110,in=290] cycle;

    % head
    \draw[fill=black] (0,0) to[out=180,in=290](-0.72,0.84) to[out=110,in=190](-0.56,1.67)
                      to[out=70,in=110](0.68,1.58) to[out=320,in=80](0.72,0.84) to[out=250,in=0] cycle;
    % face
    \draw[fill=white] (0,0.11) to[out=175,in=290](-0.53,0.65) to[out=110,in=265](-0.61,1.22)
                      to[out=80,in=235](-0.50,1.45) to[out=340,in=215](0.50,1.47)
                      to[out=310,in=85](0.60,0.92) to[out=260,in=2] cycle;
    \draw[fill=black] (-0.50,1.45) to[out=315,in=195](0.40,1.25) to[out=340,in=10](0.37,1.32)
                      to[out=190,in=310](-0.40,1.49) -- cycle;
    % neck
    \draw[line width=1.0pt] (-0.62,-0.2) to[out=50,in=290] (-0.5,0.42);
    \draw[line width=1.0pt] (0.62,-0.2) to[out=130,in=250] (0.5,0.42);
    % body
    \draw[draw=\bodycolor,fill=\bodycolor,line width=\Linewidth] (0.0,-1.0) to[out=150,in=290](-0.48,-0.14) to[out=200,in=50](-1.28,-0.44)
                   to[out=240,in=80](-1.55,-2.06) -- (1.55,-2.06)
                   to[out=100,in=300](1.28,-0.44) to[out=130,in=340](0.49,-0.14)
                   to[out=245,in=30] cycle;
    % right stet
    \draw[line width=3pt,\stetcolor] (0.8,-0.32) to[bend left=7](0.78,-0.64)
         to[out=350,in=80](0.98,-1.35) to[out=250,in=330](0.72,-1.60);
    \draw[line width=3pt,\stetcolor] (0.43,-1.53) to[out=180,in=240](0.3,-1.15)
         to[out=60,in=170](0.78,-0.64);
    % left stet
    \draw[line width=3pt,\stetcolor] (-0.75,-0.31) to[bend right=20](-0.65,-1.45);
    \node[fill=\stetcolor,circle,minimum size=5pt] at (-0.65,-1.45) {};
    % eyes
    \node[circle,fill=black,inner sep=2pt] at (0.28,0.94) {};
    \node[circle,fill=black,inner sep=2pt] at (-0.28,0.94) {};
     % mouth
    \draw[line width=1.1pt] (-0.25,0.5) to[bend right=40](0.25,0.5);
 \end{scope}
     }
  }
}
%target
\tikzset{
pics/target/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
\definecolor{col1}{RGB}{62,100,125}
\definecolor{col2}{RGB}{219,253,166}
\colorlet{col1}{\filllcolor}
\colorlet{col2}{\filllcirclecolor}
\foreach\i/\col [count=\k]in {22mm/col1,17mm/col2,12mm/col1,7mm/col2,2.5mm/col1}{
\node[circle,inner sep=0pt,draw=\drawcolor,fill=\col,minimum size=\i,line width=\Linewidth](C\k){};
}
\draw[thick,fill=brown,xscale=-1](0,0)--++(111:0.13)--++(135:1)--++(225:0.1)--++(315:1)--cycle;
\path[green,xscale=-1](0,0)--(135:0.85)coordinate(XS1);
\draw[thick,fill=yellow,xscale=-1](XS1)--++(80:0.2)--++(135:0.37)--++(260:0.2)--++(190:0.2)--++(315:0.37)--cycle;
\end{scope}
    }
  }
}
%AI style
\tikzset{
pics/llm/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
\node[circle,minimum size=12mm,draw=\drawcolor, fill=\filllcolor!70,line width=1.25*\Linewidth](C\picname) at (0,0){};
\def\startangle{90}
\def\radius{1.15}
\def\radiusI{1.1}
\foreach \i [evaluate=\i as \j using \i+1] [count =\k] in {0,2,4,6,8} {
\pgfmathsetmacro{\angle}{\startangle - \i * (360/8)}
\draw[draw=black,-{Circle[black ,fill=\filllcirclecolor,length=5.5pt,line width=0.5*\Linewidth]},line width=1.5*\Linewidth](C\picname)--++(\startangle - \i*45:\radius) ;
\node[circle,draw=black,fill=\filllcirclecolor!80!red!50,inner sep=3pt,line width=0.5*\Linewidth](2C\k)at(\startangle - \j*45:\radiusI) {};
}
\draw[line width=1.5*\Linewidth](2C1)--++(-0.5,0)|-(2C2);
\draw[line width=1.5*\Linewidth](2C3)--++(0.5,0)|-(2C4);
\node[circle,,minimum size=12mm,draw=\drawcolor, fill=\filllcolor!70,line width=0.5*\Linewidth]at (0,0){};
\end{scope}
    }
  }
}
%brain
\tikzset{pics/brain/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=BRAIN,scale=\scalefac, every node/.append style={transform shape}]
\draw[fill=\filllcolor,line width=\Linewidth](-0.3,-0.10)to(0.08,0.60)
to[out=60,in=50,distance=3](-0.1,0.69)to[out=160,in=80](-0.26,0.59)to[out=170,in=90](-0.46,0.42)
to[out=170,in=110](-0.54,0.25)to[out=210,in=150](-0.54,0.04)
to[out=240,in=130](-0.52,-0.1)to[out=300,in=240]cycle;
\draw[fill=\filllcolor,line width=\Linewidth]
(-0.04,0.64)to[out=120,in=0](-0.1,0.69)(-0.19,0.52)to[out=120,in=330](-0.26,0.59)
(-0.4,0.33)to[out=150,in=280](-0.46,0.42)
%
(-0.44,-0.03)to[bend left=30](-0.34,-0.04)
(-0.33,0.08)to[bend left=40](-0.37,0.2) (-0.37,0.12)to[bend left=40](-0.45,0.14)
(-0.26,0.2)to[bend left=30](-0.24,0.13)
(-0.16,0.32)to[bend right=30](-0.27,0.3)to[bend right=30](-0.29,0.38)
(-0.13,0.49)to[bend left=30](-0.04,0.51);
\draw[thick,rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcolor,length=2.5pt]}](-0.23,0.03)--(-0.15,-0.03)--(-0.19,-0.18)--(-0.04,-0.28);
\draw[thick,rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcolor,length=2.5pt]}](-0.17,0.13)--(-0.04,0.05)--(-0.06,-0.06)--(0.14,-0.11);
\draw[thick,rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcolor,length=2.5pt]}](-0.12,0.23)--(0.31,0.0);
\draw[thick,rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcolor,length=2.5pt]}](-0.07,0.32)--(0.06,0.26)--(0.16,0.33)--(0.34,0.2);
\draw[thick,rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcolor,length=2.5pt]}](-0.01,0.43)--(0.06,0.39)--(0.18,0.51)--(0.31,0.4);
\end{scope}
     }
  }
}
\pgfkeys{
  /channel/.cd,
   Depth/.store in=\Depth,
  Height/.store in=\Height,
  Width/.store in=\Width,
  filllcirclecolor/.store in=\filllcirclecolor,
  filllcolor/.store in=\filllcolor,
  drawcolor/.store in=\drawcolor,
  drawcircle/.store in=\drawcircle,
  scalefac/.store in=\scalefac,
  Linewidth/.store in=\Linewidth,
  picname/.store in=\picname,
  tiecolor/.store in=\tiecolor,
  bodycolor/.store in=\bodycolor,
  stetcolor/.store in=\stetcolor,
  tiecolor=red,      % default tie color
  bodycolor=blue!30,  % default body color
  stetcolor=green,  % default stet color
  filllcolor=BrownLine,
  filllcirclecolor=violet!20,
  drawcolor=black,
  drawcircle=violet,
  scalefac=1,
  Linewidth=0.5pt,
  Depth=0.2,
  Height=0.5,
  Width=0.25,
  picname=C
}
%Unlabeled Pool
\node[Box,dashed](B1){};
\pic[shift={(0,-0.28)}] at  (B1){dataS={scalefac=0.9,picname=1,Linewidth=1.0pt,
 filllcolor=cyan!90!black!40!,drawcolor=black,filllcirclecolor=orange}};
 \draw[ALineA,draw=black!70](B1.south west)--++(200:0.45)
 node[below=1pt,Box2]{Unlabeled Pool};
%Selection Strategy
\node[Box,right=3 of B1,draw=BlueLine](B2){};
 \pic[shift={(0,0)}] at  (B2){target={scalefac=0.7,picname=1,
 drawcolor=BlueD,filllcolor=green!90!black,Linewidth=0.7pt, filllcirclecolor=green!20}};
\draw[ALineA,draw=BlueLine](B2.south west)--++(200:0.45)
 node[below=1pt,Box2,fill=BlueLine]{Selection Strategy};
%Oracle (Human)
\node[Box,below=of B2,draw=OrangeLine](B3){};
 %persons 1
\begin{scope}[local bounding box=PERSON1,shift={($(B3)+(-0.5,-0.17)$)},scale=1, every node/.append style={transform shape}]
\pic at (1.0,0) {man={scalefac=0.32,tiecolor=green, bodycolor=RedLine,stetcolor=RedLine, Linewidth=0.5pt}};
\pic at (0.25,0.16) {man={scalefac=0.4,tiecolor=orange, bodycolor=BlueLine,stetcolor=BlueLine, Linewidth=0.5pt}};
\end{scope}
\draw[ALineA,draw=OrangeLine](B3.south west)--++(200:0.45)
 node[below=1pt,Box2,fill=OrangeLine]{Oracle (Human)};
%Training Set
\node[Box,below=of B3,draw=GreenD](B4){};
\begin{scope}[local bounding box=GRAPH1,shift={($(B4)+(-0.67,-0.75)$)},
scale=1.3, every node/.append style={transform shape}]
\pic[shift={(0,0)}] at  (0,0){graph={filllcirclecolor=black!60,scalefac=0.5,picname=1,drawcolor=black,filllcolor=red,Height=0.5,Linewidth=1.25pt}};
\pic[shift={(0.33,0)}] at  (0,0){graph={filllcirclecolor=none,scalefac=0.5,picname=2,drawcolor=black,filllcolor=violet,Height=1,Linewidth=1.25pt}};
\pic[shift={(0.66,0)}] at  (0,0){graph={filllcirclecolor=none,scalefac=0.5,picname=3,drawcolor=black,filllcolor=green!60!black,Height=0.25,Linewidth=1.25pt}};
\pic[shift={(0.99,0)}] at  (0,0){graph={filllcirclecolor=none,scalefac=0.5,picname=4,drawcolor=black,filllcolor=cyan,Height=0.75,Linewidth=1.25pt}};
\end{scope}
\draw[ALineA,draw=GreenD](B4.south west)--++(200:0.45)
 node[below=1pt,Box2,fill=GreenD]{Training Set};
%Model
\node[Box,left=3 of B4,draw=RedLine](B5){};
%AI
\pic[shift={(0,0)}] at  (B5){llm={scalefac=0.8,picname=1,drawcolor=GreenD,filllcolor=GreenD!20!, Linewidth=1pt,filllcirclecolor=red}};
%brain
\pic[shift={(0.12,-0.18)}] at  (C1){brain={scalefac=0.81,picname=2,filllcolor=orange!30!, filllcirclecolor=cyan!55!black!60, Linewidth=0.75pt}};
\draw[ALineA,draw=RedLine](B5.south west)--++(200:0.45)
 node[below=1pt,Box2,fill=RedLine]{Model};
%arrows
\draw[LineA,BrownLine!80!black!70](B1)--node[above,font=\footnotesize\sffamily,text=black!70]{Query}(B2);
\draw[LineA](B2)--node[right,font=\footnotesize\sffamily,text=black!70]{Uncertainty}(B3);
\draw[LineA](B3)--node[right,font=\footnotesize\sffamily,text=black!70]{Labels}(B4);
\draw[LineA](B4)--node[above,font=\footnotesize\sffamily,text=black!70]{Train}(B5);
\draw[LineD](B5)--node[left,font=\footnotesize\sffamily,text=black!70]{Update}(B1);
\end{tikzpicture}}

Figure 5: The Active Learning Closed Loop: The selection cycle trades proxy-scoring compute for queries chosen from an unlabeled pool for oracle annotation. Uncertainty, diversity, and coverage criteria seek more useful information per annotation than undirected sampling.

The economic implications can be substantial in specialist domains, where annotation cost and turnaround may dominate the compute used to score candidates. These query strategies drive each iteration of the active learning loop in figure 5, and a simple budget comparison shows how active learning can reduce labeling cost and training work under stated assumptions.

Napkin Math 1.3: The active learning ROI

Problem: A hospital team has 1 Million unlabeled scans, a $5/label specialist-label cost, a $500,000 budget, and 1 month. How does uncertainty sampling change labeling cost and per-epoch training work relative to the budget-matched random baseline?

Scenario A: Naive Labeling

Cost: Labeling all 1 Million scans would cost $5,000,000 (10× over budget).
Budget-matched baseline: $\$500,000 \div \$5/\text{label} = 100,000$ random scans.
Naive labeling outcome: Random selection does not guarantee rare-pathology coverage.

Scenario B: Active Learning

Strategy: Uncertainty sampling selects 50,000 scans for specialist labeling.
Cost: $50,000 \times \$5/\text{label} = \$250,000$ (50 percent under budget).
Training work: $100,000 \div 50,000 = 2×$ fewer examples per epoch than the budget-matched random baseline.
Active learning outcome: Matching 100,000 random scans requires empirical validation.

Systems insight: Against the budget-matched random baseline, selection saves $\$500,000 - \$250,000 = \$250,000$; accuracy still requires empirical validation.

Targeted sample selection can reduce the annotation volume required to reach a target metric. Compare the two illustrative accuracy trajectories in figure 6, noting the horizontal gap between active uncertainty sampling and baseline random selection as sample counts scale.

Figure 6: **The Active Learning Multiplier**: Model accuracy is plotted against labeled samples on a logarithmic axis. In this illustrative scenario, active learning (green solid) reaches the same accuracy with fewer labels than random sampling (gray dashed); the arrow marks an approximately 4$\times$ label-efficiency gap near 90 percent accuracy.

The figure tracks a different axis from the calculation in napkin math 1.3. It illustrates label efficiency, while the callout calculates labeling dollars under separate assumptions. Active learning also incurs scoring, acquisition, and repeated-retraining costs, so end-to-end savings can be smaller than the label-count gap.

Active learning can do more than save labeling cost: a suitable query rule can concentrate review on uncertain, diverse, or failure-relevant examples.

Example 1.2: Hard negative mining in a smart doorbell

Scenario: A smart doorbell vision detector receives continuous video feeds dominated by empty backgrounds (easy negatives) and normal pedestrian poses.

Diagnosis: Random frame sampling can miss rare false positives such as statues, laundry piles, and human-shaped shadows. Active learning routes uncertain predictions (“Person: 51%”) and sampled production errors to human reviewers for targeted labeling.

Systems lesson: Hard-negative mining can improve labeling efficiency by spending review effort on suspected false positives and other difficult negatives. The query stream still needs diversity and coverage checks so confident but systematically wrong cases are not excluded.

Semi-supervised learning: Using unlabeled data

Consider an illustrative medical-imaging scenario with 50,000 chest X-rays, of which 500 have been reviewed and labeled¹⁵ by radiologists, a labeling rate of 1 percent. Whether that seed set is adequate depends on the task and distribution. Semi-supervised learning can use the remaining 49,500 images when the unlabeled pool matches the target distribution and the method is validated on labeled data.

¹⁵ Illustrative clinical labeling economics: The review-rate and hourly-rate ranges are scenario assumptions rather than reported measurements. Under those inputs, labeling 500 scans requires 7–10 hours and costs $1,000–3,000; labeling all 50,000 costs $94,000–300,000. The arithmetic illustrates why semi-supervised learning can be attractive when expert labels are expensive, but actual review time, adjudication, prevalence, and validation requirements must be measured.

Active learning optimizes which samples to label but still requires human annotation for every selected example. Semi-supervised learning takes a more aggressive approach by extracting learning signal from unlabeled data. It uses a labeled subset to guide learning on a larger unlabeled pool and can approach fully supervised accuracy with substantially fewer labels, although the result depends on the task, distributions, and method.

The core insight behind semi-supervised learning is that unlabeled data, while it cannot directly teach the mapping from inputs to outputs, contains structural information about the input distribution $p(x)$ that can constrain the hypothesis space. Under smoothness and cluster assumptions, a decision boundary that cuts through a dense region assigns different labels to nearby inputs and is less plausible than one in a low-density region. Semi-supervised methods exploit these assumptions, which must be checked against the target task.

Three main techniques implement this insight. Pseudo-labeling¹⁶ takes the most direct approach: train on labeled data, use the model to generate “pseudo-labels” for high-confidence unlabeled predictions, then retrain on both. The confidence threshold is critical: setting it too low introduces label noise that degrades learning, while setting it too high wastes potentially useful data.

¹⁶ Pseudo-labeling: Uses a trained model’s own confident predictions as ground-truth labels for unlabeled data; the technique’s effectiveness depends on a virtuous cycle: accurate predictions on easy unlabeled examples expand the training set, improving the model, which enables accurate predictions on harder examples. The failure mode is equally self-reinforcing: incorrect pseudo-labels reinforce errors through confirmation bias, making the confidence threshold a critical systems parameter. Setting it too low compounds label noise across training iterations; setting it too high wastes unlabeled data that could contribute learning signal.

¹⁷ Consistency regularization: Rooted in the smoothness assumption: if two inputs $x_1$ and $x_2$ are close in input space, their labels should also be close; the training objective minimizes divergence between a model’s predictions on an input and its augmented version. This is conceptually distinct from data augmentation (which creates more training examples) because it explicitly enforces prediction consistency as a loss term, even for unlabeled data where the “correct” label is unknown. The systems consequence is additional computation for generating and comparing predictions under multiple perturbations, a trade-off that can favor compute-rich, label-poor settings.

¹⁸ FixMatch: It generates a pseudo-label using a weakly augmented image and then trains the model to predict that same label for a strongly augmented version of the image. This consistency training is gated by a confidence threshold; a pseudo-label is only used if the model’s prediction on the weak augmentation is highly confident (for example, >0.95) (Sohn et al. 2020). This embodies a direct systems trade-off between additional training compute and fewer manual labels; the magnitude depends on the baseline and cost model.

Sohn, Kihyuk, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. 2020. “FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence.” Advances in Neural Information Processing Systems (NeurIPS) 33: 596–608.

Consistency regularization¹⁷ takes a different angle by enforcing that the model produces similar predictions for augmented versions of the same input. A robust classifier should be invariant to realistic perturbations like cropping, rotation, or color shifts. Methods like FixMatch¹⁸ combine both approaches, assigning pseudo-labels only to samples where the unaugmented prediction is confident but training the model to predict these labels on strongly augmented versions of the same images.

Label propagation offers a third paradigm through graph-based reasoning: construct a similarity graph over all samples and propagate labels from labeled nodes to their neighbors. This approach works particularly well when the feature space exhibits clear cluster structure.

Example 1.3: FixMatch on CIFAR-10

Scenario: A semi-supervised image classification pipeline trains on CIFAR-10 with only 250 labeled samples (25 per class) alongside unlabeled data (Sohn et al. 2020).

Diagnosis: The fully labeled CIFAR-10 training set contains 50K labels. FixMatch generates high-confidence pseudo-labels (>0.95 threshold) on weakly augmented unlabeled images and enforces consistency on strongly augmented variants, reaching 94.9 percent accuracy in the cited experiment.

Systems lesson: Semi-supervised learning trades additional training compute for fewer manual labels. Under the chapter’s labeling-cost assumptions, the FixMatch scenario reduces dataset preparation expense by 8.1×; the end-to-end saving depends on the unlabeled-data and compute costs.

The systems trade-off in semi-supervised learning exchanges labeling effort for additional computation over labeled and unlabeled samples. It can approach fully supervised accuracy with fewer labels, but neither the label reduction nor the compute increase is universal. A CIFAR-10 comparison makes one such trade-off concrete (table 6).

The efficiency gains are substantial, but semi-supervised learning is not universally applicable. The technique assumes that unlabeled data comes from the same distribution as labeled data, and it struggles when unlabeled data contains out-of-distribution samples (the model confidently mislabels them), when class imbalance is severe (pseudo-labels amplify majority class bias), or when the labeled set does not cover all classes (preventing label propagation for unseen classes). Always validate on a held-out set with true labels to catch distribution mismatch.

Table 6: FixMatch Label Efficiency on CIFAR-10: With 250 labels (0.5 percent of the dataset), FixMatch reaches 94.9 percent accuracy while using 200× fewer labels than the full training set contains.

Label Budget	Method	Accuracy	Label Efficiency
50,000 (100%)	Full training set	—	Label-count reference
4,000 (8%)	FixMatch	95.7%	12.5× more efficient
250 (0.5%)	FixMatch	94.9%	200× more efficient
40 (0.08%)	FixMatch	88.6%	1250× more efficient

Despite these limitations, semi-supervised learning can reduce label requirements while maintaining accuracy on suitable tasks. Across the techniques, coreset selection and deduplication prune low-value samples before training; curriculum learning changes the presentation order; active learning chooses which samples receive human annotation; and semi-supervised learning extracts additional signal from an unlabeled pool. Each technique can reduce dependence on task-specific labels, but none eliminates the need for labeled evaluation. The progression raises a related possibility: task-specific labels may not be necessary for pretraining. The structure of data itself—that cat images resemble other cat images and coherent sentences follow grammatical patterns—may provide a supervision signal, while downstream quality still requires task-aligned validation.

Self-Check: Question

In curriculum learning, an engineer implements an ‘easy-to-hard’ pacing schedule that controls the fraction of the sorted training pool available to the model at training step $t$. What is the primary systems and statistical objective of this pacing strategy?
1. To guide optimization through stable early gradient trajectories using low-variance samples before exposing the model to high-variance boundary cases
2. To eliminate the need for backward passes during the first half of training
3. To maximize GPU memory bandwidth utilization by sorting tensors strictly by length in bytes
4. To replace human labelers with an automated oracle during the late stages of training
An active learning pipeline chooses unlabeled examples for costly radiologist annotation. The team notices that simple uncertainty sampling repeatedly selects images from a single ambiguous artifact class, starving other disease categories. Which query strategy should they adopt to resolve this pathology?
1. Least-confidence sampling, because it strictly selects the lowest top-1 probability prediction
2. Diversity sampling or hybrid uncertainty-diversity sampling (such as BADGE), which balances uncertainty near decision boundaries with feature-space coverage
3. Random undersampling of the entire unlabeled pool to reduce dataset size before scoring
4. Uniform zero-shot pseudo-labeling without confidence thresholds
Explain the mechanism of confirmation bias in semi-supervised pseudo-labeling, and specify how confidence thresholding mitigates it.
True or False: Consistency regularization methods such as FixMatch rely on the smoothness assumption, asserting that realistic perturbations of an input sample should not change the model’s predicted class distribution.
Order the steps in an active learning closed-loop iteration: (1) Select top query samples via query strategy, (2) Acquire expert annotations from the oracle, (3) Score unlabeled pool using current model, (4) Retrain or update model on expanded dataset, (5) Add newly labeled samples to the training set.

See Answers →

Self-Supervised Learning

GPT was trained to predict the next token in a sequence. BERT was trained to fill in masked tokens. Neither pretraining objective required task-specific human labels. self-supervised learning¹⁹ generalizes this insight: by designing pretext tasks that derive supervision from the data’s structure, models can learn reusable representations from unlabeled data at scale. Where active and semi-supervised learning reduce the demand for labels, SSL changes the accounting by making unlabeled structure the pretraining signal. In the three-stage map, this makes SSL less a fourth selection stage than an extension of the available pool: the question shifts from which labeled samples to train on to how unlabeled samples can support pretraining. It responds to one aspect of the data wall in section 1.1.4 by broadening what counts as a training signal, while leaving corpus quality, licensing, coverage, and labeled evaluation requirements intact.

¹⁹ Self-supervised learning: The pretext task, such as next-token or masked-token prediction, provides a supervisory signal derived from the data itself. This reduces dependence on task-specific human labels but can require substantial pretraining compute. Amortization depends on how broadly the resulting model is reused.

Labels are one form of supervision; a pretext task can also derive a learning signal from the structure of the data, as table 7 summarizes.

Table 7: Self-Supervised Pretext Tasks by Modality: Each task extracts supervision from data structure rather than human labels, enabling pretraining on large-scale unlabeled corpora.

Modality	Self-Supervised Task	Supervision Signal
Text	Masked language modeling	Predict [MASK] from context
Text	Next-token prediction	Predict next token in sequence
Images	Contrastive learning	Same image (augmented) vs. different images
Images	Masked autoencoding	Reconstruct masked patches
Multi-modal	CLIP-style alignment	Match image-text pairs

Pretext tasks generate supervision signals automatically. Masked-token objectives can encourage models to learn grammatical and semantic structure; next-token prediction teaches continuation structure; and contrastive image objectives encourage visual features that are invariant to selected transformations. These approaches build on the convolutional neural network and transformer families in Network Architectures, but the systems accounting matters: self-supervision removes task-specific annotation from the pretraining gate rather than removing data cost. Pretraining can begin on unlabeled corpora without waiting for example-level labels, but collection, filtering, licensing, storage, and processing remain. Separating pretraining from downstream labeling restructures the economics of machine learning when the representation transfers across tasks; task-specific adaptation and evaluation still carry their own data requirements.

The economics of amortization

Understanding why self-supervised learning became central to many foundation-model workflows requires examining its economic structure. A foundation model is a broadly pretrained reusable base model that can be adapted to many downstream tasks through fine-tuning. That shift translates into concrete cost savings through cost amortization, where expensive pretraining is performed once and reused across many applications (table 8).

Table 8: Illustrative Foundation-Model Amortization: The label and compute ranges are scenario assumptions. They illustrate why reuse can lower marginal task cost when representations transfer, not a universal fine-tuning ratio.

Approach	Labels per Task	Compute per Task	Data Acquisition
Train from scratch	100K–1M labeled	100% full training	Task-specific collection
Fine-tune foundation model	100–1K labeled	1–5% of full training	Reuse pretraining corpus

To illustrate this economic transformation, consider a company building 10 specialized classifiers. Under the scenario assumptions, each task trained from scratch needs 100,000 labels at $1 per label, for $1M across all tasks. The modeled compute burden is 10,000 GPU-hours.

The fine-tuning scenario pays 10,000 GPU-hours once, then assigns 1,000 labels and 50 GPU-hours of compute to each task. Across all 10 tasks, the modeled fine-tuning labels cost $10K.

Under these inputs, labeling cost drops by 100× and per-task marginal compute by 20×. These are consequences of the chosen scenario values, while total compute becomes favorable only after enough downstream tasks amortize pretraining.

The cost structure explains why fine-tuning can be economical when a pretrained model transfers across many downstream applications. The result depends on pretraining cost, reuse count, and per-task adaptation requirements.

Investing in foundation model pretraining fundamentally alters downstream task economics. Compare the side-by-side cost bar charts in figure 7, contrasting repetitive scratch training expenses on the left against amortized pretraining with low-cost fine-tuning on the right.

Figure 7: **Cost Amortization in Foundation Models**: Training from scratch (left) requires 1,000 GPU-hours per task (10,000 GPU-hours total for 10 tasks). The foundation model approach (right) pays 10,000 GPU-hours upfront for pretraining but reduces each subsequent task to just 50 GPU-hours. At 10 tasks the totals are comparable (10,000 GPU-hours vs. 10,500 GPU-hours), but the per-task marginal cost drops by 20×, and the crossover favoring the foundation model occurs around 10.5 tasks. The plotted 10-task snapshot therefore remains just below that break-even point.

Foundation model paradigm

The amortization economics can favor self-supervised learning, though different SSL methods occupy different points on the cost-efficiency frontier. The original SimCLR experiments (Chen et al. 2020) benefited from batches as large as 4,096, while MoCo (He et al. 2020) uses a queue-based dictionary and momentum encoder to decouple the number of negatives from the mini-batch. Masked modeling methods have different memory and step-count trade-offs. Generative pretraining follows empirical power-law scaling with data, parameters, and compute, making it attractive when its cost is amortized across downstream tasks. These methods rely on architecture families introduced in Network Architectures; what matters for data selection is that self-supervised pretraining can increase the value of limited labeled data when the learned representation transfers.

Chen, Ting, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. “A Simple Framework for Contrastive Learning of Visual Representations.” Proceedings of the 37th International Conference on Machine Learning, ICML’20, vol. 119: 1597–607.

He, Kaiming, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. “Momentum Contrast for Unsupervised Visual Representation Learning.” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 9726–35. https://doi.org/10.1109/cvpr42600.2020.00975.

²⁰ Foundation model: The name emphasizes that these models serve as a shared base for many downstream tasks, but this creates a single point of failure. Defects in the foundation model’s pretraining data (biases, factual errors, memorized private content) propagate to every application built upon it. From a systems perspective, this homogenization risk means that data selection quality during pretraining has an outsized blast radius: a curation error that would affect one task in the train-from-scratch paradigm now affects thousands of downstream deployments.

Bommasani, Rishi, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, et al. 2021. “On the Opportunities and Risks of Foundation Models.” arXiv Preprint arXiv:2108.07258.

When its pretraining cost is reused across tasks, SSL supports the foundation model paradigm²⁰ (Bommasani et al. 2021). The data selection principles discussed throughout this chapter (coreset selection, curriculum learning, active learning) remain relevant within this paradigm. Pretraining corpus curation applies the same deduplication and quality filtering techniques at web scale, and fine-tuning data selection determines which labeled examples maximize downstream task performance.

One pretraining corpus defect propagates into many downstream tasks.

Self-supervised learning addresses the label bottleneck by learning from data structure rather than human annotation, yet it cannot solve data scarcity itself. Rare classes may have too few examples, edge cases may never appear in the wild, and privacy constraints may prevent collecting real samples. The third stage of the data selection pipeline addresses this gap by creating new data on demand rather than selecting or curating existing data.

Self-Check: Question

In the economics of foundation models, an organization invests $C_{\text{pretrain}} = 10{,}000$ GPU-hours in self-supervised pretraining. Each downstream task fine-tuning costs $C_{\text{finetune}} = 50$ GPU-hours. Training each task from scratch would cost $C_{\text{scratch}} = 1{,}000$ GPU-hours. What is the minimum number of downstream tasks $N^*$ required to break even on the pretraining investment?
1. $N^* = 5$ downstream tasks
2. $N^* = 8$ downstream tasks
3. $N^* = 11$ downstream tasks ($10{,}000 + 11 \times 50 = 10{,}550 < 11 \times 1{,}000 = 11{,}000$)
4. $N^* = 50$ downstream tasks
How does the MoCo (Momentum Contrast) framework reduce the hardware and memory constraints of contrastive self-supervised learning compared to naive SimCLR?
1. By replacing convolutional backbones with rule-based lookup tables to avoid backpropagation
2. By requiring fully supervised class labels to filter out false negative pairs
3. By using a dynamic queue of negative keys and a slowly updating momentum encoder, decoupling negative dictionary size from mini-batch size
4. By computing exact pairwise Jaccard similarities on raw byte sequences rather than latent embeddings
What is the systems-level ‘homogenization risk’ (or blast radius) of pretraining a shared foundation model on an uncurated dataset?
True or False: Because self-supervised pretraining learns general representations from unlabeled data, it completely eliminates the need for data selection or quality filtering during the pretraining stage.
An unsupervised learning approach where the model solves an auxiliary task constructed directly from the structure of unlabeled data (such as masked token prediction or contrastive instance discrimination) is called a ____ task.

See Answers →

Synthetic Data Generation

A robot fleet may need rare collision examples, a medical model may need cases a hospital cannot share, and a wake-word model may need thousands of noisy kitchens before the product exists. Static pruning removed redundancy before training. Dynamic selection focused compute on the most informative samples during training, and self-supervised pretraining drove its label requirement to zero, reframing what counts as training data rather than adding a stage. The third and final stage of the data selection pipeline, synthetic data generation, takes the opposite approach: rather than subtracting or selecting from existing data, it creates new high-value samples when real data is scarce, expensive, or lacks diversity. The strategy shifts from curation to creation.

Data augmentation: Transformation-based synthesis

Data augmentation is the lowest-cost form of synthetic generation: it expands a dataset by applying transformations to existing samples. Because many transformations preserve label semantics while creating novel inputs, augmentation effectively multiplies the diversity of a training set without requiring additional data collection.

For image data, augmentation should match the invariance the deployed model must learn (Shorten and Khoshgoftaar 2019). Geometric transformations such as rotation, flipping, cropping, and scaling introduce spatial variation that makes models robust to viewpoint changes. Photometric transformations adjust brightness, contrast, saturation, and hue to simulate different lighting conditions and camera characteristics. More advanced techniques like Cutout²¹ (which applies random rectangular masks), MixUp (Zhang et al. 2018) (which blends two images and their labels), and CutMix²² (which pastes patches between images) push augmentation further by creating entirely synthetic training examples that regularize learning.

Shorten, Connor, and Taghi M. Khoshgoftaar. 2019. “A Survey on Image Data Augmentation for Deep Learning.” Journal of Big Data 6 (1): 1–48. https://doi.org/10.1186/s40537-019-0197-0.

²¹ Cutout: Randomly masks square regions of input images during training, forcing the model to recognize objects from partial information rather than relying on any single discriminative region (DeVries and Taylor 2017). Unlike dropout (which zeroes neurons in feature space), Cutout operates in input space. The original Cutout experiments produced 0.3–2.0 percentage-point gains across CIFAR-10/100 and SVHN with negligible compute overhead, making it a high-ICR augmentation technique when occlusion-style invariance matches the task: more information per sample at near-zero additional cost to the pipeline.

DeVries, Terrance, and Graham W. Taylor. 2017. “Improved Regularization of Convolutional Neural Networks with Cutout.” arXiv Preprint arXiv:1708.04552.

Zhang, Hongyi, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2018. mixup: Beyond Empirical Risk Minimization. OpenReview.net.

²² CutMix: Replaces Cutout’s zeroed-out region with a patch from a different training image, mixing labels proportionally to patch area (30 percent of image A replaced by image B yields a 70/30 label split) (Yun et al. 2019). This addresses Cutout’s weakness: zeroed regions waste pixel information that could carry learning signal. The CutMix paper reports ImageNet top-1 improvements of 2.28 percentage points for ResNet-50 and 1.70 percentage points for ResNet-101, while also improving localization behavior. The method provides stronger regularization than either Cutout or MixUp alone with little additional data-pipeline cost.

Yun, Sangdoo, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. 2019. “CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features.” 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 6023–32. https://doi.org/10.1109/ICCV.2019.00612.

²³ Back-translation: In the method described by Sennrich et al. (2016), target-language monolingual text is translated into the source language, and the synthetic source is paired with the unchanged target for machine-translation training. The systems trade-off is latency because each synthetic source requires translation-model inference. Pipelines can precompute these examples offline rather than generating them in the data loader.

Sennrich, Rico, Barry Haddow, and Alexandra Birch. 2016. “Improving Neural Machine Translation Models with Monolingual Data.” Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 86–96. https://doi.org/10.18653/v1/P16-1009.

Text augmentation presents different challenges because language is discrete rather than continuous. Back-translation²³ offers one solution for machine translation by translating target-language monolingual text into the source language and pairing the synthetic source with the original target. Simpler approaches include synonym replacement and random insertion or deletion, although these transformations can change meaning and must be validated for the task.

Rather than hand-designing these augmentation policies, AutoAugment²⁴ uses reinforcement learning to discover augmentation strategies for specific datasets, while RandAugment²⁵ simplifies this by sampling from a fixed set of transformations. That compute-quality trade-off depends on both the target architecture and the invariances required at deployment.

²⁴ AutoAugment: Treats augmentation policy design as a reinforcement learning search problem: a controller selects operations (rotate, translate, shear, equalize), their magnitudes, and application probabilities to maximize validation accuracy (Cubuk et al. 2019). The original ImageNet search used approximately 15,000 GPU-hours, motivating later methods that reduce the search space.

Cubuk, Ekin D., Barret Zoph, Dandelion Mané, Vijay Vasudevan, and Quoc V. Le. 2019. “AutoAugment: Learning Augmentation Strategies from Data.” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 113–23. https://doi.org/10.1109/cvpr.2019.00020.

²⁵ RandAugment: Collapses AutoAugment’s policy search to two hyperparameters: transformation count $N$ and shared magnitude $M$ (Cubuk et al. 2020). That small search space matches or exceeds AutoAugment on the reported benchmarks at far lower search cost, making it practical when 15,000 GPU-hours of policy search cannot be justified.

Cubuk, Ekin D., Barret Zoph, Jonathon Shlens, and Quoc V. Le. 2020. “Randaugment: Practical Automated Data Augmentation with a Reduced Search Space.” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 3008–17. https://doi.org/10.1109/cvprw50498.2020.00359.

Lighthouse 1.2: MobileNetV2 and aggressive augmentation

Our MobileNetV2 lighthouse model from Lighthouse roster: Model biographies exemplifies how data augmentation can complement a resource-constrained architecture. Its depthwise separable convolutions reduce computation and parameter count relative to comparable standard convolutions, while augmentation can improve generalization when training data is limited.

The appropriate augmentation strength depends on the dataset, model, and invariances required at deployment. Techniques such as cropping and RandAugment can increase training diversity without increasing inference-time model capacity, but their magnitude and resulting accuracy must be measured for the target workload.

Generative synthesis: Creating new samples

Augmentation transforms existing samples; synthetic data generation goes further by creating new examples using generative models or simulators. It can be useful in three common scenarios:

Privacy-sensitive data: Synthetic examples can reduce direct exposure to real records, but still require privacy auditing for memorization or leakage.
Rare edge cases: Synthetic examples can cover failure modes, such as autonomous driving scenarios, that must be tested but seldom occur naturally.
Expensive collection: Synthetic examples can supplement domains such as robotics or scientific experiments where each real sample requires physical resources.

The distribution sketch in figure 8 shows the central risk of that strategy: synthetic examples can be plentiful and still misaligned with deployment data.

\begin{tikzpicture}[font=\small\sffamily]
\begin{axis}[
    width=11cm, height=50mm,
    xlabel={Feature Space},
    ylabel={Density},
    xmin=-3, xmax=7.25,
    ymin=0, ymax=0.7,
    axis lines=left,
    axis line style={thick,-latex},
    axis on top,
    xtick=\empty,
    ytick=\empty,
    ylabel style={font=\footnotesize\sffamily},
    xlabel style={font=\footnotesize\sffamily},
   % legend style={at={(0.98,0.98)}, anchor=north east, font=\footnotesize\sffamily, draw=none},
legend style={at={(0.85,0.92)}, anchor=north},
  legend cell align=left,
  legend style={fill=BrownL!30,draw=BrownLine,row sep=1.15pt,
  font=\fontsize{7pt}{7}\selectfont\sffamily},
    clip=false
]
    % Synthetic data distribution (centered at 0)
    \addplot[line width=1.25pt, blue, domain=-3:4, samples=100, fill=blue!20, fill opacity=0.5]
        {0.4*exp(-0.5*(x)^2)};
    \addlegendentry{Synthetic}

    % Real data distribution (centered at 3, slightly different shape)
    \addplot[line width=1.25pt, orange, domain=0:7, samples=100, fill=orange!20, fill opacity=0.5]
        {0.35*exp(-0.4*(x-3)^2)};
    \addlegendentry{Real}

    % Decision boundary learned from synthetic
    \draw[thick, dashed, red] (axis cs: 1.5, 0) -- (axis cs: 1.5, 0.6)
    node[red, font=\footnotesize\sffamily, fill=white,above=0pt,inner sep=1pt] {Synthetic boundary};

    % Ideal boundary for real data
    \draw[thick, dotted, green!60!black] (axis cs: 3, 0) -- (axis cs: 3, 0.50);
 %   \draw[thick, dotted, green!60!black] (axis cs: 0, 0) -- (axis cs: 0, 0.50);

    % Domain gap annotation
    \draw[latex-latex, thick, purple] (axis cs: 0, 0.48) --
    node[purple, font=\footnotesize\sffamily, fill=white, inner sep=1pt]{Domain Gap} (axis cs: 3, 0.48);

\end{axis}
\end{tikzpicture}

Figure 8: The Domain Gap Problem: Synthetic data (blue) and real data (orange) occupy shifted feature distributions. The dashed red synthetic boundary and dotted green real-data boundary illustrate how a boundary learned for one distribution can be misaligned with the other.

The choice among generative approaches is a cost-fidelity decision. Generative adversarial networks (GANs) train a generator against a discriminator in an adversarial training setup, producing realistic images through competition; StyleGAN, for instance, generates photorealistic faces that have augmented facial recognition datasets. Diffusion models use iterative denoising to produce high-quality images; systems like Stable Diffusion²⁶ support targeted training example generation from natural language descriptions through text-to-image synthesis. Finally, simulation engines such as CARLA for autonomous driving or Unity and Unreal for robotics offer physics-based rendering that can generate large volumes of labeled scenarios with known simulator state, making them particularly valuable for safety-critical applications where edge case coverage is essential.

²⁶ Stable Diffusion: Performs iterative denoising in a compressed latent space, enabling targeted text-to-image examples (Rombach et al. 2022). Its iterative generation is substantially more compute-intensive than geometric transforms, so it fits cases where novelty matters more than per-sample throughput.

Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. 2022. “High-Resolution Image Synthesis with Latent Diffusion Models.” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 10674–85. https://doi.org/10.1109/cvpr52688.2022.01042.

Bridging the domain gap

Relying on synthetic data generation introduces severe distribution alignment risks.²⁷ Examine the feature space mapping in figure 8, noting how decision boundaries trained on synthetic data shift away from real production deployment clusters.

²⁷ Domain gap: The statistical divergence between synthetic and real data distributions, measurable with metrics such as maximum mean discrepancy (MMD) or Frechet Inception Distance (FID). For ML systems, the gap becomes training-serving skew: a simulator-trained model can validate well on synthetic data while failing silently on real deployment data.

²⁸ Domain randomization: Makes synthetic data deliberately varied by randomizing textures, colors, lighting, and physical properties. The goal is not photorealism; it is enough variation to improve coverage of deployment conditions, shifting the cost bottleneck from rendering fidelity to coverage.

Two complementary strategies address this distribution mismatch. Domain randomization²⁸ takes an aggressive approach: rather than trying to match the real world precisely, it trains on wildly varied synthetic data by randomizing lighting, textures, backgrounds, and camera parameters during generation.

The aim is to make deployment conditions fall within the variation seen during training, but coverage is never guaranteed by randomization alone. The method can be effective in robotics and autonomous-driving settings when the simulator spans the relevant physical and visual factors and the result is validated on real data.

Domain adaptation takes the opposite approach by explicitly aligning synthetic and real distributions. Feature alignment methods train on synthetic data while simultaneously minimizing the distance between synthetic and real feature distributions, often using adversarial training to learn domain-invariant representations. Fine-tuning offers a simpler path: pretrain on abundant synthetic data to learn general features, then fine-tune on a small real dataset to adapt to deployment conditions. Self-training combines these ideas by using a synthetic-trained model to pseudo-label real unlabeled data, then retraining on the combined labeled set.

In practice, mixing synthetic and real data provides a real-data anchor while retaining some synthetic coverage, but the best ratio is workload-dependent. Table 9 summarizes the trade-off across different mixing ratios.

Table 9: Synthetic-to-Real Data Mixing Ratios: Pure synthetic data can suffer from distribution shift; pure real data is expensive. The optimal ratio varies by domain, simulator fidelity, model family, and validation protocol, so the table should be read as a scenario scaffold rather than a universal recipe.

Synthetic Fraction	Representative Outcome
100% synthetic	No real-data anchor; validate the domain gap
80% synthetic + 20% real	Lower real-data demand; validate on deployment data
50% synthetic + 50% real	More real-data anchoring; higher collection cost
100% real	No synthetic-to-real gap; highest collection cost

Example 1.4: KWS data selection

Scenario: An embedded keyword spotting (KWS) system targets deployment on a 256 KB SRAM microcontroller with a 10,000+ audio sample requirement.

Diagnosis: Manually recording 10,000 real utterances across diverse speakers and acoustic environments costs $20K–50K. Staging a 5-tier pipeline (seed recordings $\to$ augmentation $\to$ noise injection $\to$ hard-negative mining $\to$ TTS simulation) builds dataset scale from 500 seed samples.

Systems lesson: Layered data strategies can lower acquisition costs for resource-constrained edge ML deployments. In this scenario, augmentation and noise injection reduce physical collection cost to 5 percent of the full-recording estimate; target accuracy and acoustic coverage still require validation.

The keyword-spotting scenario shows how augmentation, noise injection, and simulation can expand a small seed set, but the same creation strategy becomes riskier when samples come from learned generators. In recursive training, model collapse²⁹ can amplify errors and reduce diversity over generations. Synthetic data therefore requires provenance, mixture controls, and validation against real deployment data rather than a blanket assumption that more generated samples help.

²⁹ Model collapse: Formally analyzed by Shumailov et al. (2024), this phenomenon occurs because generative models systematically underrepresent tail distributions – rare but important patterns that appear infrequently in training data. When generation $n+1$ trains on output from generation $n$, each successive generation further compresses the tails, producing increasingly homogeneous data. The degradation can be rapid in recursive training settings, which is why the Fallacies section of this chapter warns against pure synthetic training.

Shumailov, Ilia, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. 2024. “AI Models Collapse When Trained on Recursively Generated Data.” Nature 631 (8022): 755–59. https://doi.org/10.1038/s41586-024-07566-y.

Knowledge distillation: Compressing information

While data augmentation (section 1.5.1) and generative synthesis (section 1.5.2) create new input samples, another form of synthesis creates teacher-generated targets. Knowledge distillation³⁰ (Hinton et al. 2015; Gou et al. 2021) trains a student model using a teacher model’s outputs alongside or instead of hard labels. Soft targets can expose relative class probabilities that a one-hot label omits, although their value depends on teacher quality, temperature, and task alignment.

³⁰ Knowledge distillation: Soft teacher probabilities can reveal relationships among classes that one-hot labels omit. The temperature parameter controls how soft those probabilities become; it must be tuned because excessive smoothing can erase useful distinctions. Distillation adds teacher inference and does not guarantee a higher ICR unless the transferred signal improves the student enough to justify that cost.

Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean. 2015. “Distilling the Knowledge in a Neural Network.” arXiv Preprint.

Gou, Jianping, Baosheng Yu, Stephen J. Maybank, and Dacheng Tao. 2021. “Knowledge Distillation: A Survey.” International Journal of Computer Vision 129 (6): 1789–819. https://doi.org/10.1007/s11263-021-01453-z.

A teacher prediction such as [0.7, 0.2, 0.1] exposes relative preferences that a one-hot label [1, 0, 0] omits. A student may benefit from that signal when the teacher is accurate and aligned with the target distribution. At scale, teacher-generated targets add an inference pass and require quality controls; any student cost or quality advantage must be measured against that added generation expense.

Together, augmentation, generative synthesis, and distillation complete the third stage of the data selection pipeline. Where static pruning removes redundancy and dynamic selection focuses compute on high-value samples, synthetic generation fills gaps by creating samples that never existed. Selecting the right combination of techniques from these three pipeline stages requires a structured decision framework.

Self-Check: Question

What causes the phenomenon of ‘model collapse’ (or the autophagous loop) when generative models are trained recursively on synthetic data generated by earlier model iterations?
1. GPU memory fragmentation caused by variable-length synthetic sequences during distributed training
2. Overfitting to floating-point rounding errors during fp16 mixed-precision matrix multiplication
3. A failure of the I/O storage subsystem to deliver synthetic batches at line rate
4. Systematic underrepresentation and progressive pruning of the tail distributions of the true data distribution across successive generations
In knowledge distillation, what is the primary role of the temperature parameter $T$ when computing soft targets from a teacher model’s logits $z_i$ ($p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$)?
1. To soften the output probability distribution, exposing the relative probability structure (‘dark knowledge’) over non-target classes to the student
2. To clamp gradients to prevent numerical overflow in the student’s backward pass
3. To dynamically increase learning rate when the student loss plateaus
4. To randomly drop connections in the student network like dropout
Why is training exclusively on $100\%$ synthetic data from a simulation engine often suboptimal for real-world deployment, and how does synthetic-to-real data mixing bridge this gap?
True or False: Using a high-capacity diffusion model or state-of-the-art LLM to generate synthetic training data guarantees that the resulting training set will be completely free of distribution shift relative to the real deployment environment.
Order the stages in a multi-tier audio dataset expansion pipeline starting from a small seed recording: (1) Hard-negative mining, (2) Acoustic noise injection, (3) Seed recordings collection, (4) TTS simulation, (5) Geometric/Transformation-based audio augmentation.

See Answers →

Decision Framework

When labeling budget, redundancy, rare classes, privacy, and convergence speed all matter at once, the practitioner first has to identify which constraint is binding. Each stage can raise the information-compute ratio when its measured quality gain justifies its overhead: pruning removes low-value samples before training, dynamic selection focuses compute on high-value samples during training, and synthesis creates new high-value samples on demand (table 10).

Table 10: Three-Stage Data Selection Pipeline: The ranges illustrate possible outcomes rather than portable expectations. Actual gains depend on the dataset, model, selection method, and target metric.

Stage	When Applied	Techniques	Illustrative Outcome
Static pruning	Before training	Coreset Selection, Deduplication, Quality Filtering	30–50% dataset reduction
Dynamic selection	During training	Curriculum Learning, Active Learning, Semi-Supervised	10–30% faster curricula; 2–100$\times$ fewer labels
Synthetic generation	On-demand	Augmentation, Generative Models, Distillation	2–10$\times$ effective data expansion

Once the dominant constraint is named, table 11 compares which technique changes that constraint and why.

Table 11: Technique Selection Guide by Primary Constraint: Candidate data-selection techniques mapped to the dominant resource constraint in the ML pipeline. The table begins the diagnosis; prerequisites and measured ROI determine the final choice.

Constraint	Candidate Technique	Why
Limited labeling budget	Active Learning	Maximizes label ROI by selecting informative samples
High redundancy in data	Deduplication + Coreset	Removes waste before training begins
Rare classes or edge cases	Synthetic Generation	Creates samples that do not exist in raw data
Slow convergence	Curriculum Learning	Improves gradient quality in early training
Privacy requirements	Privacy-audited synthesis	Reduce direct use of real records; verify leakage
Large model, small dataset	Knowledge Distillation	Use teacher model’s knowledge as “data”

Table 11 maps individual constraints to techniques, but real projects face multiple constraints simultaneously. The decision tree in figure 9 structures the selection process hierarchically: start by identifying the primary bottleneck, then follow the branches to narrow the field.

\begin{tikzpicture}[font=\small\sffamily]
  \tikzset{
     Decision/.style={draw=BlueD,fill=cyan!10, trapezium,aspect=2,inner sep=-1.2ex,
     text width=30mm,line width=1pt,
                        diamond, minimum width=32mm, minimum height=20mm, align= flush center,
                        node distance=15mm and 25mm},
    Leaf/.style={
      rectangle,
      draw=GreenLine,
      fill=green!90!black!10,
      align=center,
      line width=1pt,minimum width=23mm,
      minimum height=1cm,
      font=\footnotesize\sffamily
    },
    Arrow/.style={->, >=Latex,line width=1pt, color=black!70}
  }
\def\be{8mm}
% Root
\node[Decision] (D1) {Primary\\bottleneck?};
% Left
\node[Decision, below left= 17mm and 31mm of D1] (D2) {Labeling\\cost};
\node[Decision, below = \be of D2] (D3) {Oracle\\available?};
\node[Leaf, left=14mm of D2] (B1){Self-\\Supervised};
\node[Leaf, below=7mm of D3] (B2){Active\\Learning};
\node[Leaf, left=14mm of D3] (B3){Semi-\\Supervised};
%right
\node[Decision, below right=17mm and 31mm  of D1] (D4) {Data\\scarcity};
\node[Decision, below = \be of D4] (D5) {Simulator\\available?};
\node[Leaf, right=14mm of D4] (B6){Knowledge\\Distillation};
\node[Leaf, right=14mm of D5] (B7){Data\\Augmentation};
\node[Leaf, below=7mm of D5] (B8){Synthetic\\Generation};
%center
\node[Decision, below = 7mm of D1] (D6) {Compute\\cost};
\node[Decision, below = \be of D6] (D7) {Redundant\\data?};
\node[Leaf, below=7mm of $(D3.south)!0.6!(D7.south)$] (B4){Dedup +\\Coreset};
\node[Leaf, below=7mm of $(D7.south)!0.4!(D5.south)$] (B5){Curriculum\\Learning};
%%arrows
\draw[Arrow](D1)-|node[above, pos=0.25, font=\scriptsize\sffamily] {Labeling $\$\$\$$}(D2);
\draw[Arrow](D1)-|node[above, pos=0.25, font=\scriptsize\sffamily] {Not enough data}(D4);
\draw[Arrow](D1)--node[right, pos=0.25, font=\scriptsize\sffamily] {Compute $\$\$\$$}(D6);
\draw[Arrow] (D2) -- node[above, pos=0.45, font=\scriptsize\sffamily] {Large pool} (B1);
\draw[Arrow](D2)--(D3);
\draw[Arrow](D3)--node[right, pos=0.25, font=\scriptsize\sffamily] {Yes}(B2);
\draw[Arrow](D3)--node[above, pos=0.25, font=\scriptsize\sffamily] {No}(B3);
%
\draw[Arrow](D7.west)-|node[above, pos=0.25, font=\scriptsize\sffamily] {High}(B4.120);
\draw[Arrow](D7.east)-|node[above, pos=0.25, font=\scriptsize\sffamily] {Low}(B5.65);
%
\draw[Arrow](D5)--node[right, pos=0.25, font=\scriptsize\sffamily] {Yes}(B8);
\draw[Arrow](D5)--node[above, pos=0.25, font=\scriptsize\sffamily] {No}(B7);

\draw[Arrow] (D4) -- node[above, pos=0.45, font=\scriptsize\sffamily] {Teacher} (B6);

\draw[Arrow](D4)--node[right, pos=0.25, font=\scriptsize\sffamily] {Domain}(D5);

\draw[Arrow](D6)--(D7);
\end{tikzpicture}

Figure 9: Data Selection Decision Tree: The tree begins with the primary bottleneck—labeling cost, compute cost, or data scarcity—and narrows each branch using available pools, oracles, redundancy, teachers, or simulators. Its leaves identify candidate techniques; projects with multiple binding constraints may require more than one branch.

Decision process

Each path requires a structured assessment because the same dataset symptom can point to different bottlenecks. Technique selection begins by naming the binding constraint, then checks whether the data, labels, and infrastructure needed by the chosen method exist.

Step 1: Assess the bottleneck

Identify which resource constraint most severely limits the training pipeline:

Labeling cost: Label-efficiency techniques such as active learning, semi-supervised learning, and self-supervised learning maximize the value extracted from each human annotation.
Compute cost: Coreset selection and deduplication can reduce processed samples, while a validated curriculum may reduce the steps needed to reach a target metric.
Data scarcity: Data creation through augmentation, synthesis, and distillation expands the effective training set beyond what raw collection provides.

This diagnostic step keeps technique selection tied to the binding resource constraint rather than to the most familiar algorithm.

Step 2: Check prerequisites

With the bottleneck identified, verify that the corresponding techniques are feasible given the available infrastructure and data. Each approach carries specific requirements that must be met before implementation can begin (table 12).

Table 12: Technique Prerequisites: Each technique carries specific infrastructure and data requirements that must be verified before implementation. A technique with excellent theoretical gains but unmet prerequisites will fail in practice, making this checklist the first step in technique selection.

Technique	Prerequisites
Active Learning	Access to oracle, unlabeled pool, retraining infrastructure
Coreset selection	Proxy model or embedding extractor, full dataset accessible
Curriculum Learning	Difficulty scoring method, pacing schedule
Semi-Supervised	Some labeled data, unlabeled data from same distribution
Self-Supervised	Large unlabeled corpus, pretraining compute budget
Augmentation	Domain knowledge of invariances, augmentation library
Synthetic Generation	Generative model or simulator, domain gap mitigation

Step 3: Estimate ROI

Before committing engineering resources, estimate each candidate technique’s return on investment: \[ \text{ROI} = \frac{\text{(Baseline Cost)} - \text{(Technique Cost + Implementation Cost)}}{\text{Technique Cost + Implementation Cost}} \]

A technique with high theoretical gains but high implementation cost may deliver lower ROI than a simpler approach. Exact deduplication is often an inexpensive first candidate because hashing is simple and repeated bytes are easy to identify, although downstream benefit still depends on corpus redundancy and intentional sample weighting. Active learning requires oracle access, retraining infrastructure, and selection algorithm development, so its ROI depends heavily on how many labeling cycles amortize that investment.

Step 4: Combine techniques

The techniques in this chapter are not mutually exclusive, but combining them does not guarantee additive gains. One possible workflow deduplicates the raw corpus, applies a coverage-aware coreset method, orders selected samples with a validated curriculum, and augments them at runtime. A pretrained representation can supply a different starting point from random initialization. Each added stage must earn its place because overlapping filters can remove the same examples while their overheads continue to accumulate.

Stages can compound efficiency gains when their effects do not overlap and their overhead remains small; combined savings must be measured rather than assumed.

This decision framework answers the what of data selection: which samples to prune, when to select dynamically, and how to synthesize new data. Understanding these algorithmic choices is essential, but algorithms alone do not translate into faster training. A perfectly designed coreset algorithm that takes 10 hours to select samples for a two-hour training run yields no practical benefit. Similarly, a curriculum learning strategy that requires scanning the entire dataset to determine difficulty rankings may idle GPUs while CPUs compute scores. The how of implementation matters as much as the what of algorithm choice.

The gap between algorithmic elegance and practical value raises several systems challenges: preventing selection overhead from negating theoretical gains, handling nonsequential I/O patterns that confuse prefetching logic, and coordinating selection decisions across distributed workers without introducing synchronization bottlenecks. The engineering patterns that follow bridge the gap between data selection theory and production reality.

Self-Check: Question

According to the chapter’s decision framework, if an ML team has an abundant pool of unlabeled domain data, very limited annotation budget, and access to human domain experts for selective queries, which technique branch is recommended?
1. Pure transformation-based data augmentation without labeling
2. Active learning (human-in-the-loop selective query) or semi-supervised learning
3. Generative self-instruct synthesis to replace all human annotators entirely
4. Exhaustive pairwise Jaccard deduplication across all unlabeled samples
When deciding between static coreset pruning and dynamic online active selection for a production pipeline, what role does the expected number of training runs ($N$) play in the architectural choice?
Order the decision steps when triaging a data pipeline bottleneck using the chapter’s decision framework: (1) Evaluate simulator and domain synthesizer availability, (2) Identify the primary constraint (label scarcity vs. compute limits vs. data scarcity), (3) Select the specific algorithmic technique (e.g. SSL, Active Learning, Coreset Pruning, or Generative Synthesis), (4) Assess human oracle availability and budget.

See Answers →

Selection Engineering

A naive active learning loop that scans the entire dataset every epoch to select the “best” samples can turn a compute-bound training job into an I/O-bound bottleneck. Selection engineering begins where the decision framework ends: after identifying which algorithms to apply, engineering must ensure that they deliver their promised speedups on real hardware and real data pipelines. The architectural patterns that prevent this failure implement data selection in production.

The selection bottleneck

Dynamic data selection introduces a new bottleneck: selection latency. A conventional loader follows a predetermined sampling plan, whereas active learning or adaptive curricula may evaluate a selection function $f(x)$ over a large candidate pool before forming later batches. Concretely, scoring a 1M dataset with a large model can take 2.8 hours, potentially negating the savings from a 10 percent coreset if not performed with a smaller proxy model. The systems trade-off requires that selection cost remain below the training work the subset saves; otherwise, sample counts fall while end-to-end cost rises.

At a fixed target metric, selection plus subset training must cost less than full-data training. Let $T_{\text{selection}}$ be wall-clock time spent scoring the pool, $T_{\text{train}}(D)$ be wall-clock training time for a dataset of size $D$, and $D_{\text{subset}}$ and $D_{\text{total}}$ be the retained and full sample counts. Equation 2 expresses this break-even condition, the selection inequality. \[ T_{\text{selection}} + T_{\text{train}}(D_{\text{subset}}) < T_{\text{train}}(D_{\text{total}}) \tag{2}\]

If $f(x)$ repeatedly scores the pool with a large model, recurring rescoring and coordination can consume the training savings and produce negative ROI. A concrete scenario illustrates this trade-off.

Equivalently, we can isolate the allowable selection overhead: $T_{\text{selection}} < T_{\text{train}}(D_{\text{total}}) - T_{\text{train}}(D_{\text{subset}})$. The allowable overhead therefore depends on how much training work the subset saves; no fixed percentage applies across workloads.

Example 1.5: Selection inequality in practice

Scenario: A computer vision team selects a 100K coreset (10 percent) from 1M images to reduce training time on an accelerator cluster.

Diagnosis: Full-model scoring (Option A: target ResNet-50) adds 2.8 h of selection overhead, reducing net savings to 247.2 h. Proxy-model scoring (Option B: ResNet-18) drops selection overhead to 0.6 h, saving 249.4 h total wall-clock hours.

Systems lesson: Coreset selection reduces training latency only when selection overhead is smaller than full-dataset training savings. Proxy scoring preserves net FLOP savings by keeping sample scoring overhead well below full-model forward pass costs.

Data selection improves end-to-end efficiency only if its overhead is smaller than the training time saved on the reduced dataset. Compare the three illustrative stacked time bars in figure 10, observing how an expensive selection function can consume all subset-training savings.

Figure 10: **The Selection Inequality**: Data selection only improves end-to-end efficiency if the overhead of selection plus training on the subset is less than training on the full dataset. A lightweight selection function (proxy model, cached embeddings) keeps selection overhead low; an expensive selection function (full model forward pass) can negate the savings.

The consequence shown in figure 10 is unambiguous. Selection overhead can negate the benefits of training on a smaller subset. The cost compounds when selection runs every epoch instead of once; if per-epoch scoring approaches the training work saved by the subset, the net advantage disappears.

Hardware empathy: The random-access penalty

The selection inequality addresses compute overhead, but some selection strategies also change I/O patterns. Index-driven sampling can turn large sequential shard reads into smaller scattered reads, reducing the benefit of readahead and batching. The impact depends on record size, queue depth, cache state, storage tier, and data-loader design. Table 13 illustrates one 4 KB, single-device comparison rather than a training-pipeline benchmark. Selection cost therefore includes the access pattern it induces. Equal-sized subsets can produce different wall-clock savings when one preserves sequential reads and the other fragments them.

Systems can mitigate this penalty with proxy models, cached embeddings, and indexed retrieval. A smaller proxy can reduce scoring cost only if its ranking transfers to the target model. Lower-precision inference may reduce that cost further when the runtime, model, and required ranking fidelity support it. Embedding indices such as FAISS³¹ replace some full scans with exact or approximate retrieval whose cost and recall depend on the index configuration. These approaches decouple selection from target-model training so each can be measured independently.

³¹ FAISS (Facebook AI Similarity Search): Provides GPU-accelerated similarity search using exact, approximate, and compressed-domain index designs (Johnson et al. 2019); Johnson, Douze, and Jegou report billion-vector graph construction on multiple GPUs, showing why vector-index infrastructure matters for web-scale selection. For data selection pipelines, FAISS-style indexing supports $k$-nearest-neighbor retrieval for coreset selection, embedding-based deduplication, and stratified clustering for balanced sampling. Without this infrastructure, embedding-based selection would often fall back to expensive full-corpus scans.

Johnson, Jeff, Matthijs Douze, and Hervé Jégou. 2019. “Billion-Scale Similarity Search with GPUs.” IEEE Transactions on Big Data 7 (3): 535–47. https://doi.org/10.1109/tbdata.2019.2921572.

Table 13: Illustrative Small-Read Penalty: Registry device anchors compare sequential bandwidth with random 4 KB reads under a simplified IOPS conversion. Real throughput depends on request size, queue depth, concurrency, caching, and service configuration; the cloud row is only a qualitative scenario.

Storage Tier	Sequential Throughput	Random I/O (IOPS)	Random Throughput (approx)	Random Penalty
HDD (7.2k)	~150 MB/s	~100 IOPS	~0.4 MB/s	375×
SATA SSD	~550 MB/s	~10K IOPS	~40 MB/s	13.8×
NVMe SSD	~3,500 MB/s	~500K IOPS	~2,000 MB/s	1.75×
Cloud (S3)	Configuration-dependent	Not IOPS-comparable	Configuration-dependent	Potentially extreme

Data loaders also require architectural adaptation. Sharded dataset formats package many samples into sequentially readable chunks; WebDataset and FFCV are concrete examples. Shuffle buffers are a data-machine co-design: the loader reads large sequential shards into memory and samples randomly within the buffer, preserving sequential I/O throughput while achieving the statistical benefits of random sampling. In multi-accelerator training, each worker maintains its own shuffle buffer over a non-overlapping shard, so the randomization is local rather than global; rare classes or boundary cases that appear in only a few shards receive uneven coverage across workers unless the coreset is first stratified and shards are balanced before distribution.

Checkpoint 1.2: The selection inequality

Data selection is not free. It introduces a new term to the iron law and a new I/O cost.

Equation checks:

Selection cost: can you explain why $T_{\text{selection}} + T_{\text{train}}(\text{subset})$ must be less than $T_{\text{train}}(\text{full})$ for the technique to reduce end-to-end training time?
Overhead management: how do proxy models and cached embeddings keep $T_{\text{selection}}$ low?

Systems implications:

I/O patterns: why can nonsequential access for dynamic selection reduce data-loader throughput relative to sequential reads?

Data echoing: Amortizing I/O costs

The optimizations discussed so far address I/O bandwidth, but modern data selection pipelines introduce another bottleneck: CPU computation. Synthetic data generation and heavy augmentation shift the constraint from disk speed to augmentation throughput. Heavy augmentations like 3D rotations and MixUp, or on-the-fly generative synthesis, can leave the GPU idle if the CPU cannot keep pace with sample production. When the data pipeline produces samples slower than the GPU can consume them, GPU utilization drops and training time extends, negating the efficiency gains from smarter data selection.

Data echoing³² (Choi et al. 2019) reuses data multiple times before fetching new samples, trading freshness for accelerator utilization when the input pipeline is slower than training. Echoing before randomized augmentation can produce different transformed inputs on each repetition, while echoing later in the pipeline may repeat identical tensors.

³² Data echoing: The key subtlety is where in the pipeline to insert the echo point. Echoing before augmentation (upstream echoing) can apply different random augmentations to each repetition, while echoing after augmentation feeds identical tensors to the accelerator. Useful echo factors and realized speedups depend on the workload, batch size, insertion point, and shuffling policy (Choi et al. 2019).

Choi, Dami, Alexandre Passos, Christopher J. Shallue, and George E. Dahl. 2019. “Faster Neural Network Training with Data Echoing.” arXiv Preprint, ahead of print. https://doi.org/10.48550/arXiv.1907.05550.

The optimal echo factor depends on the ratio $R$ of upstream processing time to downstream training time: \[ R = \frac{T_{\text{data pipeline}}}{T_{\text{GPU training}}} \]

If $R > 1$ (data pipeline is the bottleneck), an echo factor $e < R$ partially recovers idle GPU cycles, while $e \geq R$ can fully use GPU capacity if echoed samples remain statistically useful. Increasing $e$ beyond $R$ no longer improves utilization and can reduce sample diversity. If $R < 1$ (GPU is the bottleneck), data echoing provides no benefit. A realistic scenario makes these trade-offs concrete.

Pipeline utilization saturates near the ratio threshold; statistical value remains workload-dependent.

Napkin Math 1.4: Worked example: Data echoing ROI

Scenario: Training ResNet-50 on ImageNet with heavy augmentation (RandAugment + MixUp).

Measurements:

Data pipeline throughput: 300 images/s (reading, decoding, augmenting on CPU)
GPU training throughput: 800 images/s (forward + backward pass)
Ratio $R = T_{\text{pipeline}} / T_{\text{GPU}}$ = (1/300 images/s) / (1/800 images/s) = 800 images/s/300 images/s ≈ 2.67 (GPU waiting 62.5 percent of time)

Without echoing:

Effective throughput: 300 images/s (limited by data pipeline)
Training time for 90 epochs: $90 \times 1.28\text{M}$ / 300 images/s = 384,350 seconds (106.8 hours)
GPU utilization: 37.5 percent

With echo factor: $e$ = 2.

Each batch is processed twice with different augmentations
Effective throughput: 600 images/s (still below GPU capacity)
Unique images per second: 300 images/s (unchanged)
Training time: $90 \times 1.28\text{M}$ / 600 images/s = 192,175 seconds (53.4 hours) if echoed data is equally valuable

Trade-off: Repeated samples are not guaranteed to be as useful as fresh samples; the data echoing paper evaluates echo factor, insertion point, and shuffling because those choices determine whether reuse preserves predictive performance. In one network-fed ResNet-50/ImageNet configuration, Choi et al. (2019) reports a 3.25$\times$ reduction in wall-clock time to the target metric.

Systems insight: Data echoing can trade sample diversity for accelerator utilization when the input pipeline is the bottleneck. The useful echo factor must be measured for the workload because it depends on the batch size, insertion point, augmentation, shuffling, and target metric.

Echoing can introduce correlations when the same example appears within or across nearby batches, so implementations must preserve adequate shuffling and validate the target metric. Choi et al. (2019) did not observe a negative batch-normalization interaction in their ResNet-50 or Single Shot MultiBox Detector (SSD) experiments, but that result does not guarantee the same behavior for every workload.

The engineering patterns in this section make data selection deployable. Proxy selection can reduce scoring cost when its ranking transfers to the target model. Sharded formats and shuffle buffers reconcile randomized sampling with sequential storage access, while data echoing can recover idle accelerator time when input service is the bottleneck. Each pattern changes a different part of the pipeline, and each still requires workload-specific measurement of quality and end-to-end time.

Engineering patterns solve the how, but a more fundamental question remains: whether to invest in data selection at all. A deduplication pipeline that costs $50K to build but saves $10K per training run requires a cost model to justify.

Self-Check: Question

A training team reduces a dataset from $1{,}000{,}000$ to $100{,}000$ images (a $10\times$ coreset). Training on the full dataset takes 10 hours ($T_{\text{train}}(\text{full}) = 10\text{ hr}$), while training on the coreset takes 1 hour ($T_{\text{train}}(\text{subset}) = 1\text{ hr}$). However, scoring the $1\text{M}$ pool with the full production model takes 12 hours ($T_{\text{selection}} = 12\text{ hr}$). Does this configuration satisfy the Selection Inequality, and what engineering change restores positive ROI?
1. Yes, because the dataset was reduced by $90\%$; no engineering change is needed
2. Yes, because $10\text{ hr} - 1\text{ hr} = 9\text{ hr}$ of savings outweighs the scoring cost; increase GPU count by $2\times$
3. No, because $T_{\text{selection}} + T_{\text{train}}(\text{subset}) = 13\text{ hr} > 10\text{ hr}$; replace full-model scoring with a lightweight proxy model or cached embeddings
4. No, because coreset training always increases memory bandwidth consumption; switch from NVMe SSDs to HDDs
Why do naive random sample lookups across non-contiguous indices in large un-sharded dataset files severely degrade I/O throughput on storage hardware, and how do shuffle buffers mitigate this?
1. Random lookups bypass host CPU caches, forcing floating-point registers to re-encode all labels
2. Random lookups violate PCIe parity checks, causing GPU kernel timeouts during backward passes
3. Random lookups trigger hash collisions in the Python garbage collector, halting dataloading threads
4. Random 4 KB reads achieve only a tiny fraction of peak sequential storage bandwidth due to IOPS limits, whereas shuffle buffers read large sequential chunks and randomize locally in memory
Contrast upstream data echoing (echoing before data augmentation) with downstream data echoing (echoing after augmentation) in terms of computational overhead and sample diversity.
True or False: If the GPU training step takes 20 ms and the CPU data loading/augmentation pipeline takes 10 ms ($R = T_{\text{pipeline}} / T_{\text{GPU}} = 0.5$), applying a data echoing factor of $e = 2$ will double the end-to-end training throughput.
The pipeline optimization technique that reuses intermediate data samples multiple times before fetching new batches to keep accelerators saturated when CPU data processing or I/O is the bottleneck is called data ____.

See Answers →

Cost Modeling

That investment decision demands cost modeling and quantitative answers. Practitioners must determine whether to label 10,000 more samples or buy more GPU hours, when active learning pays for itself, and what ROI a deduplication infrastructure investment delivers.

Quantifying data costs and ROI

Answering these questions requires understanding what training data costs. The total cost of data encompasses the full lifecycle of data acquisition, preparation, and utilization, extending well beyond storage fees. Table 14 breaks down the four cost components. Labeling is usually the component data selection can change most directly, while storage and processing tend to scale more mechanically with retained volume and training passes. \[ C_{\text{total}} = C_{\text{acquire}} + C_{\text{label}} + C_{\text{store}} + C_{\text{process}} \]

Table 14: Total Cost of Training Data: The four cost components span the full data lifecycle. Here $D$ is the total sample count, $D_{\text{labeled}}$ is the labeled subset, $D_{\text{vol,store}}$ is stored byte volume, $T_{\text{months}}$ is retention time, $N_{\text{epochs}}$ is the number of training passes, and $O_{\text{sample}}$ is per-sample training work. The ranges are scenario anchors rather than universal market prices.

Component	Formula	Illustrative Range
$C_{\text{acquire}}$	$D \times c_{\text{sample}}$	$0.001–$10/sample (web scrape vs. licensed)
$C_{\text{label}}$	$D_{\text{labeled}} \times c_{\text{label}}$	$0.01–$200/sample (crowd vs. expert)
$C_{\text{store}}$	$D_{\text{vol,store}} \times c_{\text{storage}} \times T_{\text{months}}$	$0.02–$0.10/GB/month
$C_{\text{process}}$	$D \times N_{\text{epochs}} \times O_{\text{sample}} \times c_{\text{FLOP}}$	Proportional to training FLOPs

The interplay of these four cost terms becomes concrete when evaluating an ImageNet-scale vision model training run.

Napkin Math 1.5: Cost breakdown: ImageNet-scale training

Table 15 itemizes one illustrative acquisition, labeling, storage, and compute scenario at ImageNet scale. The resulting ratio follows from the stated licensing, annotation, storage, runtime, and compute-cost assumptions; it is not a typical ratio for every supervised workload.

Table 15: Illustrative ImageNet-Scale Cost Breakdown: Itemized acquisition, labeling, storage, and compute costs under the stated scenario assumptions. Each amount follows from the rate shown beside it, using the book’s published crowd-labeling, object-storage, and cloud-GPU rates. Data costs dominate this particular calculation because licensing and annotation are included while the training run itself is short.

Cost Component	Calculation	Amount
Raw data (1.2M images)	Licensed dataset, flat fee	$50,000
Labels (crowd annotation)	1.2M $\times$ $0.05/label	$60,000
Storage (cloud object store)	150 GB $\times$ $0.02/GB/month $\times$ 12 months	$36
Training campaign (30 runs, each 100 epochs in 24 h on 8 A100s)	5,760 GPU-hours $\times$ $4/GPU-hour	$23,040
Total		$133,076
Data vs. Compute ratio		82.7% data, 17.3% compute

Systems insight: Acquisition and labeling can dwarf compute cost in supervised vision workloads, as they do under these assumptions. The compute figure prices the whole training campaign, not one run: a published model is the survivor of sweeps, restarts, and ablations, and costing a single run would understate compute roughly thirtyfold. Even so, acquisition and labeling remain the larger share. The run count is an assumption like any other, so a complete ROI calculation must state it and price both categories rather than assuming which one dominates.

ROI framework for data selection techniques

Understanding total costs enables rational decisions about which efficiency techniques merit investment. Every technique carries both a cost (implementation effort, compute overhead) and a benefit (reduced data requirements, faster training). Comparing these trade-offs requires a common framework: Return on Investment (ROI). \[ \text{ROI} = \frac{\text{Savings} - \text{Investment}}{\text{Investment}} \times 100\% \]

The challenge lies in quantifying both sides accurately. Table 16 summarizes the cost and benefit categories to measure; their ranking depends on the corpus, task, and existing infrastructure.

Table 16: ROI Profiles for Data Selection Techniques: The table identifies costs and possible savings. No technique guarantees positive ROI; each must satisfy the selection inequality under measured workload conditions.

Technique	Investment (Cost)	Savings (Benefit)
Deduplication	One-time compute for hashing + infrastructure	Reduced storage and repeated-sample processing
Coreset Selection	Proxy model training + selection compute	Fewer retained samples if target quality and coverage hold
Active Learning	Inference on unlabeled pool + human-in-the-loop latency	Lower labeling demand when query selection transfers
Data Augmentation	CPU/GPU cycles for transforms	Effective dataset size increase without new data acquisition

Break-even analysis

ROI calculations assume that techniques deliver their promised benefits, but actual outcomes vary. For any technique, there exists a break-even point where investment equals savings. Below this threshold, the technique costs more than it saves; above it, the technique generates value. Identifying this threshold determines whether a technique makes sense for a given project.

Suppose labeling costs $10/sample, active learning starts from 1,000 labeled samples ($10,000), issues 100 queries per round at $50 inference cost, and the random-labeling baseline requires 5,000 samples for target accuracy. If active learning reaches target accuracy with only 2,000 labeled samples, the ROI follows from comparing labeling and compute costs. \[\begin{gather*} \text{Random labeling cost} = \text{5,000} \times \text{\$10/sample} = \text{\$50,000} \\ \text{Active learning cost} = \text{2,000} \times \text{\$10/sample} + \text{10 rounds} \times \text{\$50} = \text{\$20,500} \\ \text{ROI} = \frac{\text{\$50,000} - \text{\$20,500}}{\text{\$20,500}}= \text{$143.9\%$} \end{gather*}\]

Break-even occurs when avoided-label cost equals selection overhead. A 20 percent labeling reduction may still yield negative ROI if scoring and retraining are expensive.

Amortization across training runs

Break-even analysis captures a snapshot in time, but many data selection investments span multiple projects. Techniques with high upfront costs yield significant returns when their benefits compound across repeated training runs. Amortized return on investment (ROI) accounts for this temporal dimension, as table 17 and table 18 illustrate for a deduplication pipeline: \[ \text{Amortized ROI} = \frac{N_{\text{runs}} \times \text{Per-Run Savings} - \text{One-Time Investment}}{\text{One-Time Investment}} \times 100\% \]

Table 17: Deduplication Infrastructure Cost Components: The one-time investment covers engineering effort and initial compute; the per-run savings accrue with every subsequent training run on the deduplicated data.

Component	Cost
Build deduplication pipeline	$50,000 (engineering time)
Compute MinHash signatures (one-time)	$5,000
Per-run savings	$10,000/run

The ROI pattern in table 18 reveals which circumstances favor infrastructure investment. Three conditions tend to strengthen the return from data-selection investments:

Repeated training runs: Hyperparameter search, model iterations, and scheduled retraining reuse the same selection infrastructure many times.
Shared datasets: A cleaned or deduplicated corpus can support multiple teams or model architectures.
Broadly reusable techniques: Methods such as deduplication transfer across models, whereas task-specific coresets may not.

Table 18: Amortized ROI over Multiple Training Runs: Under this scenario’s costs and savings, a deduplication pipeline that loses money on its first use becomes profitable when reused across enough runs. Infrastructure investments should be evaluated over their expected lifetime rather than from one use.

Number of Runs	Amortized ROI
1 run	-81.8% (net loss)
5 runs	-9.1% (near break-even)
10 runs	+81.8% (positive)
50 runs	+809.1% (highly profitable)

Deduplication can be a high-transfer investment because every model using the corpus inherits the storage and repeated-content changes, although quality effects can still differ by task. Task-specific coresets may transfer less reliably across architectures, limiting their amortization potential. For one-off runs, simple techniques such as random sampling or basic augmentation may yield better ROI than methods requiring substantial infrastructure.

The investment decision therefore reduces to a practical split between high-reuse, data-limited workflows and one-off, already-curated ones.

These ROI calculations all assume a single machine. Production ML training distributes data across many workers, introducing coordination overhead that can erode or amplify those returns: a coreset algorithm designed for a single GPU may behave differently once its dataset is sharded across hundreds of workers.

Self-Check: Question

A company invests $C_{\text{select}} = \$30{,}000$ to compute a high-quality coreset. Training on the full dataset costs $C_{\text{train}}(D) = \$10{,}000$ per run, whereas training on the coreset costs $C_{\text{train}}(S) = \$4{,}000$ per run. What is the break-even number of training runs $N^*$ required to justify this static selection investment, and what is the ROI after 10 training runs?
1. $N^* = 5$ runs, and $\text{ROI} = 100\%$ after 10 runs (Net savings = $\$30{,}000$ on a $\$30{,}000$ investment)
2. $N^* = 3$ runs, and $\text{ROI} = 300\%$ after 10 runs
3. $N^* = 8$ runs, and $\text{ROI} = 50\%$ after 10 runs
4. $N^* = 10$ runs, and $\text{ROI} = 0\%$ after 10 runs
In the full lifecycle cost equation for machine learning data systems ($C_{\text{total}} = C_{\text{acquire}} + C_{\text{label}} + C_{\text{filter}} + C_{\text{train}} + C_{\text{eval}}$), which scenario demonstrates the most effective use of upstream data filtering to minimize total expenditure?
1. Spending $\$0$ on filtering to ensure maximum raw token count reaches the final evaluation cluster
2. Spending $\$5{,}000$ on automated heuristic filtering to discard $60\%$ of corrupt samples before paying $\$100{,}000$ in human labeling and training fees
3. Doubling human labeling rates to manually review every web-scraped token before filtering
4. Eliminating model evaluation to offset the compute cost of running unpruned training runs
Explain why calculating Return on Investment (ROI) for data selection requires tracking engineering implementation and pipeline maintenance costs in addition to raw accelerator compute hours.
True or False: In a production setting where a single model will be trained exactly once ($N=1$) with no hyperparameter tuning or future refreshes, spending 50 GPU-hours to compute static EL2N coreset scores that save 30 GPU-hours of training time is an economically sound decision.
The metric defined as $\text{ROI} = \frac{N \cdot \Delta C_{\text{train}} - C_{\text{select}}}{C_{\text{select}}}$, which measures the net financial or compute return generated by a data selection technique over $N$ training runs, is known as Return on ____.

See Answers →

Distributed Selection

The earlier sections in this chapter used a centralized view in which one process can see the full dataset, compute global statistics, and coordinate selection. That abstraction lets coreset methods rank all samples, curricula establish a shared ordering, and active learning compare a common candidate pool. Distributed training breaks the abstraction when data is sharded and scores are refreshed at different model states. Two difficult problems arise: approximating a global selection from local views and keeping training-time rankings comparable as the model evolves.

Whether distributed selection stays faithful to the global dataset or collapses into local shard heuristics depends on how much cross-worker coordination each technique requires against the bandwidth available to sustain it. The selection problem therefore has to be evaluated at the boundary where statistical value meets sharding and locality.

These are independent requirements. A coreset can remain representative yet cost too much to coordinate, while a cheap shard-local method can scale but systematically miss rare groups visible only in the global corpus. A distributed design therefore needs two acceptance tests: compare selected-data quality with a centralized or otherwise auditable reference, and compare selection overhead with the end-to-end training time it saves. Passing only one test is insufficient.

Strategies for distributed selection

In a basic data-parallel layout, each worker processes a distinct shard and model synchronization is handled separately. Data selection can add dependencies across those shards (table 19).

The selection dependencies admit several architectural solutions, each navigating a different point in the consistency-scalability trade-off space. The most straightforward approach centralizes selection while distributing training. A coordinator node performs selection on the full dataset, then distributes selected indices to workers. This preserves selection quality but introduces a single bottleneck:

Coordinator: score_all_samples() → selected_indices
Broadcast: selected_indices → all workers
Workers: train on subset(local_shard, selected_indices)

Table 19: Selection Dependencies in Distributed Training: The table starts from a common centralized formulation of each technique and identifies the coordination problem introduced by sharding. Distributed variants may relax rather than reproduce the centralized result exactly.

Technique	Single-Node Assumption	Distributed Challenge
Coreset Selection	Global view of dataset	Each worker sees only its shard
Active Learning	Centralized uncertainty scoring	Scoring requires model synchronization
Curriculum Learning	Global difficulty ordering	Workers may have different “hardest” samples
Deduplication	Hash table fits in memory	Distributed hash tables add latency

The semantics remain clean, but the coordinator becomes a single point of failure and a possible bandwidth bottleneck. Whether that overhead is acceptable depends on the selection payload, refresh rate, network path, and cluster size.

Hierarchical selection addresses this scalability limitation by distributing the selection computation itself. Each worker performs local selection on its shard, then a coordinator merges results:

Workers: local_selected = select_top_k(local_shard)
Coordinator: global_selected = merge_and_rerank(all local_selected)
Broadcast: final_indices → all workers

Shard-local selection reduces coordinator load but introduces a quality trade-off: local quotas and score distributions may not preserve the global ranking or rare groups spread unevenly across shards. The merge step therefore needs normalization, coverage constraints, or a second global pass.

When even hierarchical approaches prove too expensive, approximate global selection offers a fallback. These methods trade exactness for scalability through distributed approximate algorithms. Distributed MinHash enables deduplication by having each worker compute MinHash signatures independently; signatures are then aggregated to find near-duplicates across shards without requiring any single node to see all the data. Similarly, distributed uncertainty sampling allows workers to compute local uncertainty scores, with a global threshold determined by score distribution statistics rather than exact ranking.

Consistency challenges in active learning

The approximate selection strategies assume static selection criteria, but active learning introduces an additional complication: the model changes during selection. Consider what happens when Worker A scores samples using the model at step $t$ while Worker B simultaneously updates the model to step $t+1$. Worker A’s scores are now stale and may select samples that the updated model would rank differently.

The scoring schedule therefore defines the meaning of a query, not merely its cost. A reproducible system versions the model checkpoint, candidate-pool snapshot, scoring rule, and selected indices together. It can then measure how much ranking agreement decays with checkpoint age and choose a refresh interval from that evidence rather than from an arbitrary number of steps.

Several strategies mitigate this staleness problem, each with distinct overhead characteristics:

Synchronous scoring: All workers pause training and score simultaneously, guaranteeing consistency but at substantial cost in GPU utilization.
Periodic score refresh: Workers re-score every $k$ epochs rather than every batch, trading freshness for reduced overhead.
Checkpoint-robust selection: The system selects samples that exhibit high uncertainty under multiple model checkpoints, keeping selection decisions valid as the model evolves.

An illustrative 8-node GPU-cluster scenario demonstrates how these refresh strategies interact with storage bandwidth and compute efficiency.

Example 1.6: Distributed coreset selection

Scenario: An 8-node A100 GPU cluster selects a 10 percent coreset from ImageNet (1.3M images) across distributed workers.

Diagnosis: Centralized coreset scoring on a single node creates memory and compute bottlenecks. Distributed pipeline staging (shard-local embedding $\to$ parallel deduplication $\to$ local EL2N scoring $\to$ centralized top-$k$ merge) processes dataset selection in 67 minutes.

Systems lesson: Distributed coreset selection can remove single-node memory as the scoring limit, as illustrated in figure 11. Sharding embedding and scoring spreads work across the cluster, while the centralized merge remains a coordination cost that must satisfy the selection inequality.

\scalebox{0.85}{%
\begin{tikzpicture}[font=\small\sffamily]
\tikzset{
  ScoreLine/.style={-{Triangle[width=6pt,length=6pt]}, line width=1.2pt,BrownLine!70,text=black},
  IndexLine/.style={-{Triangle[width=6pt,length=6pt]}, line width=1.2pt,BlueLine!75,text=black},
  Box/.style={inner xsep=2pt, draw=BlueD, line width=0.75pt,fill=BlueL!70,
    text width=70mm,align=left,minimum width=110mm, minimum height=18mm},
   Box2/.style={Box, draw=GreenLine,fill=green!80!black!10,align= flush center,
   text width=23mm,minimum width=23mm, minimum height=16mm},
   Text/.style={right,fill=none,text=black!70,font=\footnotesize\sffamily,align=left},
}

% Coordinator node
\node[Box] (B0) {};
\node[draw=none,fill=BlueD,text=white,below=0pt of B0.north] (BB0) {\textbf{Coordinator Node}};
\node[draw=none,align=left,anchor=south]at($(B0.south)+(0,2pt)$){%
    \textbullet\ Maintains global embedding index (FAISS)\\
    \textbullet\ Merges local selections\\
    \textbullet\ Broadcasts final coreset indices};

% Worker 1
\node[Box2,below=of B0.south] (W0){};
\node[Box2,below=0pt of W0.north,minimum height=6mm,
fill=GreenLine,text=white] (WW0){\textbf{Worker 1}};
\node[align=center]at($(WW0.south)!0.5!(W0.south)$){150K images\\
    Local EL2N};
% Worker 0
\node[Box2,below=of B0.194] (W1){};
\node[Box2,below=0pt of W1.north,minimum height=6mm,
fill=GreenLine,text=white] (WW1){\textbf{Worker 0}};
\node[align=center]at($(WW1.south)!0.5!(W1.south)$){150K images\\
    Local EL2N};
% Worker N
\node[Box2,below=of B0.346] (W2){};
\node[Box2,below=0pt of W2.north,minimum height=6mm,
fill=GreenLine,text=white] (WW2){\textbf{Worker N}};
\node[align=center]at($(WW2.south)!0.5!(W2.south)$){150K images\\
    Local EL2N};
% Bidirectional coordination: local scores flow up; final coreset indices flow down.
\foreach \i in{0,1,2}{
\draw[ScoreLine] ([xshift=-1.7mm]W\i.north) -- ([xshift=-1.7mm]W\i.north |- B0.south);
\draw[IndexLine] ([xshift=1.7mm]W\i.north |- B0.south) -- ([xshift=1.7mm]W\i.north);
}
\node at ($(W1)!0.5!(W0)$) {\Large$\cdots$};
\node at ($(W2)!0.5!(W0)$) {\Large$\cdots$};
\end{tikzpicture}}

Figure 11: Distributed Coreset Selection Architecture: Parallel shard-local scoring reduces the work assigned to a centralized rank-merging coordinator. Workers send local EL2N scores upward; the coordinator merges them and broadcasts final coreset indices. The design is beneficial only when this coordination overhead remains below the training time saved by the coreset.

Positive ROI can erode quickly when workers coordinate frequently during training. Distributed selection introduces a coordination tax whose size depends on what is synchronized and how often. That tax must remain smaller than the training time saved; if it approaches the limit, simplify the strategy or increase the refresh interval.

A further constraint arises from cluster network topology. Gradient synchronization may use accelerator collectives over dedicated links, while embedding vectors, score arrays, and coreset indices may follow host or storage paths instead. The actual route depends on system architecture. Selection traffic can therefore expose CPU, network interface card, storage, or collective-network bottlenecks that the training profile alone does not reveal.

Real ML systems combine data selection with model-level efficiency, machine-level throughput, and distributed training simultaneously. These optimizations interact in ways that can amplify or undermine each other, and understanding these interactions is essential for designing efficient end-to-end pipelines.

Self-Check: Question

In a distributed training environment with hundreds of data-parallel worker nodes, why does standard centralized coreset selection fail to scale, and what trade-off does hierarchical selection introduce?
1. Centralized selection requires all workers to share a single GPU; hierarchical selection distributes weights across SSDs
2. Centralized selection fails because sharding prevents network cards from transmitting floating-point values
3. Centralized selection creates a communication and memory bottleneck at the coordinator node; hierarchical selection prunes locally per shard, risking loss of globally rare samples across shards
4. Centralized selection eliminates gradient synchronization; hierarchical selection disables local backpropagation
In distributed active learning, Worker A scores candidate pool samples using model checkpoint step $t$, while asynchronous Worker B updates the shared model parameters to step $t+100$. What consistency challenge arises, and what is the systems remedy?
1. Worker A encounters deadlock in CUDA streams; the remedy is disabling PyTorch autograd
2. Worker B overwrites Worker A’s local storage; the remedy is mounting read-only NFS drives
3. Worker A’s GPU runs out of memory; the remedy is reducing batch size to 1
4. Worker A scores samples against a stale model state, producing invalid uncertainty rankings; the remedy is checkpoint versioning or periodic synchronized score refreshes
Explain why performing independent, shard-local coreset pruning on an unstratified, partitioned dataset can cause minority class collapse during distributed training.
Order the execution phases of a distributed coreset selection workflow across a GPU cluster: (1) Compute shard-local embeddings and perform local near-deduplication, (2) Aggregate local candidate indices at the central coordinator, (3) Perform global proxy scoring and thresholding to produce final indices, (4) Broadcast final coreset index list to all worker nodes.

See Answers →

Cross-Layer Interactions

Data selection does not exist in isolation. A coreset-trained model may later be simplified for deployment. A curriculum-learning pipeline will run on specialized accelerators. An actively-learned dataset will feed into distributed training. These cross-layer interactions can amplify gains or introduce unexpected conflicts. Understanding these interactions helps practitioners design end-to-end efficient systems rather than optimizing components independently.

Model-level efficiency

Model-level efficiency reduces the size or arithmetic cost of the trained model through techniques such as pruning, lower-precision representation, and distillation. The training corpus can affect the learned representation and therefore the outcome of later compression, but a smaller or curated dataset does not guarantee a more compressible model. Model Compression and Hardware Acceleration return to the hardware consequences of model simplification.

Systems Perspective 1.2: The sparsity latency trap

Context: The compressibility advantage from data selection is counted in arithmetic operations, but model compression research repeatedly shows that pruning reduces those counts more easily than it reduces wall-clock latency.

Failure mode: FLOPs can decrease dramatically while inference latency stays flat or increases. Dense matrix multiplication hardware rewards regular layout and reuse, while sparse matrices require irregular memory access, metadata checks, and address jumps. Unless the sparsity pattern matches the execution substrate, the overhead of managing sparsity can outweigh the reduction in arithmetic.

Systems insight: FLOPs are not latency. A 99 percent reduction in operations can yield a 0 percent reduction in time if the remaining operations are memory bound or cache-inefficient. Optimization must target the hardware’s binding bottleneck rather than an abstract metric alone (Hoefler et al. 2021).

Hoefler, Torsten, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. 2021. “Sparsity in Deep Learning: Pruning and Growth for Efficient Inference and Training in Neural Networks.” Journal of Machine Learning Research 22 (241): 1–124.

The mechanism relates to how models encode information. A model trained on repetitive data can learn redundant features that pruning later removes. The training compute required to learn those features was wasted, only to be discarded during compression. By contrast, a model trained on diverse, informative samples may learn compact, nonredundant representations from the start, making subsequent compression easier to evaluate and sometimes easier to apply. Treat this as an engineering hypothesis to measure after curation rather than a guaranteed property of every selected dataset. Data selection and model-level efficiency are complementary tools, but their interaction must be evaluated jointly by comparing compression quality and deployment performance after curation rather than assuming an ordering advantage.

Machine-level throughput

While model-level efficiency affects what work remains after training, machine-level throughput determines how efficiently training itself proceeds. Specialized accelerators, kernel optimization, and parallel execution increase effective throughput. Data selection affects which machine bottlenecks dominate, and this relationship is more nuanced than simple speedup calculations suggest, as table 20 illustrates.

Table 20: Diagnosing Bottlenecks after Data Selection: Dataset and access-pattern changes can expose a different bottleneck, but dataset size alone does not determine the limiting resource. Re-profile the pipeline and match the response to the measured condition.

Observed Condition	Possible Bottleneck	Candidate Response
Input rate below accelerator demand	Storage, decode, or host link	Sharding, prefetching, parallel decode
Low-intensity training kernels	Accelerator memory bandwidth	Fusion, reduced traffic, lower precision
Dynamic scoring dominates iteration	Selection compute or index I/O	Proxy models, cached scores or embeddings

Data selection can therefore shift the system from one bottleneck regime to another. A technique that reduces dataset size by 80 percent may expose input-pipeline latency, selection compute, or true GPU compute as the next bottleneck, requiring different optimizations in each case. Before applying aggressive data reduction, profile the system to understand which bottleneck is being targeted.

Distributed training

The hardware bottleneck analysis in section 1.10.2 assumes single-machine training. The interactions become more complex when scaling to multiple machines, because data selection affects different parallelism strategies in distinct ways.

With worker count, batch size, and epoch count fixed, fewer retained samples mean fewer gradient synchronizations and less communication over the full run. Step-based schedules or compensating epochs may remove that saving. Smaller per-worker shards can also improve locality, although access order, record size, cache capacity, and sharding determine the actual I/O effect.

The benefits must be weighed against the distributed selection challenges discussed in section 1.9. A technique that works well on a single GPU may incur prohibitive coordination overhead across 1,000 workers, negating its efficiency gains.

The optimization stack

The earlier subsections in this section examined pairwise interactions, but production systems apply all these optimizations together. Trace the full optimization stack in figure 12, from data to deployment: each stage in this pipeline amplifies or attenuates the effects of others.

\begin{tikzpicture}[font=\small\sffamily]
\tikzset{
Box/.style={align=center, inner xsep=2pt,draw=GreenLine, line width=1pt,
node distance=10mm,fill=none, minimum width=28mm, minimum height=20mm},
Box1/.style={Box,draw=OrangeLine,fill=none},
BoxD/.style={Box,font=\footnotesize\sffamily\bfseries,anchor=south,draw=none,text=white,fill=GreenLine,minimum height=5mm},
BoxD1/.style={Box,font=\footnotesize\sffamily\bfseries,anchor=south,draw=none,text=white,fill=OrangeLine,minimum height=5mm},
Circle1/.style={circle,  minimum size=33mm, draw=none, fill=BrownLine!20},
LineD/.style={BrownLine!60!black!20,line width=4.0pt,dashed,dash pattern=on 5pt off 2pt,
{-{Triangle[width=1.5*6pt,length=2.0*5pt]}},shorten <=5pt,shorten >=1pt},
LineA/.style={BrownLine!80!black!40,line width=4.0pt,
{-{Triangle[width=1.5*6pt,length=2.0*5pt]}},shorten <=1pt,shorten >=1pt},
ALineA/.style={violet!60,{Circle[line width=1.1pt,fill=white,round,length=5pt,width=5pt]}-,
line width=1.5pt,shorten <=-3pt,shorten >=-6pt}
}

%dataS
\tikzset{%
 pics/dataS/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=FUNNEL,scale=\scalefac, every node/.append style={transform shape}]
%plats
\draw[fill=\filllcolor,line width=\Linewidth,draw=\drawcolor](0,-0.3)--(-0.67,0.03)--(0,0.37)--(0.67,0.03)--cycle;
\draw[fill=\filllcirclecolor,line width=\Linewidth,draw=\drawcolor](0,0)--(-0.67,0.33)--(0,0.67)--(0.67,0.33)--cycle;
\draw[fill=\filllcolor,line width=\Linewidth,draw=\drawcolor](0,0.3)--(-0.67,0.63)--(0,0.97)--(0.67,0.63)--cycle;
%left
\draw[line width=\Linewidth,draw=\drawcolor](-0.39,1.21)--++(210:0.55)--++(270:1.22)--(-0.39,-0.56);
\fill[line width=\Linewidth,fill=\filllcirclecolor!60!violet!,draw=green,draw=\drawcolor](-0.39,1.21)circle(3pt);
\fill[line width=\Linewidth,fill=\filllcolor,draw=green,draw=\drawcolor](-0.39,-0.56)circle(3pt);
\draw[line width=\Linewidth,draw=\drawcolor](0.39,1.21)--++(330:0.55)--++(270:1.22)--(0.39,-0.56);
\fill[line width=\Linewidth,fill=\filllcirclecolor!60!violet!,draw=green,draw=\drawcolor](0.39,1.21)circle(3pt);
\fill[line width=\Linewidth,fill=\filllcolor,draw=green,draw=\drawcolor](0.39,-0.56)circle(3pt);
\end{scope}
     }
  }
}
%AI style
\tikzset{
pics/llm/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
\node[circle,minimum size=12mm,draw=\drawcolor, fill=\filllcolor!70,line width=1.25*\Linewidth](C\picname) at (0,0){};
\def\startangle{90}
\def\radius{1.15}
\def\radiusI{1.1}
\foreach \i [evaluate=\i as \j using \i+1] [count =\k] in {0,2,4,6,8} {
\pgfmathsetmacro{\angle}{\startangle - \i * (360/8)}
\draw[draw=black,-{Circle[black ,fill=\filllcirclecolor,length=5.5pt,line width=0.5*\Linewidth]},line width=1.5*\Linewidth](C\picname)--++(\startangle - \i*45:\radius) ;
\node[circle,draw=black,fill=\filllcirclecolor!80!red!50,inner sep=3pt,line width=0.5*\Linewidth](2C\k)at(\startangle - \j*45:\radiusI) {};
}
\draw[line width=1.5*\Linewidth](2C1)--++(-0.5,0)|-(2C2);
\draw[line width=1.5*\Linewidth](2C3)--++(0.5,0)|-(2C4);
\node[circle,,minimum size=12mm,draw=\drawcolor, fill=\filllcolor!70,line width=0.5*\Linewidth]at (0,0){};
\end{scope}
    }
  }
}
%brain
\tikzset{pics/brain/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=BRAIN,scale=\scalefac, every node/.append style={transform shape}]
\draw[fill=\filllcolor,line width=\Linewidth](-0.3,-0.10)to(0.08,0.60)
to[out=60,in=50,distance=3](-0.1,0.69)to[out=160,in=80](-0.26,0.59)to[out=170,in=90](-0.46,0.42)
to[out=170,in=110](-0.54,0.25)to[out=210,in=150](-0.54,0.04)
to[out=240,in=130](-0.52,-0.1)to[out=300,in=240]cycle;
\draw[fill=\filllcolor,line width=\Linewidth]
(-0.04,0.64)to[out=120,in=0](-0.1,0.69)(-0.19,0.52)to[out=120,in=330](-0.26,0.59)
(-0.4,0.33)to[out=150,in=280](-0.46,0.42)
%
(-0.44,-0.03)to[bend left=30](-0.34,-0.04)
(-0.33,0.08)to[bend left=40](-0.37,0.2) (-0.37,0.12)to[bend left=40](-0.45,0.14)
(-0.26,0.2)to[bend left=30](-0.24,0.13)
(-0.16,0.32)to[bend right=30](-0.27,0.3)to[bend right=30](-0.29,0.38)
(-0.13,0.49)to[bend left=30](-0.04,0.51);
\draw[thick,line width=\Linewidth,rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcolor,length=2.5pt]}](-0.23,0.03)--(-0.15,-0.03)--(-0.19,-0.18)--(-0.04,-0.28);
\draw[thick,line width=\Linewidth,rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcolor,length=2.5pt]}](-0.17,0.13)--(-0.04,0.05)--(-0.06,-0.06)--(0.14,-0.11);
\draw[thick,line width=\Linewidth,rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcolor,length=2.5pt]}](-0.12,0.23)--(0.31,0.0);
\draw[thick,line width=\Linewidth,rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcolor,length=2.5pt]}](-0.07,0.32)--(0.06,0.26)--(0.16,0.33)--(0.34,0.2);
\draw[thick,line width=\Linewidth,rounded corners=0.8pt,\drawcircle,-{Circle[fill=\filllcolor,length=2.5pt]}](-0.01,0.43)--(0.06,0.39)--(0.18,0.51)--(0.31,0.4);
\end{scope}
     }
  }
}
%funnel
\tikzset{%
 pics/funnel/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=FUNNEL,scale=\scalefac, every node/.append style={transform shape}]
\draw[fill=\filllcolor!50,line width=\Linewidth,draw=\drawcolor](-0.12,-0.81)--(-0.19,-0.25)--(-0.7,0.41)--(0.7,0.41)--(0.19,-0.25)--(0.12,-0.81)--cycle;
\draw[fill=\filllcolor!50,line width=\Linewidth,draw=\drawcolor](-0.19,-0.25)--(0.08,-0.25);
\draw[fill=\filllcolor!50,line width=\Linewidth,draw=\drawcolor](0.16,-0.09)--(0.41,0.31);
%
\node[line width=\Linewidth,draw=\drawcolor,fill=\filllcolor!50,inner sep=1pt,
rectangle,rounded corners=2pt,minimum width=16mm,minimum height=5pt]at(0,0.5){};
%
\foreach \i in{-0.5,0,0.5}{
\node[single arrow, line width=0.8*\Linewidth,draw=black,fill=cyan!90!black!30, rotate=270,inner sep=1pt,
      minimum width =9pt, single arrow head extend=2pt,
      minimum height=5mm]at(\i,0.9) {}; % length of arrow
   }
\node[single arrow,line width=0.8*\Linewidth,draw=black,fill=cyan!90!black!30, rotate=270,inner sep=1pt,
      minimum width =11pt, single arrow head extend=2pt,
      minimum height=5mm]at(0,-1.1) {}; % length of arrow
 \end{scope}
     }
  }
}
\tikzset{
pics/algorithm/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
\node[fill=\filllcolor!60,draw=\drawcolor,line width=\Linewidth,rectangle,minimum size=2.5mm,
rounded corners=1pt,inner sep=1pt](\picname-B2)at(0,-0.47){};
\node[fill=\filllcolor!30,draw=\drawcolor,line width=\Linewidth,rectangle,minimum size=2.5mm,
rounded corners=1pt,inner sep=1pt](\picname-B3)at(-0.6,-0.47){};
\node[fill=\filllcolor!30,draw=\drawcolor,line width=\Linewidth,rectangle,minimum size=2.5mm,
rounded corners=1pt,inner sep=1pt](\picname-B1)at(0.6,-0.47){};
%
\node[fill=\filllcolor!99!violet!80,draw=\drawcolor,line width=\Linewidth,rectangle,
minimum width=8.5mm,minimum height=3mm,
rounded corners=2pt,inner sep=1pt](\picname-B0)at(0,0.53){};
\draw[draw=\drawcolor,shorten >=4pt,shorten <=4pt,line width=1.5*\Linewidth]
($(\picname-B0.north west)!0.33!(\picname-B0.south west)$)--($(\picname-B0.north east)!0.33!(\picname-B0.south east)$);
\draw[draw=\drawcolor,shorten >=4pt,shorten <=4pt,line width=1.5*\Linewidth]
($(\picname-B0.north west)!0.66!(\picname-B0.south west)$)--($(\picname-B0.north east)!0.66!(\picname-B0.south east)$);
\draw[draw=\drawcolor,rounded corners](\picname-B1)|-(\picname-B0);
\draw[draw=\drawcolor,rounded corners](\picname-B3)|-(\picname-B0);
\draw[draw=\drawcolor,rounded corners](\picname-B2)--(\picname-B0);
\node[fill=\filllcirclecolor!60,draw=\drawcolor,line width=\Linewidth,rectangle,minimum size=2.5mm,
rounded corners=1pt,inner sep=1pt,rotate=45](R2){};
\node[fill=\filllcirclecolor!30,draw=\drawcolor,line width=\Linewidth,rectangle,minimum size=2.5mm,
rounded corners=1pt,inner sep=1pt,rotate=45](R1) at (-0.6,0){};
\node[fill=\filllcirclecolor!30,draw=\drawcolor,line width=\Linewidth,rectangle,minimum size=2.5mm,
rounded corners=1pt,inner sep=1pt,rotate=45](R3) at (0.6,0){};
\end{scope}
    }
  }
}
%CPU
\tikzset{%
 pics/cpu/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=FUNNEL,scale=\scalefac, every node/.append style={transform shape}]
\node[fill=\filllcolor,minimum width=66, minimum height=66,
            rounded corners=2,outer sep=2pt] (C1) {};
\node[fill=white,minimum width=54, minimum height=54] (C2) {};
\node[fill=\filllcolor!40,minimum width=44, minimum height=44] (C3) {\large CPU};

\foreach \x/\y in {0.11/1,0.26/2,0.41/3,0.56/4,0.71/5,0.85/6}{
\node[fill=\filllcolor,minimum width=3, minimum height=15,
           inner sep=0pt,anchor=south](GO\y)at($(C1.north west)!\x!(C1.north east)$){};
}
\foreach \x/\y in {0.11/1,0.26/2,0.41/3,0.56/4,0.71/5,0.85/6}{
\node[fill=\filllcolor,minimum width=3, minimum height=15,
           inner sep=0pt,anchor=north](DO\y)at($(C1.south west)!\x!(C1.south east)$){};
}
\foreach \x/\y in {0.11/1,0.26/2,0.41/3,0.56/4,0.71/5,0.85/6}{
\node[fill=\filllcolor,minimum width=15, minimum height=3,
           inner sep=0pt,anchor=east](LE\y)at($(C1.north west)!\x!(C1.south west)$){};
}
\foreach \x/\y in {0.11/1,0.26/2,0.41/3,0.56/4,0.71/5,0.85/6}{
\node[fill=\filllcolor,minimum width=15, minimum height=3,
           inner sep=0pt,anchor=west](DE\y)at($(C1.north east)!\x!(C1.south east)$){};
}
 \end{scope}
     }
  }
}
\tikzset{
pics/llm1/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
\node[circle,minimum size=13mm,draw=\drawcolor, fill=\filllcolor!70,line width=\Linewidth](C\picname) at (0,0){\bfseries LLM};
\def\startangle{110}
\def\radius{1.75}
\def\radiusI{1.4}
\foreach \i [evaluate=\i as \j using \i+1] in {0,2,4,6,8} {
\pgfmathsetmacro{\angle}{\startangle - \i * (360/10)}
\draw[draw=\drawcolor,line width=0.7*\Linewidth,-{Circle[\drawcolor,fill=\filllcirclecolor,length=7.0pt]}](C\picname)--++(\startangle - \i*36:\radius) ;
\draw[draw=\drawcolor,line width=0.7*\Linewidth,-{Circle[\drawcolor,fill=\filllcirclecolor!80!red!50,length=4.5pt]}](C\picname)--++(\startangle - \j*36:\radiusI) ;
}
\end{scope}
    }
  }
}
\tikzset{
pics/rocket/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[shift={($(0,0)+(0,0)$)},scale=\scalefac,every node/.append style={transform shape}]
%vrh
\draw[fill=\filllcolor,draw=\drawcolor,line width=\Linewidth](-0.26,0.5)to[bend right=12](0.26,0.5)to[bend right=7] (0,0.85)to[bend right=7] cycle;
%krila
\draw[fill=\filllcolor!70!red,,draw=\drawcolor,line width=\Linewidth,rounded corners=1pt](-0.2,-0.7)--(-0.45,-0.9)--(-0.567,-0.4)--(-0.3,-0.17)--cycle;
\draw[fill=\filllcolor!70!red,draw=\drawcolor,line width=\Linewidth,rounded corners=1pt](0.2,-0.7)--(0.45,-0.9)--(0.567,-0.4)--(0.3,-0.17)--cycle;
%rep
\draw[fill=\filllcolor,draw=\drawcolor,line width=\Linewidth](0.16,-0.76)--(0.22,-0.9)--(-0.2,-0.9)--(-0.15,-0.76)--cycle;
%body
\draw[fill=\filllcolor,draw=\drawcolor,line width=\Linewidth](-0.2,-0.7)--(0.2,-0.7)to[out=75,in=320](0,0.85)to[out=220,in=105] cycle;
%krug
\node[circle,draw=\drawcolor,minimum size=4mm,fill=\filllcirclecolor,line width=\Linewidth]{};
\draw[draw=\drawcolor](0,-0.99)--(0,-1.3);
\draw[draw=\drawcolor](-0.11,-0.99)--(-0.11,-1.2);
\draw[draw=\drawcolor](0.11,-0.99)--(0.11,-1.2);
\end{scope}
    }
  }
}
\tikzset{mycylinder/.style={cylinder, shape border rotate=90, aspect=1.3, draw, fill=white,
minimum width=25mm,minimum height=11mm,line width=\Linewidth,node distance=-0.15},
pics/data/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=STREAMING,scale=\scalefac, every node/.append style={transform shape}]
\node[mycylinder,fill=\filllcolor!50] (A) {};
\node[mycylinder, above=of A,fill=\filllcolor!30] (B) {};
\node[mycylinder, above=of B,fill=\filllcolor!10] (C) {};
\fill[\filllcolor!50!black]($(C.west)!0.12!(C.east)$)circle(3pt);
\fill[\filllcolor!50!black]($(B.west)!0.12!(B.east)$)circle(3pt);
\fill[\filllcolor!50!black]($(A.west)!0.12!(A.east)$)circle(3pt);
 \end{scope}
     }
  }
}
\tikzset{%
 pics/dataFolderS/.style = {
        code = {
        \pgfkeys{/channel/.cd, #1}
\begin{scope}[local bounding box=DATAFOLDER,scale=\scalefac, every node/.append style={transform shape},
LinE/.style={\filllcirclecolor,line width=2.0pt,
{{Triangle[width=1.5*3pt,length=2.0*3pt]}-},shorten <=1pt,shorten >=1pt},
ELin/.style={\filllcirclecolor,line width=2.0pt,
{-{Triangle[width=1.5*3pt,length=2.0*3pt]}},shorten <=1pt,shorten >=1pt},]
\draw[line width=\Linewidth,draw=\drawcolor,rounded corners=0.1pt,fill=\filllcolor!20] (0,0) -- (-0.20,2.45)coordinate(\picname-GL)--
(0.4,2.45)to[out=360,in=180](0.9,2.1)-- (2.5,2.1)--(2.5,0)--cycle ;
\draw[line width=\Linewidth,draw=\drawcolor,rounded corners=2pt,
fill=\filllcolor!50] (0,0)coordinate(\picname-DL) -- (2.8,0)
coordinate(\picname-DD)-- (3,1.8) -- (0.2,1.8) -- cycle;
\draw[LinE](\picname-DD)--++(330:2.0);
\draw[LinE](-0.3,1.8)--++(150:2.0);
\draw[LinE](3,1.8)--++(30:2.0);
\draw[LinE](\picname-DL)--++(210:2.0);
 \end{scope}
     }
  }
}
\pgfkeys{
  /channel/.cd,
   Depth/.store in=\Depth,
  Height/.store in=\Height,
  Width/.store in=\Width,
  filllcirclecolor/.store in=\filllcirclecolor,
  filllcolor/.store in=\filllcolor,
  drawcolor/.store in=\drawcolor,
  drawcircle/.store in=\drawcircle,
  scalefac/.store in=\scalefac,
  Linewidth/.store in=\Linewidth,
  picname/.store in=\picname,
  tiecolor/.store in=\tiecolor,
  bodycolor/.store in=\bodycolor,
  stetcolor/.store in=\stetcolor,
  tiecolor=red,      % default tie color
  bodycolor=blue!30,  % default body color
  stetcolor=green,  % default stet color
  filllcolor=BrownLine,
  filllcirclecolor=violet!20,
  drawcolor=black,
  drawcircle=violet,
  scalefac=1,
  Linewidth=0.5pt,
  Depth=0.2,
  Height=0.5,
  Width=0.25,
  picname=C
}
%Unlabeled Pool
\node[Box](B1){};
\node[BoxD](BB1)at(B1.south){Raw Data};
\coordinate(S1)at($(B1.north)!0.5!(BB1.north)$);
\pic[shift={(0,-0.22)}] at  (S1){dataS={scalefac=0.6,picname=1,Linewidth=1.0pt,
 filllcolor=red!90!black!40!,drawcolor=black,filllcirclecolor=cyan}};
 % Data Selection
 \node[Box1,right=of B1](B2){};
\node[BoxD1](BB2)at(B2.south){Data Selection};
\coordinate(S2)at($(B2.north)!0.5!(BB2.north)$);
\pic[shift={(0,0.05)}] at  (S2){funnel={scalefac=0.53,picname=1,Linewidth=0.5pt,
 filllcolor=BrownL,drawcolor=black,filllcirclecolor=orange!70!yellow!80}};

 % Curated Data
 \node[Box,right=of B2](B3){};
\node[BoxD](BB3)at(B3.south){Curated Data};
\coordinate(S3)at($(B3.north)!0.5!(BB3.north)$);
\pic[shift={(0,-0.45)}] at  (S3){data={scalefac=0.4,picname=1,filllcolor=magenta, Linewidth=0.7pt}};
% Training
 \node[Box1,right=of B3](B4){};
\node[BoxD1](BB4)at(B4.south){Training};
\coordinate(S4)at($(B4.north)!0.5!(BB4.north)$);
\pic[shift={(0,-0.04)}] at  (S4){algorithm={scalefac=0.9,picname=1,
drawcolor=black,filllcolor=cyan!50!, Linewidth=0.75pt,filllcirclecolor=green}};

 % Model
 \node[Box,below right=0 and 1 of B4](B5){};
\node[BoxD](BB5)at(B5.south){Model};
\coordinate(S5)at($(B5.north)!0.5!(BB5.north)$);
%AI
\pic[shift={(0,0)}] at  (S5){llm={scalefac=0.6,picname=1,drawcolor=GreenD,
filllcolor=GreenD!20!, Linewidth=0.7pt,filllcirclecolor=red}};
%brain
\pic[shift={(0.09,-0.13)}] at  (C1){brain={scalefac=0.6,picname=2,filllcolor=orange!30!,
filllcirclecolor=cyan!55!black!60, Linewidth=0.5pt}};
% Compression
 \node[Box1,below left=0 and 1 of B5](B6){};
\node[BoxD1](BB6)at(B6.south){Compression};
\coordinate(S6)at($(B6.north)!0.5!(BB6.north)$);
\pic[shift={(-0.32,-0.27)}] at  (S6){dataFolderS={scalefac=0.29,picname=1,Linewidth=0.5pt,
 filllcolor=BrownLine,drawcolor=BrownLine,filllcirclecolor=RedLine}};
% Compact Model
 \node[Box,left=of B6](B7){};
\node[BoxD](BB7)at(B7.south){Compact Model};
\coordinate(S7)at($(B7.north)!0.5!(BB7.north)$);
\pic[shift={(0,0)}] at  (S7){llm1={scalefac=0.38,drawcolor=black,filllcolor=orange!50!, Linewidth=1.0pt,filllcirclecolor=violet!70!}};
% Hardware
 \node[Box1, left=of B7](B8){};
\node[BoxD1](BB8)at(B8.south){Hardware};
\coordinate(S8)at($(B8.north)!0.5!(BB8.north)$);
\pic[shift={(0,0)}] at  (S8){cpu={scalefac=0.35,picname=1,filllcolor=BlueLine, Linewidth=0.7pt}};
% Deployed System
 \node[Box, left=of B8](B9){};
\node[BoxD](BB9)at(B9.south){Deployed System};
\coordinate(S9)at($(B9.north)!0.5!(BB9.north)$);
\pic[shift={(0,0.1)}] at  (S9){rocket={scalefac=0.6,picname=1,
drawcolor=black,filllcolor=cyan!10!, Linewidth=0.75pt,filllcirclecolor=red}};
%arrows
\foreach \i in {1,2,3,6,7,8}{
\pgfmathtruncatemacro{\x}{\i + 1} %
\draw[LineA](B\i)--(B\x);
}
\draw[LineA](B4)-|(B5);
\draw[LineA](B5)|-(B6);
 \end{tikzpicture}

Figure 12: The Optimization Stack: Data artifacts flow through selection and training before the resulting model enters compression and deployment. Upstream changes propagate quality and cost effects downstream, but reducing training data does not automatically simplify the model or deployment hardware.

The pipeline in figure 12 reveals why data selection occupies a strategic position at the head of the optimization stack. With the training schedule held fixed, reducing the dataset by 50 percent can halve sample-level training work. It does not by itself shrink the trained model, simplify compression, or relax serving hardware requirements. Each downstream stage inherits any efficiency gains or quality losses from upstream decisions, and poor selection can force compensation through longer training or less aggressive model simplification.

Quantifying this multiplicative effect requires determining whether a 50 percent dataset reduction delivers 50 percent compute savings, or whether it has inadvertently degraded model quality in ways that surface only in production. Answering this question requires a rigorous measurement framework: metrics that capture both the efficiency gains and the quality costs of data selection decisions.

Self-Check: Question

In the D·A·M optimization stack (Data Selection, Algorithm/Model Compression, Machine Hardware Optimization), an ML team achieves a $2\times$ reduction in dataset size via coreset pruning, a $2\times$ reduction in operations per sample via model pruning/quantization, and a $2\times$ increase in hardware arithmetic throughput via kernel optimization. What is the total combined speedup factor for training?
1. An $8\times$ total speedup, because optimizations across distinct layers of the ML systems stack compound multiplicatively ($2 \times 2 \times 2 = 8$)
2. A $6\times$ total speedup, because speedup factors add linearly across layers ($2 + 2 + 2 = 6$)
3. A $2\times$ total speedup, because the lowest-layer optimization bottleneck dominates all others (Amdahl’s law min-factor)
4. A $4\times$ total speedup, because data selection cancels out model compression gains
Why are upstream data selection (Workload layer) and downstream model compression (Algorithm layer) fundamentally complementary rather than interchangeable techniques in system design?
1. Model compression can only be applied to computer vision models, whereas data selection is restricted to NLP
2. Data selection eliminates backward passes entirely, whereas model compression eliminates forward passes
3. Data selection optimizes inference latency on edge devices, whereas model compression only affects training time
4. Data selection reduces the total number of training samples processed ($N_{\text{samples}}$), whereas model compression reduces the compute and memory cost per individual sample forward/backward pass ($O_{\text{sample}}$)
A training pipeline aggressively reduces dataset size with a $10\times$ coreset. However, the engineering team observes that the end-to-end training job speedup is only $2\times$ instead of the expected $10\times$. Using systems principles, diagnose the likely bottleneck shift.
Explain how data selection operates upstream of all algorithm- and hardware-level optimizations in the D·A·M optimization stack.

See Answers →

Measurement Framework

The cross-layer stack makes a measurement framework unavoidable: every data selection technique claims to improve efficiency, but only rigorous measurement separates real savings from shifted costs or hidden quality loss. The core metrics in section 1.11.1 tie sample reduction to accuracy, cost, and deployment coverage so that a smaller dataset is judged by what it preserves, rather than by what it removes alone.

Core metrics

The core metrics connect sample reduction to model quality instead of reporting accuracy alone. They measure learning per sample, performance-per-data, and compression at a target accuracy.

Performance-per-data

The most direct metric, performance-per-data (PPD), measures accuracy gain per sample: \[ \text{PPD}(n) = \frac{\text{Accuracy}(n) - \text{Accuracy}(0)}{n} \] where $n$ is the number of training samples. A higher PPD indicates greater average improvement per sample over the stated interval. Its shape must be measured because diminishing returns and their onset are workload-dependent.

Area under the learning curve

Rather than comparing at a single point, the area under the learning curve (AULC) integrates performance across all dataset sizes: \[ \text{AULC} = \int_0^D \text{Accuracy}(n) \, dn \] where $n$ is the dataset size and $D$ is the total dataset size.

For the same metric and integration range, a higher AULC means the strategy achieves stronger performance with fewer samples. Comparisons must use the same upper limit $D$ or normalize the integral.

Data compression ratio

For coreset methods, the data compression ratio (DCR) measures how much data reduction is achieved at a target accuracy: \[ \text{DCR} = \frac{D_{\text{full}}}{D_{\text{coreset}}} \text{ at } \text{Accuracy}_{\text{target}} \]

A DCR of 5$\times$ means the coreset achieves target accuracy with 20 percent of the data.

The compute-optimal frontier

While the core metrics in section 1.11.1 measure individual techniques, a higher-level diagnostic is also needed to test whether the overall training strategy is data-limited or compute-limited. Neural scaling-law experiments (Kaplan et al. 2020; Hoffmann et al. 2022) fit power-law relationships over particular model families, datasets, and compute ranges. Controlled sweeps can use those fitted relationships to compare model and data allocations, but no single ratio diagnoses every training run.

³³ Chinchilla: Chinchilla is a 70-billion-parameter language model trained on 1.4 trillion tokens. In the experiments reported by Hoffmann et al. (2022), it outperformed the larger GPT-3 on most evaluated tasks under a similar training-compute budget. The fitted compute-optimal allocation scaled parameters and tokens at roughly equal rates within the study’s experimental regime; it is not a universal prescription for every architecture, dataset, or training recipe.

Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, et al. 2022. “Training Compute-Optimal Large Language Models.” Advances in Neural Information Processing Systems (NeurIPS) 35: 30016–30. https://doi.org/10.52202/068431-2176.

The Chinchilla study³³ (Hoffmann et al. 2022) fit a compute-optimal balance between model size and training data within its experimental regime. Allocations on either side of that fitted balance used the study’s compute budget less effectively.

The optimal balance defines a compute-optimal frontier: the best achievable performance at each compute budget when data and model size are properly balanced. Figure 13 sketches this diagnostic as a conceptual frontier rather than a fitted Chinchilla result.

Figure 13: **Illustrative Compute-Optimal Frontier**: The green curve is a conceptual frontier over normalized compute, not a fitted result from Chinchilla. Points below it illustrate allocations that a controlled model-size and token-count sweep might identify as data- or compute-limited.

Against a frontier fitted from controlled sweeps, a point’s location can suggest the next experiment. A data-limited allocation motivates tests of corpus quality, coverage, or selection; a compute-limited allocation motivates tests of effective throughput, duration, or model size. The conceptual points in figure 13 do not diagnose a real run by themselves, and points near a fitted frontier remain specific to the model family, data, objective, and compute range used to estimate it.

Interpreting the Chinchilla result

Within the fitted Chinchilla regime, compute-optimal model parameters and training tokens grew at roughly equal rates.³⁴ Combining that empirical fit with the dense-transformer compute approximation, we derive $D_{\text{opt}} \propto \sqrt{C}$, so doubling compute corresponds to about $\sqrt{2} - 1$, or 41.4 percent, more tokens under those assumptions. The commonly quoted 20 tokens per parameter is a useful reference point from that study, not a universal optimum.

³⁴ Tokens per parameter: The ratio $D/P$ compares training tokens with model parameters; GPT-3 used approximately 1.7 tokens per parameter (Brown et al. 2020), while Llama 2 70B used approximately 28.6 (Touvron et al. 2023). These descriptive ratios do not establish whether either run is compute-optimal under a different architecture, corpus, objective, or hardware budget. Controlled model-size and token-count sweeps are required for that diagnosis.

Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. 2020. “Language Models Are Few-Shot Learners.” Advances in Neural Information Processing Systems 33: 1877–901. https://doi.org/10.48550/arxiv.2005.14165.

Touvron, Hugo, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, et al. 2023. “Llama 2: Open Foundation and Fine-Tuned Chat Models.” arXiv Preprint arXiv:2307.09288.

Applying the diagnostic

If a training run underperforms expectations, extending one run can reveal whether additional optimization steps still help, but a plateau does not identify data starvation by itself. Learning-rate schedules, optimization limits, model capacity, and data quality can all produce a plateau. A reliable diagnosis compares controlled runs that vary model size, token count, and compute allocation while holding the evaluation protocol fixed.

In figure 14 the two curves diverge: a data-efficient selection strategy (blue) reaches the performance plateau with fewer samples than random sampling (gray). The horizontal gap represents that sample reduction and can translate into compute savings when the per-sample work and training schedule remain fixed; the vertical gap marked by the red arrow represents the performance gained at a fixed dataset size.

Figure 14: **Illustrative Diminishing Returns of Data**: The synthetic data-efficient curve (blue) reaches the same plateau with fewer samples than random sampling (gray). The vertical red arrow marks the modeled performance gap at a fixed dataset size; measured curves need not have this shape.

For practitioners, the metrics in this section answer a practical question: at what point to stop collecting data and start curating it, and when adding more samples wastes compute rather than improving accuracy. Knowing that a strategy is efficient, however, is not the same as confirming that a curated dataset preserved model quality.

Data selection techniques implicitly rank sample value, and validating that a curated dataset preserves model quality requires systematic benchmarking across three dimensions. Coverage metrics validate that coreset selection preserved representation across classes and demographic groups. Distribution alignment metrics (such as KL divergence and population stability index, which Measuring drift (divergence) defines) detect whether the curated training set drifted from the deployment distribution. Label quality metrics (inter-annotator agreement, confident learning) validate that active learning did not introduce systematic labeling errors. A 50 percent dataset reduction is only valuable if benchmarking confirms the model maintains target accuracy, calibration, and robustness. Benchmarking later generalizes these ideas into broader model-and-data evaluation protocols; here, the point is narrower: a curated dataset must be validated against the task distribution rather than against its reduction ratio alone.

The validation target can itself be unreliable. A widely cited replication study shows how benchmark accuracy can overstate what a model has learned, which is why the evaluation set deserves the same scrutiny as the curated training set.

Example 1.7: The benchmark replication gap

Scenario: Classifiers evaluated on newly collected test sets experienced 3 to 15 percentage-point accuracy drops on CIFAR-10 and 11 to 14 points on ImageNet (Recht et al. 2019).

Diagnosis: The replicated collection produced slightly harder images than the original test sets. The study found that benchmark adaptivity did not explain the gap, even though repeated use of a fixed test set remains a general evaluation risk.

Systems lesson: Evaluation-set construction affects measured generalization. Data-selection pipelines should validate curated subsets on independently collected and deployment-relevant test sets rather than trusting one long-reused benchmark.

Recht, Benjamin, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. 2019. “Do ImageNet Classifiers Generalize to ImageNet?” Proceedings of the 36th International Conference on Machine Learning (ICML), 5389–400.

The lesson carries directly to data selection: an efficiency gain measured against a single benchmark can evaporate on newly collected data, so a curated subset must hold up across multiple held-out distributions before its reduction ratio is trusted.

Lighthouse 1.3: Lighthouse data selection

Data selection principles apply to all five lighthouse models, though the dominant constraint determines the technique. Table 21 maps each workload to a priority: for example, compute-bound ResNet-50 benefits from coreset selection, memory-bound GPT-2/Llama from deduplication, and the tiny KWS model from augmentation and synthesis.

Table 21: Data Selection Priorities by Lighthouse: Primary bottleneck and candidate data-selection priority for each lighthouse workload. The pairing suggests a hypothesis to test; the dominant resource constraint alone does not guarantee that a technique pays off.

Lighthouse	Primary Bottleneck	Data Selection Priority
ResNet-50	Compute	Coreset selection directly reduces training FLOPs
GPT-2/Llama	Memory bandwidth	Deduplication reduces corpus size; curriculum learning improves token efficiency
MobileNetV2	Latency/Power	Validate augmentation against the target model and deployment invariances
DLRM	Memory capacity	Interaction deduplication and embedding pruning reduce table size
Keyword Spotting	Extreme constraints	Augmentation and synthesis create datasets from minimal seeds

Across these models, data selection is not a single technique but a systems optimization tailored to whichever resource is most constrained.

The measurement tools and lighthouse examples demonstrate what data selection can achieve when applied correctly. The techniques, however, involve counterintuitive trade-offs, and practitioners frequently fall into predictable traps.

Self-Check: Question

Match the data-selection efficiency metric with its precise definition: A team wants to measure the ratio of full dataset size to selected subset size ($\text{DCR} = |D| / |S|$), and the ratio of final accuracy achieved on the subset versus the full dataset ($\text{ARR} = \text{Acc}(S) / \text{Acc}(D)$). What do DCR and ARR stand for?
1. Data Curation Rate and Accuracy Reduction Ratio
2. Data Compression Ratio and Accuracy Retention Ratio
3. Dynamic Checkpoint Rate and Active Retention Rate
4. Data Convergence Ratio and Amortized Risk Ratio
An ML systems diagnostic plot maps normalized training compute (FLOPs) on the horizontal log-axis against model performance on the vertical axis. A training run with 10M parameters sits significantly below the green compute-optimal frontier, and increasing token count yields no accuracy improvement while scaling model parameters to 100M immediately restores optimal frontier scaling. What was the diagnosis of the original operating point?
1. Data-starved regime
2. I/O bandwidth-saturated regime
3. Compute-starved (capacity-limited) regime
4. Over-echoing regime
Why is Area Under the Learning Curve (AULC) a more informative metric than single-point final validation accuracy when evaluating dynamic data selection and curriculum learning algorithms?
True or False: If two training runs (Run A on a raw dataset and Run B on a coreset) reach the exact same validation loss plateau, they must have processed identical amounts of informative tokens.
The Pareto frontier that defines the maximum achievable model accuracy or minimum loss for every given training compute budget (FLOPs) under balanced parameter and token allocation is known as the compute-____ frontier.

See Answers →

Fallacies and Pitfalls

Data selection involves counterintuitive diminishing returns that contradict the “more is better” intuition from traditional machine learning. The following errors fall into three groups: conceptual fallacies about what data selection can achieve, implementation pitfalls that arise when correct strategies meet engineering realities, and transfer errors that occur when benchmark results are applied uncritically to new domains.

Fallacy: Data is the new oil, so more is always better.

More data does not imply proportional accuracy gains because learning curves commonly show diminishing returns. The numerical comparison in this callout is illustrative rather than measured: it assumes a move from 1M to 10M samples for only 4 percentage points of accuracy gain. Table 1 likewise uses scenario growth rates. Teams should measure the learning curve and marginal compute cost for their own data rather than infer either from these example values.

Pitfall: Replacing real-data validation with synthetic-only training data.

Engineers assume generative models can replace data collection with inexhaustible generated examples at marginal cost. Synthetic-only training can fail through two different mechanisms. First, section 1.5.3 and table 9 show the domain-gap problem: generated data can diverge from the real deployment distribution, causing the learned decision boundary in figure 8 to misclassify real-world inputs. Second, recursive training on model-generated data can cause model collapse (Shumailov et al. 2024): in this illustrative scenario, accuracy degrades from 95 percent to 78 percent after five generations, a 17 percentage-point drop. The illustrative 50–80 percent synthetic range is workload-dependent and must be validated against real deployment data.

Recursive synthetic-data training degrades accuracy across generations.

Fallacy: Data selection is merely data cleaning.

Engineers can conflate data quality, which includes removing errors, with data value under a particular task and training procedure. Figure 4 illustrates one uncertainty-based strategy, while section 1.2.2 also covers geometric objectives. The 1.8× ICR difference is a worked scenario rather than a general EL2N or GraNd result. Cleaning addresses data defects; selection evaluates the learning value and cost of retained examples.

Pitfall: Treating data selection as a budget-only tactic.

Practitioners may view data selection as relevant only for TinyML or budget-limited teams. It applies wherever high-quality examples, not raw volume, constrain learning. A 10 percent efficiency gain on a $100M training run saves $10M under the scenario assumptions. The data wall (figure 1) can become especially visible at large scale, when compute capacity outruns accessible, legally usable, target-relevant data. Section 1.8.4 shows why reuse can make selection infrastructure economical even for well-funded teams.

Conceptual misunderstandings often lead to flawed strategies. Equally damaging are the implementation pitfalls that arise when correct strategies meet messy engineering realities.

Fallacy: Selection overhead is too small to change training economics.

Take a scenario in which the full run takes 8 hours and training on the coreset takes 2-hour, so selection has to buy back its cost out of the hours saved. A sophisticated coreset algorithm requiring 10 hours to do that selection spends 5× the training run it enables and more than the full run it was meant to avoid, yielding negative ROI. Equation 2 defines the boundary. Use lightweight proxy models or cached embeddings when they preserve selection quality. In the illustrative comparison, proxy-based EL2N scoring completes in 30 minutes, or 6.2 percent of that full-run baseline, and satisfies the inequality.

Pitfall: Pruning rare classes into oblivion.

Aggressive selection without group constraints can remove rare classes because an average-loss objective gives them little weight. In a 1M dataset with 0.1 percent rare-class samples (1,000 examples), an unstratified 10 percent subset retains only 100 such examples in expectation, below the scenario’s 150-sample target. Importance sampling can reweight rare samples, but large weights can inflate gradient variance. Section 1.2.2 therefore recommends minimum representation by class or deployment-relevant slice before optimizing the remaining selection budget.

Fallacy: Deduplicating training data is enough to make evaluation reliable.

Some standard language-modeling datasets have measurable train-test overlap; a deduplication study reports overlap affecting over 4 percent of validation examples in evaluated datasets (Lee et al. 2022). Deduplicating only training data is therefore insufficient: evaluation sets must also be checked against the training corpus. This extends the threshold-calibration principle in section 1.2.3 across dataset splits. Teams should validate the effect because near-duplicate thresholds and domain distributions matter.

Lee, Katherine, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. “Deduplicating Training Data Makes Language Models Better.” Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 8424–45. https://doi.org/10.18653/v1/2022.acl-long.577.

Pitfall: Active learning without considering annotation latency.

Active-learning analyses often abstract away oracle response time. In real annotation workflows, turnaround can affect the usefulness of a query batch. The 14-day latency and batch sizes in this callout are illustrative; the correct batch size depends on measured annotation capacity, model-update cadence, and diversity requirements. Active learning ROI therefore depends on both label cost and turnaround time.

Fallacy: If a technique works on ImageNet, it will work on my dataset.

Data-selection effectiveness depends on dataset redundancy, representation, model, and target metric. Results from CIFAR-10 or ImageNet do not establish a safe pruning ratio for medical, satellite, or other domain-specific data, and those datasets cannot be assumed to have near-zero redundancy either. Start with a measured baseline, vary the retained fraction, and validate overall and slice-level quality before aggressive reduction.

Pitfall: Optimizing data selection metrics instead of deployment metrics.

The illustrative failure scenario uses a 10 percent coreset that performs well on majority classes but poorly on rare subgroups. Section 1.11.1 defines efficiency metrics, while section 1.11 requires stratified evaluation. Selection must preserve demographic groups, rare classes, and deployment failure modes even when doing so reduces average PPD.

Self-Check: Question

A research paper reports that an EL2N coreset selection method successfully pruned $50\%$ of CIFAR-10 with $0\%$ loss in accuracy. A medical imaging team applies the exact same $50\%$ pruning ratio to a rare tumor detection dataset and suffers a disastrous $28\%$ drop in recall. What fallacy explains this failure?
1. The team failed to use GPU acceleration during the inference pass
2. The team used float32 precision instead of bfloat16 mixed precision
3. CIFAR-10 contains more total classes than medical imaging datasets
4. Assuming that benchmark coreset pruning ratios transfer directly to specialized, highly imbalanced production domains with rare failure modes
A team implements an elaborate multi-stage active learning pipeline that reduces training dataset size by $40\%$. However, the continuous clustering, proxy scoring, and cross-worker all-gather synchronization take 3 times longer than the GPU time saved during training. Which pitfall does this represent?
1. Violating the Selection Inequality by incurring selection overheads that exceed downstream training savings ($T_{\text{selection}} > \Delta T_{\text{train}}$)
2. Encountering model collapse due to recursive generator loops
3. Failing to implement 4 KB small-read alignment on host NVMe storage
4. Violating the smoothness assumption in semi-supervised consistency regularization
Why is evaluating a data selection strategy solely on aggregate validation accuracy a dangerous pitfall when dealing with imbalanced datasets?
True or False: Because modern high-capacity generative models produce photorealistic images and fluent text, a model trained on $100\%$ recursively generated synthetic data will never suffer from performance degradation.

See Answers →

Summary

These fallacies and pitfalls arise when practitioners treat data selection as a purely algorithmic exercise divorced from the systems context in which it operates. Smaller, curated datasets sometimes outperform massive ones because data selection is a systems problem rather than a purely statistical one. It addresses the first question in the optimization ordering by reducing work before it begins. Traditional machine learning frames the problem as the number of samples needed to achieve target accuracy; the systems perspective frames it as minimizing total cost across the entire pipeline.

This systems framing becomes actionable through the ICR metric, the selection inequality, and the cost modeling framework. The goal is to minimize total cost across compute, storage, labeling, energy, and time rather than maximize accuracy alone.

The three-stage optimization pipeline addresses different phases of this cost equation: static pruning removes redundancy before training through coreset selection and deduplication, dynamic selection prioritizes informative examples during training through curriculum and active learning, and synthetic generation creates data where none exists through augmentation, simulation, and distillation. Together, these strategies address the “data wall,” the structural asymmetry between rapidly growing compute capacity and slowly growing high-quality data.

Self-supervised learning occupies one end of the data-selection spectrum: pretraining shifts much of the learning signal into a reusable model, so downstream tasks can often use fewer task-specific labels. The economics improve when the pretraining cost is amortized across enough tasks.

Translating these techniques into production requires systems engineering: the selection inequality in equation 2 gates every technique, proxy models and shard-based data loaders reconcile selection algorithms with storage hardware, and data echoing can recover idle GPU cycles when input service is the bottleneck. The cost modeling framework (total data cost, ROI analysis, and break-even thresholds) provides the quantitative tools to evaluate which techniques merit investment for a given workload, while core metrics (PPD, AULC, DCR) and a fitted compute-optimal frontier help practitioners test whether training is data-limited or compute-limited.

Key Takeaways: Curate, do not accumulate

Selection optimizes system cost: The goal is reduced total cost across the entire pipeline (compute, storage, labeling, energy), rather than “fewer samples for same accuracy” alone. The information-compute ratio quantifies learning gained per FLOP spent. Its time benefit matches a throughput improvement only under the corresponding compute-bound assumptions.
Start with deduplication: Exact deduplication is often the lowest-risk first technique because it removes repeated bytes without a learned scoring pass. Apply near-duplicate and semantic deduplication only after validating thresholds against downstream quality metrics.
The selection inequality gates every technique: $T_{\text{selection}} + T_{\text{train}}(\text{subset}) < T_{\text{train}}(\text{full})$. Selection overhead must remain below the training time saved by the subset. Proxy models and cached embeddings can keep $T_{\text{selection}}$ low; expensive selection algorithms can consume all the savings they promise.
Dynamic selection adapts the data diet as the model learns: Curriculum learning changes presentation order, while active learning changes which samples receive labels. Either can improve efficiency when its scoring rule, coverage, and refresh overhead are validated.
Self-supervised pretraining can amortize adaptation cost: Pretraining once and fine-tuning many times spreads pretraining cost across downstream tasks. The worked scenario assumes a 100$\times$ label reduction and a 20$\times$ marginal-compute reduction; realized savings depend on transfer quality and reuse.
Synthetic data is a supplement, not a replacement: The 50–80 percent synthetic mixture is illustrative and workload-dependent, not a universal optimum. Pure synthetic training risks model collapse and domain-gap degradation.
Validated workload reduction compounds downstream: When selection safely eliminates examples or tokens without adding compensating steps, that work never reaches later model or machine optimizations. Dynamic scoring and synthesis add their own costs, so the combined saving must be measured rather than assumed.

The techniques explored throughout this chapter (deduplication, coreset selection, curriculum learning, active learning, and synthetic generation) provide a toolkit for addressing the data wall. When validated for the workload, they can reduce labeling, iteration, storage, or training costs while preserving the coverage needed for generalization.

The answer to the chapter’s opening question is that examples can differ substantially in learning value. Some benchmark datasets can shed half their examples with no measured loss under a validated selection method, while more aggressive pruning usually trades additional savings for some quality loss. Selection is the discipline of distinguishing informative, representative examples from redundant or harmful ones without removing rare cases that matter at deployment. Through D·A·M, this is data-algorithm co-design working upstream of everything else, because a sample excluded safely from training is a cost no downstream optimization has to absorb.

What’s Next: From data to algorithms

Data selection changes what the system learns from; even the best data cannot make an inefficient model run fast on constrained hardware. Model Compression shifts the focus to how the system represents what it learns, applying pruning, quantization, and knowledge distillation to reduce the computational cost of the model artifact itself.

Self-Check: Question

Which summary statement best captures the central systems principle of data selection established throughout this chapter?
1. Data selection is an offline heuristic that only applies to small academic image classification benchmarks
2. Data selection is the highest-leverage input optimization layer because it eliminates FLOPs, memory traffic, and communication before model or hardware execution begins
3. Data selection replaces all algorithm-level model compression and hardware acceleration optimizations
4. Data selection is strictly bounded by the requirement that datasets must grow linearly with GPU cluster node counts
In one integrated explanation, relate the Information-Compute Ratio (ICR), the Selection Inequality, and the three-stage data selection pipeline.
How does workload-level data selection interact with the broader D·A·M optimization stack to maximize end-to-end training efficiency?

See Answers →

Self-Check Answers

Self-Check: Answer

In an ML infrastructure scaling scenario where available GPU compute grows by approximately $10\times$ every 3 years while high-quality web data grows by only $2\times$ every 5 years, what primary systems regime emerges, and what is the appropriate systems response?
1. A compute-rich, data-constrained Data Wall regime where intelligent data selection and curation must maximize the learning signal extracted per token
2. A memory bandwidth-bound regime where model parallel sharding must replace data parallelism across all training clusters
3. An I/O ingestion bottleneck where disk read bandwidth must be quadrupled to keep GPUs saturated
4. A compute-starved regime where synthetic data generation should be eliminated to avoid wasting accelerator cycles
Answer: The correct answer is A. Compute throughput growing faster than accessible high-quality human data creates a compute-to-data imbalance known as the Data Wall. In this regime, additional raw compute yields diminishing returns on uncurated corpora, making data selection essential to maximize the Information-Compute Ratio (ICR). Shifting model parallelism addresses parameter memory limits rather than data scarcity, scaling disk bandwidth solves I/O stalls rather than token information quality, and eliminating synthetic generation ignores a key strategy for expanding scarce data pools.

Learning Objective: Analyze the systems implications of the compute-to-data scaling asymmetry and diagnose the Data Wall regime.
Under the illustrative analytical model where dataset information content scales logarithmically as $I(D) \propto \log D$ and training compute scales linearly with per-sample operations $C(D) = O_{\text{sample}} \cdot D$, how does the marginal Information-Compute Ratio $\text{ICR}(D)$ scale with dataset size $D$?
1. It remains constant at $\mathcal{O}(1)$ because additional compute scales proportionally with dataset size
2. It decays as $\mathcal{O}(1 / (O_{\text{sample}} \cdot D))$, turning additional unselected data into a data tax that consumes compute with minimal learning progress
3. It grows logarithmically as $\mathcal{O}(\log D / O_{\text{sample}})$ due to power-law parameter scaling
4. It decays exponentially as $\mathcal{O}(\exp(-D))$ once the training corpus exceeds accelerator memory capacity
Answer: The correct answer is B. Taking the derivative of dataset information content D$ with respect to dataset size $ gives /D$, and differentiating compute cost = O_{ ext{sample}} \cdot D$ gives { ext{sample}}$. The marginal ICR is their ratio, $ ext{ICR} pprox 1 / (O_{ ext{sample}} D)$. As $ grows large, ICR drops toward zero, meaning marginal tokens consume compute without providing new learning signal—acting as a data tax. Constant scaling ignores the diminishing returns of information content, logarithmic growth incorrectly treats marginal signal as expanding, and exponential decay overstates the rate of informational saturation.

Learning Objective: Calculate the scaling behavior of the Information-Compute Ratio (ICR) under logarithmic information gain.
Explain why a dataset where $100\%$ of the sample labels are verifiably correct can still exhibit a very low Information-Compute Ratio (ICR).

Answer: Label correctness measures ground-truth accuracy, whereas ICR measures the marginal learning signal contributed per unit of compute. A dataset of completely correct examples will have near-zero ICR if the samples are redundant, easily classified by the current model state, or already mastered, because processing them consumes forward and backward FLOPs without updating model parameters meaningfully.

Learning Objective: Compare data correctness with data value in the Information-Compute Ratio framework.
True or False: Because deep learning models benefit from large-scale training, collecting and training on twice as much raw, deduplicated web data will always double the total information learned by the model.

Answer: False. Marginal information gain follows diminishing returns (e.g., $I(D) \propto \log D$) rather than linear scaling. Doubling raw data volume increases compute linearly while marginal information content per sample decays, eventually hitting the ICR frontier where redundant or uninformative tokens act as a compute tax.

Learning Objective: Evaluate the misconception that data information value scales linearly with raw corpus volume.
Order the three primary stages of the high-efficiency data selection pipeline according to their execution in an ML system lifecycle: (1) Dynamic Selection, (2) Static Pruning, (3) Synthetic Data Generation.

Answer: The correct order is: (2) Static Pruning, (1) Dynamic Selection, (3) Synthetic Data Generation. Static pruning is performed offline prior to training (filtering and deduplication), dynamic selection adapts the data stream during training (curriculum and active learning), and synthetic data generation produces targeted samples on demand to fill remaining domain gaps.

Learning Objective: Explain the sequential structure and purpose of the three-stage data selection pipeline.

← Back to Questions

Self-Check: Answer

A team wants to select a coreset of size $K$ from a dataset of size $D$ before training a large production model. They need a method that accounts for model uncertainty near decision boundaries but cannot afford a full target-model training run for scoring. Which method and systems trade-off best fits their requirement?
1. $k$-Center clustering on raw pixel inputs, because it guarantees zero-cost label-aware boundary identification
2. Forgetting Events scoring on the full production model, because it requires no proxy architecture and computes in $\mathcal{O}(1)$ time
3. EL2N (Error L2-Norm) scoring using an inexpensive proxy model trained for a few epochs, leveraging proxy score transferability
4. Herding on Gaussian-distributed features, because it completely avoids computing feature representations
Answer: The correct answer is C. EL2N calculates the error L2-norm early in training ($\mathcal{O}(\text{epochs} \times D)$). Crucially, EL2N scores computed from an inexpensive, small proxy model transfer reliably to large target models, enabling boundary-focused coreset selection without paying the full target model training cost. The $k$-Center approach operates on geometry and ignores label information, Forgetting Events requires an expensive full training run on the model, and Herding still requires feature embeddings and assumes specific moment distributions.

Learning Objective: Compare coreset selection algorithms across compute complexity, training dependencies, and scoring proxy trade-offs.
Why is Locality-Sensitive Hashing (LSH) with MinHash preferred over exhaustive pairwise Jaccard similarity comparison for large-scale text deduplication?
1. MinHash LSH guarantees $100\%$ precision in detecting semantic paraphrases across different natural languages
2. MinHash eliminates the need to tokenize or shingle input documents before hashing
3. Exhaustive pairwise Jaccard comparison requires training a deep neural network, whereas LSH is purely rule-based
4. Exhaustive pairwise comparison requires $\mathcal{O}(D^2)$ document comparisons, whereas MinHash LSH hashes compact signatures into sublinear candidate collision buckets
Answer: The correct answer is D. Comparing all pairs in a dataset of $D$ documents requires $\mathcal{O}(D^2)$ comparisons, which is computationally intractable for web-scale datasets ($D \ge 10^7$). MinHash compresses documents into fixed-size sketches where collision probability equals Jaccard similarity, and LSH bands these sketches into hash buckets so only candidate pairs colliding in a bucket are compared. MinHash LSH does not detect cross-lingual semantic paraphrases, still requires shingling/tokenization, and pairwise Jaccard comparison is an exact set operation rather than a deep learning method.

Learning Objective: Analyze the computational scaling benefits of MinHash and Locality-Sensitive Hashing for web-scale deduplication.
In recommendation systems like DLRM where embedding tables consume terabytes of memory, how does interaction deduplication differ in systems impact from cold embedding pruning?

Answer: Interaction deduplication removes duplicate user-item interaction records, reducing training sample count, forward/backward FLOPs, and I/O bandwidth without changing the embedding table capacity. Cold embedding pruning removes rarely accessed entity IDs from the embedding table, directly reducing memory capacity and memory-bandwidth footprints on the host or accelerator.

Learning Objective: Compare the systems effects of interaction deduplication versus cold embedding pruning in recommendation models.
True or False: Removing near-duplicate documents with MinHash LSH always improves model accuracy because duplicate data has zero statistical value in all training regimes.

Answer: False. While deduplication reduces redundant computation and memorization risk, in some domains duplicate frequency reflects natural empirical data distributions or intentional weighting. Aggressive deduplication with miscalibrated thresholds can prune legitimate variations or distort class priors, necessitating empirical validation on downstream accuracy.

Learning Objective: Evaluate the systems and statistical trade-offs of aggressive near-duplicate filtering.
The training-dynamics coreset metric that measures the Euclidean norm of the difference between predicted class probabilities and the one-hot target vector early in training is known as ____.

Answer: EL2N (or Error L2-Norm). EL2N measures $\|p(x) - y\|_2$ early in training, capturing sample difficulty and transferring effectively from small proxy models to large production architectures.

Learning Objective: Explain the definition and purpose of EL2N as a proxy-based coreset scoring metric.

← Back to Questions

Self-Check: Answer

In curriculum learning, an engineer implements an ‘easy-to-hard’ pacing schedule that controls the fraction of the sorted training pool available to the model at training step $t$. What is the primary systems and statistical objective of this pacing strategy?
1. To guide optimization through stable early gradient trajectories using low-variance samples before exposing the model to high-variance boundary cases
2. To eliminate the need for backward passes during the first half of training
3. To maximize GPU memory bandwidth utilization by sorting tensors strictly by length in bytes
4. To replace human labelers with an automated oracle during the late stages of training
Answer: The correct answer is A. Curriculum learning presents easy (low-noise, canonical) examples early to establish stable feature representations and prevent gradient divergence, gradually introducing harder and noisier examples as the model matures. Curriculum pacing does not eliminate backward passes, does not sort tensors merely for memory bandwidth alignment, and does not replace human labelers in supervised pipelines.

Learning Objective: Analyze the optimization and statistical mechanisms of curriculum learning pacing schedules.
An active learning pipeline chooses unlabeled examples for costly radiologist annotation. The team notices that simple uncertainty sampling repeatedly selects images from a single ambiguous artifact class, starving other disease categories. Which query strategy should they adopt to resolve this pathology?
1. Least-confidence sampling, because it strictly selects the lowest top-1 probability prediction
2. Diversity sampling or hybrid uncertainty-diversity sampling (such as BADGE), which balances uncertainty near decision boundaries with feature-space coverage
3. Random undersampling of the entire unlabeled pool to reduce dataset size before scoring
4. Uniform zero-shot pseudo-labeling without confidence thresholds
Answer: The correct answer is B. Pure uncertainty sampling often suffers from sampling bias by selecting clustered points near a single ambiguous boundary region. Diversity sampling (or hybrid strategies like BADGE that incorporate gradient embeddings) ensures that selected queries span diverse clusters across the feature space while still targeting model uncertainty. Least-confidence sampling exacerbates the clustering issue, random undersampling discards informative candidates arbitrarily, and unthresholded pseudo-labeling introduces catastrophic label noise.

Learning Objective: Compare active learning query strategies to mitigate sampling bias and redundancy.
Explain the mechanism of confirmation bias in semi-supervised pseudo-labeling, and specify how confidence thresholding mitigates it.

Answer: Confirmation bias occurs when a model makes confident but incorrect predictions on unlabeled data, converts them into ground-truth pseudo-labels, and retrains on them, reinforcing its own errors across subsequent iterations. Setting a high confidence threshold ($\tau$) ensures that only predictions with high posterior probability receive pseudo-labels, filtering out uncertain and error-prone samples before they corrupt the training distribution.

Learning Objective: Explain confirmation bias in semi-supervised pseudo-labeling and the role of confidence thresholds.
True or False: Consistency regularization methods such as FixMatch rely on the smoothness assumption, asserting that realistic perturbations of an input sample should not change the model’s predicted class distribution.

Answer: True. Consistency regularization enforces that if two inputs $x$ and $x'$ are close in input space (e.g. through weak and strong data augmentations), their model predictions should also be close, thereby driving decision boundaries into low-density regions.

Learning Objective: Evaluate the foundational distributional assumptions underlying consistency regularization.
Order the steps in an active learning closed-loop iteration: (1) Select top query samples via query strategy, (2) Acquire expert annotations from the oracle, (3) Score unlabeled pool using current model, (4) Retrain or update model on expanded dataset, (5) Add newly labeled samples to the training set.

Answer: The correct order is: (3) Score unlabeled pool using current model, (1) Select top query samples via query strategy, (2) Acquire expert annotations from the oracle, (5) Add newly labeled samples to the training set, (4) Retrain or update model on expanded dataset. The cycle begins with proxy or model scoring over the candidate pool, selecting candidate queries, querying the human oracle, incorporating verified labels into the dataset, and updating the model.

Learning Objective: Design the closed-loop workflow of an iterative active learning system.

← Back to Questions

Self-Check: Answer

In the economics of foundation models, an organization invests $C_{\text{pretrain}} = 10{,}000$ GPU-hours in self-supervised pretraining. Each downstream task fine-tuning costs $C_{\text{finetune}} = 50$ GPU-hours. Training each task from scratch would cost $C_{\text{scratch}} = 1{,}000$ GPU-hours. What is the minimum number of downstream tasks $N^*$ required to break even on the pretraining investment?
1. $N^* = 5$ downstream tasks
2. $N^* = 8$ downstream tasks
3. $N^* = 11$ downstream tasks ($10{,}000 + 11 \times 50 = 10{,}550 < 11 \times 1{,}000 = 11{,}000$)
4. $N^* = 50$ downstream tasks
Answer: The correct answer is C. The break-even condition is $C_{\text{pretrain}} + N \cdot C_{\text{finetune}} \le N \cdot C_{\text{scratch}}$, which gives $10{,}000 + 50N \le 1000N$, or $950N \ge 10{,}000$. Solving for $N$ yields $N \ge 10{,}000 / 950 \approx 10.53$. Thus, at least 11 downstream tasks are required for the self-supervised foundation model investment to be cheaper than training from scratch. At 10 tasks, scratch training costs $10{,}000$ hours while the foundation model costs $10{,}500$ hours. At 11 tasks, scratch training costs $11{,}000$ hours while the foundation model costs $10{,}550$ hours.

Learning Objective: Calculate the break-even threshold for amortizing self-supervised pretraining across downstream tasks.
How does the MoCo (Momentum Contrast) framework reduce the hardware and memory constraints of contrastive self-supervised learning compared to naive SimCLR?
1. By replacing convolutional backbones with rule-based lookup tables to avoid backpropagation
2. By requiring fully supervised class labels to filter out false negative pairs
3. By using a dynamic queue of negative keys and a slowly updating momentum encoder, decoupling negative dictionary size from mini-batch size
4. By computing exact pairwise Jaccard similarities on raw byte sequences rather than latent embeddings
Answer: The correct answer is C. Contrastive learning requires large sets of negative examples for effective representation learning. While SimCLR scales the mini-batch size (requiring massive GPU memory across many accelerators, e.g. batch size 4096), MoCo decouples the dictionary size from the mini-batch size by maintaining a memory queue of negative keys updated with a momentum-averaged teacher encoder. MoCo still uses neural backbones and backpropagation, operates in an unsupervised manner without human labels, and computes cosine similarities in latent space rather than raw byte Jaccard comparisons.

Learning Objective: Compare contrastive self-supervised learning architectures and their memory-compute scaling trade-offs.
What is the systems-level ‘homogenization risk’ (or blast radius) of pretraining a shared foundation model on an uncurated dataset?

Answer: Because a single foundation model serves as the common upstream base for dozens or thousands of downstream applications, any flaw, bias, toxic pattern, or memorized sensitive data in the pretraining corpus propagates universally into every downstream fine-tuned model, multiplying the blast radius of upstream curation errors.

Learning Objective: Evaluate the blast radius and homogenization risk associated with foundation model pretraining corpora.
True or False: Because self-supervised pretraining learns general representations from unlabeled data, it completely eliminates the need for data selection or quality filtering during the pretraining stage.

Answer: False. Self-supervised pretraining at web scale remains highly sensitive to pretraining data quality; uncurated corpora containing boilerplate, corrupted text, toxic samples, or pervasive duplicates degrade representation quality and inflate training FLOPs without improving downstream transfer.

Learning Objective: Evaluate the misconception that self-supervised learning eliminates data curation requirements.
An unsupervised learning approach where the model solves an auxiliary task constructed directly from the structure of unlabeled data (such as masked token prediction or contrastive instance discrimination) is called a ____ task.

Answer: pretext (or self-supervised pretext). Pretext tasks generate supervision signals directly from input structure without manual annotations, enabling representation pretraining.

Learning Objective: Explain the concept and role of pretext tasks in self-supervised learning.

← Back to Questions

Self-Check: Answer

What causes the phenomenon of ‘model collapse’ (or the autophagous loop) when generative models are trained recursively on synthetic data generated by earlier model iterations?
1. GPU memory fragmentation caused by variable-length synthetic sequences during distributed training
2. Overfitting to floating-point rounding errors during fp16 mixed-precision matrix multiplication
3. A failure of the I/O storage subsystem to deliver synthetic batches at line rate
4. Systematic underrepresentation and progressive pruning of the tail distributions of the true data distribution across successive generations
Answer: The correct answer is D. Generative models sample with higher probability from the mode of their learned distribution, inherently underrepresenting rare tail events. When generation $n+1$ trains on outputs from generation $n$, the tails of the distribution are progressively truncated and compressed, causing the generated data to lose diversity and collapse into homogeneous, degraded outputs. GPU memory fragmentation, fp16 rounding, and storage I/O bottlenecks are systems execution issues, not the statistical cause of model collapse.

Learning Objective: Analyze the mathematical and statistical causes of model collapse in recursive synthetic training.
In knowledge distillation, what is the primary role of the temperature parameter $T$ when computing soft targets from a teacher model’s logits $z_i$ ($p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}$)?
1. To soften the output probability distribution, exposing the relative probability structure (‘dark knowledge’) over non-target classes to the student
2. To clamp gradients to prevent numerical overflow in the student’s backward pass
3. To dynamically increase learning rate when the student loss plateaus
4. To randomly drop connections in the student network like dropout
Answer: The correct answer is A. Higher temperature $T > 1$ softens the softmax distribution over logits, revealing the rich relative similarities between incorrect classes (the teacher’s ‘dark knowledge’) that are hidden by standard hard argmax or sharp low-temperature one-hot predictions. Temperature scaling is not gradient clipping, does not adjust the optimizer learning rate, and is distinct from dropout regularization.

Learning Objective: Explain the purpose of temperature scaling in knowledge distillation for information transfer.
Why is training exclusively on $100\%$ synthetic data from a simulation engine often suboptimal for real-world deployment, and how does synthetic-to-real data mixing bridge this gap?

Answer: Simulation engines have an inherent domain gap ($\mathcal{D}_{\text{synth}} \ne \mathcal{D}_{\text{real}}$) caused by simplified physics, missing sensor artifacts, and unmodeled environmental noise. Mixing synthetic data with a curated set of real-world examples anchors the model to real feature distributions while using synthetic samples to expand volume, cover edge cases, and balance rare classes.

Learning Objective: Analyze domain gap challenges and justify synthetic-to-real mixing ratios.
True or False: Using a high-capacity diffusion model or state-of-the-art LLM to generate synthetic training data guarantees that the resulting training set will be completely free of distribution shift relative to the real deployment environment.

Answer: False. Generator quality does not eliminate domain gap. Synthetic generators can amplify subtle distribution shifts, introduce generator-specific artifacts, and omit edge cases present in real-world deployment environments, requiring domain validation against real ground truth.

Learning Objective: Evaluate the misconception that advanced generative models eliminate domain shift.
Order the stages in a multi-tier audio dataset expansion pipeline starting from a small seed recording: (1) Hard-negative mining, (2) Acoustic noise injection, (3) Seed recordings collection, (4) TTS simulation, (5) Geometric/Transformation-based audio augmentation.

Answer: The correct order is: (3) Seed recordings collection, (5) Geometric/Transformation-based audio augmentation, (2) Acoustic noise injection, (1) Hard-negative mining, (4) TTS simulation. The pipeline starts with high-quality seed utterances, applies transformation augmentations (pitch/speed shifts), injects background noise environments, mines hard negatives that confuse the baseline model, and uses TTS simulation to generate missing vocabulary and speaker variations.

Learning Objective: Design a tiered data expansion pipeline combining real seed collection, augmentation, noise injection, and synthetic simulation.

← Back to Questions

Self-Check: Answer

According to the chapter’s decision framework, if an ML team has an abundant pool of unlabeled domain data, very limited annotation budget, and access to human domain experts for selective queries, which technique branch is recommended?
1. Pure transformation-based data augmentation without labeling
2. Active learning (human-in-the-loop selective query) or semi-supervised learning
3. Generative self-instruct synthesis to replace all human annotators entirely
4. Exhaustive pairwise Jaccard deduplication across all unlabeled samples
Answer: The correct answer is B. When the primary bottleneck is labeling cost, the decision framework routes based on oracle availability: if a human oracle is available, active learning queries the most informative samples; when combined with unlabeled data, semi-supervised learning leverages structural priors. Pure augmentation does not provide ground truth for novel concepts, generative self-instruct cannot replace domain experts where ground-truth verification is required, and pairwise deduplication is a pruning step rather than a labeling strategy.

Learning Objective: Apply the data selection decision framework to match operational constraints with appropriate selection paradigms.
When deciding between static coreset pruning and dynamic online active selection for a production pipeline, what role does the expected number of training runs ($N$) play in the architectural choice?

Answer: Static coreset pruning incurs an upfront scoring and curation cost that amortizes across all subsequent training runs ($N \gg 1$), making it highly cost-effective for hyperparameter sweeps, architecture searches, and repeated model retraining. For a one-off single training run ($N=1$), expensive static scoring cannot be amortized, favoring lightweight dynamic selection or random sampling to avoid violating the Selection Inequality.

Learning Objective: Compare static coreset pruning and dynamic active selection based on training run amortization.
Order the decision steps when triaging a data pipeline bottleneck using the chapter’s decision framework: (1) Evaluate simulator and domain synthesizer availability, (2) Identify the primary constraint (label scarcity vs. compute limits vs. data scarcity), (3) Select the specific algorithmic technique (e.g. SSL, Active Learning, Coreset Pruning, or Generative Synthesis), (4) Assess human oracle availability and budget.

Answer: The correct order is: (2) Identify the primary constraint (label scarcity vs. compute limits vs. data scarcity), (4) Assess human oracle availability and budget, (1) Evaluate simulator and domain synthesizer availability, (3) Select the specific algorithmic technique (e.g. SSL, Active Learning, Coreset Pruning, or Generative Synthesis). The decision tree begins by establishing the primary system bottleneck, branches through resource availability (human oracles or simulators), and concludes with selecting the matching algorithmic technique.

Learning Objective: Design a systematic triage procedure using the data selection decision framework.

← Back to Questions

Self-Check: Answer

A training team reduces a dataset from $1{,}000{,}000$ to $100{,}000$ images (a $10\times$ coreset). Training on the full dataset takes 10 hours ($T_{\text{train}}(\text{full}) = 10\text{ hr}$), while training on the coreset takes 1 hour ($T_{\text{train}}(\text{subset}) = 1\text{ hr}$). However, scoring the $1\text{M}$ pool with the full production model takes 12 hours ($T_{\text{selection}} = 12\text{ hr}$). Does this configuration satisfy the Selection Inequality, and what engineering change restores positive ROI?
1. Yes, because the dataset was reduced by $90\%$; no engineering change is needed
2. Yes, because $10\text{ hr} - 1\text{ hr} = 9\text{ hr}$ of savings outweighs the scoring cost; increase GPU count by $2\times$
3. No, because $T_{\text{selection}} + T_{\text{train}}(\text{subset}) = 13\text{ hr} > 10\text{ hr}$; replace full-model scoring with a lightweight proxy model or cached embeddings
4. No, because coreset training always increases memory bandwidth consumption; switch from NVMe SSDs to HDDs
Answer: The correct answer is C. The Selection Inequality states $T_{\text{selection}} + T_{\text{train}}(D_{\text{subset}}) < T_{\text{train}}(D_{\text{total}})$. Here, $12\text{ hr} + 1\text{ hr} = 13\text{ hr} > 10\text{ hr}$, resulting in a net wall-clock loss of 3 hours despite a $90\%$ sample reduction. Using a fast proxy model or precomputed embeddings to score the candidate pool drops $T_{\text{selection}}$ to a fraction of an hour, satisfying the inequality. Sample reduction percentage alone does not guarantee net savings, and switching to HDDs worsens I/O latency.

Learning Objective: Calculate and evaluate the Selection Inequality to identify selection latency bottlenecks.
Why do naive random sample lookups across non-contiguous indices in large un-sharded dataset files severely degrade I/O throughput on storage hardware, and how do shuffle buffers mitigate this?
1. Random lookups bypass host CPU caches, forcing floating-point registers to re-encode all labels
2. Random lookups violate PCIe parity checks, causing GPU kernel timeouts during backward passes
3. Random lookups trigger hash collisions in the Python garbage collector, halting dataloading threads
4. Random 4 KB reads achieve only a tiny fraction of peak sequential storage bandwidth due to IOPS limits, whereas shuffle buffers read large sequential chunks and randomize locally in memory
Answer: The correct answer is D. Storage devices (HDDs, SATA SSDs, NVMe SSDs, and cloud object stores) provide peak throughput under large, contiguous sequential reads. Non-contiguous random reads drop realized bandwidth dramatically (e.g. from gigabytes/sec to megabytes/sec on SSDs, or hundreds of times worse on HDDs). Shuffle buffers co-design data loading by reading large sequential shards into host RAM and performing pseudo-random shuffling locally within memory, preserving peak sequential disk bandwidth. The other choices describe fictitious hardware or software failures.

Learning Objective: Analyze the hardware empathy principles governing storage I/O and shuffle buffer design.
Contrast upstream data echoing (echoing before data augmentation) with downstream data echoing (echoing after augmentation) in terms of computational overhead and sample diversity.

Answer: Upstream data echoing re-reads raw samples once from storage and applies different randomized augmentations on each repeated pass, maximizing gradient diversity while amortizing I/O load. Downstream data echoing repeats the identical post-augmented tensor to the accelerator, eliminating both I/O and CPU augmentation compute at the cost of zero intra-sample diversity.

Learning Objective: Compare upstream versus downstream data echoing architectures and their systems trade-offs.
True or False: If the GPU training step takes 20 ms and the CPU data loading/augmentation pipeline takes 10 ms ($R = T_{\text{pipeline}} / T_{\text{GPU}} = 0.5$), applying a data echoing factor of $e = 2$ will double the end-to-end training throughput.

Answer: False. When $R < 1$, the training job is accelerator-bound (the data pipeline is already faster than the GPU), so the GPU is never starved for data. Applying data echoing in this regime provides zero throughput gain and merely feeds duplicate or stale data to the accelerator.

Learning Objective: Evaluate pipeline balance conditions under which data echoing provides zero throughput improvement.
The pipeline optimization technique that reuses intermediate data samples multiple times before fetching new batches to keep accelerators saturated when CPU data processing or I/O is the bottleneck is called data ____.

Answer: echoing. Data echoing amortizes upstream I/O and CPU transformation overhead by repeating samples through downstream pipeline stages.

Learning Objective: Explain the definition and purpose of data echoing in ML data pipelines.

← Back to Questions

Self-Check: Answer

A company invests $C_{\text{select}} = \$30{,}000$ to compute a high-quality coreset. Training on the full dataset costs $C_{\text{train}}(D) = \$10{,}000$ per run, whereas training on the coreset costs $C_{\text{train}}(S) = \$4{,}000$ per run. What is the break-even number of training runs $N^*$ required to justify this static selection investment, and what is the ROI after 10 training runs?
1. $N^* = 5$ runs, and $\text{ROI} = 100\%$ after 10 runs (Net savings = $\$30{,}000$ on a $\$30{,}000$ investment)
2. $N^* = 3$ runs, and $\text{ROI} = 300\%$ after 10 runs
3. $N^* = 8$ runs, and $\text{ROI} = 50\%$ after 10 runs
4. $N^* = 10$ runs, and $\text{ROI} = 0\%$ after 10 runs
Answer: The correct answer is A. Per-run training savings is $\Delta C = C_{ ext{train,full}} - C_{ ext{train,subset}} = \{,}000 - \{,}000 = \{,}000$. The break-even number of runs is ^* = C_{ ext{select}} / C = {,}000 / {,}000 = 5$ runs. After = 10$ runs, total gross training savings is imes {,}000 = {,}000$. Net savings is $\{,}000 - \{,}000 = \{,}000$. Return on Investment is $ ext{ROI} = ext{Net Savings} / C_{ ext{select}} = {,}000 / {,}000 = 1.0$, or \%$. The alternative choices miscalculate either the per-run savings difference or the ROI denominator.

Learning Objective: Calculate break-even training runs and Return on Investment (ROI) for static data selection.
In the full lifecycle cost equation for machine learning data systems ($C_{\text{total}} = C_{\text{acquire}} + C_{\text{label}} + C_{\text{filter}} + C_{\text{train}} + C_{\text{eval}}$), which scenario demonstrates the most effective use of upstream data filtering to minimize total expenditure?
1. Spending $\$0$ on filtering to ensure maximum raw token count reaches the final evaluation cluster
2. Spending $\$5{,}000$ on automated heuristic filtering to discard $60\%$ of corrupt samples before paying $\$100{,}000$ in human labeling and training fees
3. Doubling human labeling rates to manually review every web-scraped token before filtering
4. Eliminating model evaluation to offset the compute cost of running unpruned training runs
Answer: The correct answer is B. Spending a small amount ($C_{\text{filter}} = \$5{,}000$) on automated heuristic filtering upstream prevents wasting large labeling ($C_{\text{label}}$) and training ($C_{\text{train}}$) expenditures on corrupt or uninformative data. Spending zero on filtering shifts massive costs downstream, manual review of all raw web text is economically unfeasible, and eliminating evaluation destroys model quality verification.

Learning Objective: Analyze the lifecycle cost components of data systems to optimize pipeline investments.
Explain why calculating Return on Investment (ROI) for data selection requires tracking engineering implementation and pipeline maintenance costs in addition to raw accelerator compute hours.

Answer: A data selection method that saves GPU training hours may require custom storage infrastructure, indexing services, proxy model scoring pipelines, and human maintenance. If these engineering overheads exceed the cloud compute savings, the net systems ROI is negative despite apparent model FLOP reductions.

Learning Objective: Evaluate non-compute engineering overheads in the data selection ROI equation.
True or False: In a production setting where a single model will be trained exactly once ($N=1$) with no hyperparameter tuning or future refreshes, spending 50 GPU-hours to compute static EL2N coreset scores that save 30 GPU-hours of training time is an economically sound decision.

Answer: False. For $N=1$, the selection cost (50 GPU-hours) exceeds the training savings (30 GPU-hours), resulting in a net loss of 20 GPU-hours ($C_{\text{select}} + C_{\text{train}}(S) = 50 + C_{\text{train}}(S) > C_{\text{train}}(D)$), directly violating the Selection Inequality.

Learning Objective: Evaluate the Selection Inequality under single-run versus multi-run amortization regimes.
The metric defined as $\text{ROI} = \frac{N \cdot \Delta C_{\text{train}} - C_{\text{select}}}{C_{\text{select}}}$, which measures the net financial or compute return generated by a data selection technique over $N$ training runs, is known as Return on ____.

Answer: Investment (or ROI). Return on Investment quantifies the proportional gain of selection expenditures relative to training cost reductions.

Learning Objective: Explain the Return on Investment metric for amortized data selection techniques.

← Back to Questions

Self-Check: Answer

In a distributed training environment with hundreds of data-parallel worker nodes, why does standard centralized coreset selection fail to scale, and what trade-off does hierarchical selection introduce?
1. Centralized selection requires all workers to share a single GPU; hierarchical selection distributes weights across SSDs
2. Centralized selection fails because sharding prevents network cards from transmitting floating-point values
3. Centralized selection creates a communication and memory bottleneck at the coordinator node; hierarchical selection prunes locally per shard, risking loss of globally rare samples across shards
4. Centralized selection eliminates gradient synchronization; hierarchical selection disables local backpropagation
Answer: The correct answer is C. Centralized selection requires routing all candidate scores or embeddings to a single coordinator, creating severe network bandwidth and memory bottlenecks at scale. Hierarchical selection distributes selection by having workers select local coresets on their shards before merging at the coordinator; however, this shard-local pruning can introduce distribution skew if rare classes or boundary cases are unevenly distributed across shards. The other choices state incorrect network, hardware, or backpropagation limitations.

Learning Objective: Compare centralized, shard-local, and hierarchical architectures for distributed coreset selection.
In distributed active learning, Worker A scores candidate pool samples using model checkpoint step $t$, while asynchronous Worker B updates the shared model parameters to step $t+100$. What consistency challenge arises, and what is the systems remedy?
1. Worker A encounters deadlock in CUDA streams; the remedy is disabling PyTorch autograd
2. Worker B overwrites Worker A’s local storage; the remedy is mounting read-only NFS drives
3. Worker A’s GPU runs out of memory; the remedy is reducing batch size to 1
4. Worker A scores samples against a stale model state, producing invalid uncertainty rankings; the remedy is checkpoint versioning or periodic synchronized score refreshes
Answer: The correct answer is D. Active learning relies on current model uncertainty to query valuable samples. When workers score against stale checkpoints while training progresses asynchronously, the uncertainty scores become misaligned with the active model’s true error distribution. Checkpoint versioning and periodic score refresh intervals guarantee score consistency across distributed workers without stalling pipelines every step. The other choices misdiagnose software deadlocks, NFS overwrites, or OOM issues.

Learning Objective: Analyze consistency challenges and staleness mitigation strategies in distributed active learning.
Explain why performing independent, shard-local coreset pruning on an unstratified, partitioned dataset can cause minority class collapse during distributed training.

Answer: When a dataset is sharded across workers without stratification, rare minority classes may appear in very few shards. If each worker independently selects its top-$k$ coreset based on local metrics, workers with only a few minority samples may discard them as outliers, systematically erasing the minority class from the global training corpus.

Learning Objective: Explain how unstratified data sharding leads to minority class loss during distributed pruning.
Order the execution phases of a distributed coreset selection workflow across a GPU cluster: (1) Compute shard-local embeddings and perform local near-deduplication, (2) Aggregate local candidate indices at the central coordinator, (3) Perform global proxy scoring and thresholding to produce final indices, (4) Broadcast final coreset index list to all worker nodes.

Answer: The correct order is: (1) Compute shard-local embeddings and perform local near-deduplication, (2) Aggregate local candidate indices at the central coordinator, (3) Perform global proxy scoring and thresholding to produce final indices, (4) Broadcast final coreset index list to all worker nodes. The workflow begins with parallelized local feature extraction and deduplication on each worker, aggregates candidates, runs global scoring/selection at the coordinator, and broadcasts the finalized index partition.

Learning Objective: Design the execution sequence of a distributed coreset selection pipeline.

← Back to Questions

Self-Check: Answer

In the D·A·M optimization stack (Data Selection, Algorithm/Model Compression, Machine Hardware Optimization), an ML team achieves a $2\times$ reduction in dataset size via coreset pruning, a $2\times$ reduction in operations per sample via model pruning/quantization, and a $2\times$ increase in hardware arithmetic throughput via kernel optimization. What is the total combined speedup factor for training?
1. An $8\times$ total speedup, because optimizations across distinct layers of the ML systems stack compound multiplicatively ($2 \times 2 \times 2 = 8$)
2. A $6\times$ total speedup, because speedup factors add linearly across layers ($2 + 2 + 2 = 6$)
3. A $2\times$ total speedup, because the lowest-layer optimization bottleneck dominates all others (Amdahl’s law min-factor)
4. A $4\times$ total speedup, because data selection cancels out model compression gains
Answer: The correct answer is A. The Iron Law of ML Systems demonstrates that optimizations at different layers (Workload/Data $\times$ Algorithm/Model $\times$ Machine/Hardware) multiply together when acting on the same execution path. Reducing samples by $2\times$, model operations by $2\times$, and boosting hardware throughput by $2\times$ yields a total speedup of $2 \times 2 \times 2 = 8\times$, far exceeding the additive sum of 6. Optimizations do not cancel each other out, nor do they collapse to the minimum factor when applied across sequential stages.

Learning Objective: Calculate the multiplicative compounding effects across data, algorithm, and machine optimization layers.
Why are upstream data selection (Workload layer) and downstream model compression (Algorithm layer) fundamentally complementary rather than interchangeable techniques in system design?
1. Model compression can only be applied to computer vision models, whereas data selection is restricted to NLP
2. Data selection eliminates backward passes entirely, whereas model compression eliminates forward passes
3. Data selection optimizes inference latency on edge devices, whereas model compression only affects training time
4. Data selection reduces the total number of training samples processed ($N_{\text{samples}}$), whereas model compression reduces the compute and memory cost per individual sample forward/backward pass ($O_{\text{sample}}$)
Answer: The correct answer is D. Upstream data selection determines which samples enter the workload (reducing sample count $N_{\text{samples}}$), while model compression determines how much compute each sample requires (reducing operations per pass $O_{\text{sample}}$ and memory footprint). Applying both multiplies total efficiency. Neither technique is restricted by modality, both affect forward and backward execution, and model compression is widely used for edge inference while data selection optimizes training workloads.

Learning Objective: Compare the architectural roles of workload-layer data selection and algorithm-layer model compression.
A training pipeline aggressively reduces dataset size with a $10\times$ coreset. However, the engineering team observes that the end-to-end training job speedup is only $2\times$ instead of the expected $10\times$. Using systems principles, diagnose the likely bottleneck shift.

Answer: Aggressively shrinking the dataset by $10\times$ reduces compute time tenfold, shifting the primary bottleneck from GPU compute to fixed overheads such as un-amortized framework initialization, model checkpointing, distributed barrier synchronization, or I/O data loader startup latency, capping overall speedup per Amdahl’s Law.

Learning Objective: Analyze bottleneck shifts caused by aggressive dataset pruning.
Explain how data selection operates upstream of all algorithm- and hardware-level optimizations in the D·A·M optimization stack.

Answer: Data selection operates at the Workload layer by pruning uninformative or duplicate samples before they enter the training pipeline. Because an eliminated sample requires zero forward passes, zero backward passes, zero optimizer steps, and zero communication across nodes, data selection prevents downstream compute and memory operations from ever executing.

Learning Objective: Explain how data selection acts as an upstream workload filter in the optimization stack.

← Back to Questions

Self-Check: Answer

Match the data-selection efficiency metric with its precise definition: A team wants to measure the ratio of full dataset size to selected subset size ($\text{DCR} = |D| / |S|$), and the ratio of final accuracy achieved on the subset versus the full dataset ($\text{ARR} = \text{Acc}(S) / \text{Acc}(D)$). What do DCR and ARR stand for?
1. Data Curation Rate and Accuracy Reduction Ratio
2. Data Compression Ratio and Accuracy Retention Ratio
3. Dynamic Checkpoint Rate and Active Retention Rate
4. Data Convergence Ratio and Amortized Risk Ratio
Answer: The correct answer is B. DCR is the Data Compression Ratio ($|D_{ ext{full}}| / |D_{ ext{subset}}|$), quantifying how many times smaller the training subset is relative to the original pool. ARR is the Accuracy Retention Ratio ($ ext{Acc}{ ext{subset}} / ext{Acc}{ ext{full}}$), measuring what proportion of the baseline full-data accuracy is preserved by the selected subset. The other terms are incorrect nomenclature.

Learning Objective: Classify and define the core metrics of the data selection measurement framework.
An ML systems diagnostic plot maps normalized training compute (FLOPs) on the horizontal log-axis against model performance on the vertical axis. A training run with 10M parameters sits significantly below the green compute-optimal frontier, and increasing token count yields no accuracy improvement while scaling model parameters to 100M immediately restores optimal frontier scaling. What was the diagnosis of the original operating point?
1. Data-starved regime
2. I/O bandwidth-saturated regime
3. Compute-starved (capacity-limited) regime
4. Over-echoing regime
Answer: The correct answer is C. The run was compute-starved (or parameter capacity-limited): the 10M model lacked the representational capacity to absorb more data, causing performance to plateau below the frontier. Increasing model capacity to 100M allowed the system to utilize the compute budget efficiently and move up to the compute-optimal frontier. In a data-starved regime, the model has excess capacity but lacks high-quality tokens, which would be resolved by adding data or improving selection, not merely increasing parameter count.

Learning Objective: Analyze operating points on the compute-optimal frontier to diagnose compute-starved versus data-starved training regimes.
Why is Area Under the Learning Curve (AULC) a more informative metric than single-point final validation accuracy when evaluating dynamic data selection and curriculum learning algorithms?

Answer: Single-point final accuracy evaluates model capability only at the end of training, ignoring the rate of learning progress. AULC integrates accuracy across all intermediate compute steps or epochs, rewarding methods that achieve high accuracy rapidly and reach target performance with fewer FLOPs.

Learning Objective: Justify using Area Under the Learning Curve (AULC) over single-point final accuracy.
True or False: If two training runs (Run A on a raw dataset and Run B on a coreset) reach the exact same validation loss plateau, they must have processed identical amounts of informative tokens.

Answer: False. Run A on the raw dataset may have processed a large volume of redundant and uninformative tokens, wasting compute to reach the plateau. Run B on the coreset concentrated compute on high-ICR boundary samples, reaching the same loss plateau with a fraction of the total tokens and FLOPs.

Learning Objective: Evaluate loss curve plateaus to distinguish compute efficiency from raw sample volume.
The Pareto frontier that defines the maximum achievable model accuracy or minimum loss for every given training compute budget (FLOPs) under balanced parameter and token allocation is known as the compute-____ frontier.

Answer: optimal (or compute-optimal frontier). The compute-optimal frontier represents the theoretical efficiency boundary where model size and dataset size are optimally balanced.

Learning Objective: Explain the concept and significance of the compute-optimal frontier.

← Back to Questions

Self-Check: Answer

A research paper reports that an EL2N coreset selection method successfully pruned $50\%$ of CIFAR-10 with $0\%$ loss in accuracy. A medical imaging team applies the exact same $50\%$ pruning ratio to a rare tumor detection dataset and suffers a disastrous $28\%$ drop in recall. What fallacy explains this failure?
1. The team failed to use GPU acceleration during the inference pass
2. The team used float32 precision instead of bfloat16 mixed precision
3. CIFAR-10 contains more total classes than medical imaging datasets
4. Assuming that benchmark coreset pruning ratios transfer directly to specialized, highly imbalanced production domains with rare failure modes
Answer: The correct answer is D. Pruning ratios calibrated on balanced, homogeneous academic benchmarks like CIFAR-10 rarely transfer to specialized production domains like medical imaging. In domains with extreme class imbalance or subtle pathological features, a $50\%$ global prune disproportionately discards rare minority samples and critical edge cases, causing severe domain-specific performance drops. Hardware precision and class counts are irrelevant to this transfer fallacy.

Learning Objective: Analyze why benchmark data pruning ratios fail to transfer to specialized production domains.
A team implements an elaborate multi-stage active learning pipeline that reduces training dataset size by $40\%$. However, the continuous clustering, proxy scoring, and cross-worker all-gather synchronization take 3 times longer than the GPU time saved during training. Which pitfall does this represent?
1. Violating the Selection Inequality by incurring selection overheads that exceed downstream training savings ($T_{\text{selection}} > \Delta T_{\text{train}}$)
2. Encountering model collapse due to recursive generator loops
3. Failing to implement 4 KB small-read alignment on host NVMe storage
4. Violating the smoothness assumption in semi-supervised consistency regularization
Answer: The correct answer is A. This is the canonical violation of the Selection Inequality: an engineering team designs a selection algorithm that successfully reduces sample count, but the computational and synchronization overhead of scoring and filtering ($T_{\text{selection}}$) exceeds the training time saved on the reduced subset, leading to a net increase in total wall-clock time. Model collapse relates to recursive generative training, NVMe alignment relates to I/O access patterns, and the smoothness assumption belongs to semi-supervised learning.

Learning Objective: Analyze violations of the Selection Inequality in complex data selection architectures.
Why is evaluating a data selection strategy solely on aggregate validation accuracy a dangerous pitfall when dealing with imbalanced datasets?

Answer: In imbalanced datasets, aggregate accuracy is heavily dominated by majority classes. A data pruning algorithm could discard $90\%$ of rare minority or safety-critical examples to optimize overall compute while aggregate accuracy appears unchanged, completely compromising model performance on rare real-world failure modes.

Learning Objective: Evaluate the pitfall of aggregate metric optimization under severe class imbalance.
True or False: Because modern high-capacity generative models produce photorealistic images and fluent text, a model trained on $100\%$ recursively generated synthetic data will never suffer from performance degradation.

Answer: False. Recursive training on synthetic data inevitably triggers model collapse and distribution tail erosion, as generative models progressively drop rare modes and compound statistical errors across generations, severely degrading model capability.

Learning Objective: Evaluate the fallacy that high generative fidelity prevents model collapse in synthetic data training.

← Back to Questions

Self-Check: Answer

Which summary statement best captures the central systems principle of data selection established throughout this chapter?
1. Data selection is an offline heuristic that only applies to small academic image classification benchmarks
2. Data selection is the highest-leverage input optimization layer because it eliminates FLOPs, memory traffic, and communication before model or hardware execution begins
3. Data selection replaces all algorithm-level model compression and hardware acceleration optimizations
4. Data selection is strictly bounded by the requirement that datasets must grow linearly with GPU cluster node counts
Answer: The correct answer is B. Data selection operates as the first and highest-leverage optimization layer in the ML systems hierarchy: by identifying and retaining only high-ICR tokens and samples, it eliminates forward passes, backward passes, memory bandwidth consumption, and gradient communications upstream. Data selection does not replace downstream model compression or hardware optimizations (it compounds multiplicatively with them) and is not an academic heuristic.

Learning Objective: Explain the overarching systems role of data selection in the optimization hierarchy.
In one integrated explanation, relate the Information-Compute Ratio (ICR), the Selection Inequality, and the three-stage data selection pipeline.

Answer: ICR provides the theoretical optimization metric (maximizing learning progress per unit of compute), the Selection Inequality defines the practical feasibility constraint ($C_{\text{select}} + C_{\text{train}}(S) < C_{\text{train}}(D)$), and the three-stage pipeline (static pruning, dynamic selection, synthetic generation) provides the architectural framework to achieve high ICR while satisfying the selection inequality.

Learning Objective: Compare the roles of ICR, the Selection Inequality, and the three-stage data selection pipeline.
How does workload-level data selection interact with the broader D·A·M optimization stack to maximize end-to-end training efficiency?

Answer: Workload-level data selection reduces the number of samples ($N_{\text{samples}}$) entering the system, which compounds multiplicatively with algorithm-level reductions in operations per sample ($O_{\text{sample}}$) and machine-level increases in hardware throughput ($R_{\text{peak}} \cdot \eta_{\text{hw}}$), maximizing end-to-end efficiency across the entire training stack.

Learning Objective: Explain how data selection compounds with algorithmic and hardware optimizations in the D·A·M stack.

← Back to Questions

Component	Formula	Illustrative Range
\(C_{\text{acquire}}\)	\(D \times c_{\text{sample}}\)	$0.001–$10/sample (web scrape vs. licensed)
\(C_{\text{label}}\)	\(D_{\text{labeled}} \times c_{\text{label}}\)	$0.01–$200/sample (crowd vs. expert)
\(C_{\text{store}}\)	\(D_{\text{vol,store}} \times c_{\text{storage}} \times T_{\text{months}}\)	$0.02–$0.10/GB/month
\(C_{\text{process}}\)	\(D \times N_{\text{epochs}} \times O_{\text{sample}} \times c_{\text{FLOP}}\)	Proportional to training FLOPs