Development Principles

Building a machine learning system requires more than assembling model components; it requires managing the flow of information and energy through silicon. Part I established that data is both the program and the physical anchor of every ML system. With that anchor in place, Part II turns to the algorithm-machine interaction: how mathematical models are co-designed around the physical limits of the hardware that must execute them. The principles here begin with the accounting that determines why certain architectures succeed while others fail at scale.

Principle 3: The Iron Law of ML Systems

Invariant: Let \(T\) denote idealized wall-clock time in seconds. Here \(D_{\text{vol}}\) is data volume in bytes, \(\text{BW}\) is the effective bandwidth of the relevant memory or network path in bytes/s, \(O\) is total floating-point operations, \(R_{\text{peak}}\) is peak compute rate in FLOP/s, \(\eta_{\text{hw}}\) is dimensionless hardware utilization efficiency, and \(L_{\text{lat}}\) is fixed latency overhead in seconds, such as a kernel launch or network round trip. When data movement, compute, and fixed overhead execute serially, their idealized times add: \[ T \approx \frac{D_{\text{vol}}}{\text{BW}} + \frac{O}{R_{\text{peak}} \cdot \eta_{\text{hw}}} + L_{\text{lat}} \] The full treatment appears in Iron Law of ML Systems.

When data movement and compute overlap on modern hardware, wall-clock time follows the critical path and cannot be shorter than the slower of those stages plus any fixed latency that remains outside the overlap. The practical lesson is about dominance, not unconditional summation.

Implication: Optimization is rarely free of trade-offs. Reducing one term often shifts the bottleneck to another. For example, exploiting unstructured sparsity can reduce executed arithmetic, but sparse representations and kernels may lower effective bandwidth or add index traffic, increasing data-movement time \((D_{\text{vol}}/\text{BW})\). An optimization improves wall-clock time only to the extent that it shortens the critical path on the target hardware.

The iron law identifies what to optimize, but not how. The architecture decides how before implementation begins.

Principle 4: The Silicon Contract

Invariant: Every model architecture and workload regime makes an implicit commitment to the hardware, a wager on which resource it will saturate first.

• ResNet-50 assumes high-density floating-point compute. In batched training or inference on accelerators, it is often compute bound: performance is limited by \(O/(R_{\text{peak}} \cdot \eta_{\text{hw}})\).
• 8-billion-parameter Llama 3 assumes high-bandwidth memory access during autoregressive decoding. At small batch sizes, it is often bandwidth bound: performance is limited by \(D_{\text{vol}}/\text{BW}\).
• DLRM assumes massive embedding tables and sparse lookups. It is shaped by both capacity and bandwidth: performance depends on whether embedding tables fit in the available memory hierarchy and how quickly sparse accesses can be served.

Implication: Designing a model without knowing which hardware resource it will saturate is like designing a bridge without knowing the strength of the steel. The design must target the bottleneck.

Together, the iron law and the silicon contract frame every design decision in Part II. The chapters that follow translate these principles into the components of the ML stack: the mathematical foundations of gradient flow, the architectural patterns that commit to specific hardware resources, the frameworks that map model operations onto hardware, and the training systems that execute the same physics at scale.