Foundation Principles

Machine learning systems exhibit a deceptively simple heuristic: complexity often moves rather than disappearing, although good design can remove accidental complexity. Complexity flows among the three domains of the D·A·M taxonomy: Data as information, Algorithm as logic, and Machine as physics. Simplifying one domain can burden the others. A hand-crafted feature pipeline reduces algorithmic complexity but demands more data engineering effort. A larger model absorbs messy data but shifts complexity onto the hardware that must train and serve it. This Conservation of Complexity heuristic motivates everything in this book. The quantitative bounds, models, and principles introduced throughout the book describe constraints that emerge from where complexity currently resides.

Architectures, frameworks, and optimizations succeed only when they respect constraints imposed by hardware, mathematics, and information theory. Just as civil engineers cannot ignore gravity, ML engineers cannot ignore the physical laws that govern data, computation, and system throughput. Part I establishes these foundations: not best practices that evolve with frameworks or opinions that differ between teams, but enduring physical constraints and quantitative models for ML engineering. The first constraint starts with data itself, where the familiar boundary between program and input begins to disappear.

Principle 1: The Data-as-Code Principle

Invariant: Data functions as the source code of the ML system. A change to the training dataset can alter the system’s learned behavior. Here, learned behavior means the trained input-output mapping; \(f\) denotes the training process, and \(\approx\) marks a qualitative dependency rather than a predictive equality. Model architecture, initialization, optimizer configuration, and training stochasticity are included in the training procedure. \[ \text{Learned Behavior} \approx f(\text{Training Data}, \text{Training Procedure}) \]

Implication: Data engineering requires the same rigor as software engineering. Datasets should be versioned (like Git), unit-tested (data quality checks), and debugged. Deleting a row of training data can alter the rebuilt model just as deleting a line of code can alter a compiled artifact.

If data functions as source code, then it is not merely a logical artifact; it also has physical properties that constrain system architecture when data and computation occupy different locations.

Principle 2: The Data-Gravity Principle

Invariant: As data volume (\(D_{\text{vol}}\)) grows, transferring the dataset can cost more than sending a much smaller query or code payload over the same constrained link (McCrory 2010). Here, cost denotes one transfer metric at a time, such as elapsed transfer time or bytes sent, under identical link conditions; it does not combine latency and bandwidth into one quantity. \[ \text{Cost to transfer dataset} \gg \text{Cost to transfer query or code} \quad \text{in this regime} \]

Implication: Large datasets can become the gravitational center of the architecture. Systems may move compute to data by shipping queries or code to the storage layer rather than moving data to compute by repeatedly downloading large datasets.

McCrory, Dave. 2010. Data Gravity – in the Clouds.

Together, these two principles establish that data can be both the logical program and a physical anchor of an ML system. Part I develops these implications as a dependency chain. Deployment constraints define where a system can execute; the lifecycle carries those constraints through problem definition, data, modeling, validation, and operation; and data engineering makes them enforceable in versioned, tested pipeline artifacts. Ignoring any link returns complexity as integration cost: an accurate model that cannot execute, a workflow that cannot reproduce its inputs, or a pipeline whose outputs no longer match deployment conditions.