The harness comes first
Before a single layer is defined, there should be a script that takes a set of predictions and returns a scored report with fixed seeds, a pinned dataset hash and a per-slice breakdown.
Written first, the harness is neutral. Written afterwards, it tends to be quietly shaped by the numbers you already saw.
Where leakage hides
Leakage rarely announces itself. It arrives through duplicated rows across splits, through features computed over the full dataset before splitting, and through time-ordered data shuffled as if it were independent.
- Deduplicate before splitting, on content rather than on identifier.
- Fit every scaler, encoder and imputer inside the training fold only.
- Split time series by time, never at random.
- Keep a held-out set that is scored once, at the end.
Slices over averages
An aggregate metric is an average over populations that do not behave alike. Report by slice — language, device class, record length, source — and the model's real shape appears.
A two-point gain that comes entirely from the largest slice while a smaller slice regresses is usually not a gain worth shipping.
Reproducibility as a habit
Track the dataset version, the seed, the code commit and the environment for every run. A result you cannot reproduce next month is an anecdote, not a measurement.