A model's predictions on data it has already trained on are not a fair estimate of how it will behave on unseen observations. Training data has influenced the fitted model, so its scores can be overly optimistic. Out-of-fold predictions provide a way to create predictions for training rows using models that did not train on those rows.
They are especially important when one model's predictions become input features for another model.
The basic idea
Divide the development data into folds. For each fold, train the model on the other eligible training observations and predict the held-out fold. Repeat until each eligible row has a prediction from a model that did not train on it.
The result is an out-of-fold prediction column. Every row has been scored out of sample relative to its own fold, making the predictions more suitable for evaluating a training pipeline or constructing a second-stage dataset.
In ordinary random cross-validation, folds are often shuffled. That may be acceptable for independent and identically distributed data, but it can be inappropriate when observations are time-ordered and labels look into the future.
Respect time order
For a time-dependent prediction problem, the validation fold should come after the training period. An expanding-window design might train on the earliest period, score the next period, then expand the training window and score a later period.
If a label uses a future outcome horizon, observations near the fold boundary can overlap. A multi-hour outcome window, for example, may extend from a training observation into the validation period. Purging removes training observations whose label windows overlap the validation interval. The purge length should be derived from the actual label horizon and data timing, not selected arbitrarily.
The earliest observations may not receive OOF predictions because there is not enough earlier data to train a model. That is expected. Report how many rows receive OOF predictions and how many remain training-only.
Why OOF matters for meta-labelling
Suppose Stage 1 predicts whether a launch is a candidate and Stage 2 uses the Stage 1 score as an input. If Stage 2 is trained using Stage 1 in-sample scores, those scores may be more optimistic than the scores Stage 2 will receive later. Stage 2 can learn a relationship that disappears when Stage 1 is applied to unseen data.
Instead, construct Stage 2 training rows from Stage 1 OOF predictions. Each row's Stage 1 score must come from a model that did not train on that row. Preserve the candidate threshold and time logic consistently.
For final testing, fit Stage 1 on the permitted development period, apply it to the later holdout and pass those genuine out-of-sample scores to Stage 2. Do not use holdout labels to choose the Stage 2 model or threshold.
What OOF does not solve
OOF predictions reduce one important form of leakage, but they do not guarantee a sound experiment. The folds can still be too small, the features can contain future information, the labels can be wrong, or the validation population can differ from the eventual use case.
Any preprocessing that learns from data—imputation values, scaling parameters, feature selection, target encoding or dimensionality reduction—must be fitted within the training fold and applied to the held-out fold. Otherwise, information can leak even when the model itself is trained correctly.
Audit the OOF process
Record the fold boundaries, training and validation counts, positive and negative counts, purge interval, model configuration and number of rows with predictions. Check that no row is scored by a model that included its label in training. Verify that the OOF dates are in chronological order.
The BSC project used expanding chronological folds and a purge matched to the label horizon. The earliest development observations did not receive OOF predictions because they were used to establish the first training window. That is a normal consequence of the design, not automatically a data-loss error.
Verify every OOF row mechanically
Keep a fold-assignment table with each row’s timestamp, fold identifier, training cutoff, validation interval and whether the row received an OOF score. Add assertions that the row timestamp is inside the validation interval and that the model’s training cutoff precedes it. Check that no validation label is present in the fold’s training data. A mechanical audit is more dependable than assuming the fold code is correct because the final metrics look plausible.
The practical lesson
OOF predictions are a bridge between model training and honest downstream evaluation. Generate them with correct fold boundaries, keep all learned preprocessing inside each fold and document uncovered rows. They are especially important whenever a second model learns from the first model's scores.