When a model disappoints, the instinct is often to change several things at once: replace the algorithm, redesign the architecture, tune dozens of parameters and add a new feature set. That makes it difficult to know what caused any improvement—or deterioration.
A more disciplined approach is to hold the model architecture constant and test whether a new feature adds useful information. This isolates one question: did the representation of the data improve?
Separate the model from the inputs
A model pipeline has several distinct components: the label definition, eligible population, feature-generation process, model architecture, training configuration, validation design and decision threshold. Changing all of them at once creates an attribution problem.
Suppose the current model uses a fixed algorithm and hyperparameters. You want to test incremental volume from minute five to minute ten. Keep the label definition, training rows, folds, purge interval, model settings and threshold-selection procedure unchanged. Add the feature, rerun the same evaluation and compare the results.
This is not a perfect experiment if the feature itself changes the eligible population or creates missing values. Those effects must be documented. But it is much easier to interpret than a wholesale redesign.
Define the feature before training
Write down exactly how the feature is calculated, its unit, time window, missing-value behaviour and availability time. Confirm that it would exist at the moment the model is meant to make its prediction. A feature that uses information from after the prediction timestamp is leakage, even if its name sounds harmless.
For a volume feature, specify whether it is cumulative or incremental. For a ratio, document the numerator and denominator. For a wallet metric, explain which transactions and wallets are included. The same feature name should not silently mean different things in different versions of the pipeline.
Use a controlled comparison
Create a baseline run and a feature-added run. Use identical chronological folds and candidate thresholds. If the model is trained with randomness, fix and record the seeds, while recognising that one seed does not measure all variability.
Compare more than one metric. A new feature might improve PR-AUC but reduce precision at the operational threshold. It might increase recall but also increase the number of false positives. It could improve the average fold score while creating a severe failure in one period.
Also compare calibration, selected counts, missingness and feature stability. If the feature is unavailable or delayed in some cases, the model's offline result may not translate to the live data pipeline.
Avoid the hidden hyperparameter search
Even when the architecture stays unchanged, repeatedly trying feature variations can overfit the validation process. Record every trial, predefine the primary metric and keep the final holdout away from the selection process.
If the same holdout is used to decide that a feature should be added, it has become development data. A later period is then needed for a fresh evaluation.
A useful example from the BSC pipeline
The research already contained related volume representations: cumulative volume, a growth ratio and measures of buyer or liquidity activity. A sensible next test is to add or isolate incremental five-to-ten-minute volume while keeping the model configuration and chronological folds fixed. The purpose is to discover whether the additional feature contributes information beyond the existing baseline—not to assume that an intuitive feature must improve the model.
If the new feature is derived from existing columns, inspect multicollinearity and run ablation. It may be useful as a clearer representation even when it adds limited raw information, but that benefit should be demonstrated.
Version the feature calculation
A feature should have a stable name only when its meaning is stable. Store the formula version, lookback window, source fields, units and event cutoff with the model artefacts. When a calculation changes, treat it as a new feature version and compare it against the old implementation. This prevents a model from being evaluated on a column that has the same name but different semantics from the one used during training.
The practical lesson
Change one major component at a time. Define the feature precisely, keep the architecture and validation constant, compare fold-level and threshold-level results, record all trials and reserve a later period for confirmation. A small, interpretable experiment is more informative than a large change whose effects cannot be separated.