Adding another feature can make a model look more sophisticated, but it does not guarantee better predictions. A new column may duplicate information already present, introduce noise, create leakage risk or encourage overfitting. Feature selection is therefore not simply a search for the largest possible feature set.
A good feature set should represent the problem clearly and improve performance on data the model has not seen.
What feature selection is trying to achieve
Feature selection is the process of choosing which inputs a model should use. The goals may include better generalisation, faster training, simpler interpretation, lower data requirements or reduced operational complexity.
There are several broad approaches. Filter methods rank features using statistics such as correlation or mutual information. Wrapper methods evaluate subsets by repeatedly training models. Embedded methods select or shrink features during training, as in L1-regularised models or tree-based methods.
Each approach has limitations. A univariate filter can miss interactions. A wrapper can be expensive and may overfit the validation process if it tests too many combinations. Embedded importance is specific to the model and data used to calculate it.
Ablation asks a more practical question
Instead of asking which feature has the highest importance score, ablation asks: what happens when a feature or a coherent group is removed?
Start with a baseline feature set. Train and evaluate it using a fixed chronological validation design. Then remove one feature group at a time—for example, volume structure, liquidity, wallet participation, concentration or early selling. Compare the changes in PR-AUC, ROC-AUC, precision, recall, selected count and fold-to-fold stability.
Ablation reveals whether a feature group contributes information beyond the rest of the model. If performance remains unchanged after removing a group, the group may be redundant. If performance falls consistently, the group may be useful. If performance rises, the group may be adding noise or encouraging overfitting.
Keep the experiment fair
Use the same folds, candidate population, label definition and evaluation metrics for each comparison. If feature selection uses a statistic estimated from data, fit that statistic on the training fold only. Never choose features using the final holdout.
This matters especially when the pipeline includes a first-stage model. If Stage 2 uses Stage 1 predictions, generate out-of-fold Stage 1 scores for the Stage 2 training rows. Otherwise, feature selection may reward the Stage 2 model for relying on scores that were produced in-sample.
Record all tests, not only the winners. If dozens of combinations are tried, the best result may partly reflect chance. A pre-defined ablation plan reduces the temptation to keep searching until a favourable number appears.
Interpreting feature importance carefully
Tree-based importance can be split among correlated features. Logistic coefficients are conditional on other included predictors and can become unstable when features overlap. Permutation importance can also be misleading when correlated columns substitute for one another. No one importance score is a universal measure of a feature's value.
In the BSC Stage 2 LightGBM run, buyer concentration, volume-service share, unique buyers, early selling, wash share and liquidity appeared near the top of one feature-importance ranking. That ranking describes how that particular fitted model used its inputs. It does not prove that these are universally the most important drivers or that they will remain so in later periods.
Prefer stability to a single peak score
A feature group that gives a small improvement in every fold may be more valuable than one that produces a large gain in a single period and hurts in others. Report fold-level results and the final chronological holdout separately. Check whether the selected candidate count changes dramatically.
Also consider operational cost. A feature that requires unreliable data or long delays may not be useful even if it improves offline metrics slightly. The feature set must be available at the prediction time and calculated consistently.
Keep an experiment ledger
For every feature-set run, record a unique run name, included and excluded columns, code version, data version, fold definitions, random seed, model settings and all evaluation metrics. Include unsuccessful experiments. This makes it possible to distinguish a genuine improvement from a change caused by different rows, thresholds or preprocessing. The ledger also limits accidental cherry-picking when many versions have been tried.
The practical lesson
Feature selection should reduce uncertainty, not create a leaderboard of lucky experiments. Use domain-based groups, test them with ablation, prevent leakage, record every trial and prioritise performance that repeats across chronological folds. More features are useful only when they add information that generalises.