Model validation is the process of testing whether a predictive model actually works — whether its predictions are accurate, calibrated and reliable in real-world conditions. It is the step that separates models that sound good from models that are good.
The Fundamental Problem
Any model can be made to fit the data it was trained on. With enough complexity, a model can memorize training examples rather than learning the underlying pattern. This produces excellent performance on training data and poor performance on new data — a phenomenon called overfitting. Validation exists to detect this problem before deployment, not after.
Hold-Out Validation
The most basic validation technique is hold-out validation: split your data into training and testing sets, train the model on the training set, and evaluate it on the testing set. The testing set should not be used in any way during training — not for feature selection, not for hyperparameter tuning, not for any decision that changes the model.
A common error is "leaking" testing data into the training process — using testing-set outcomes to guide model development, then evaluating on the same testing set. This produces falsely optimistic performance estimates.
Cross-Validation
Cross-validation (specifically k-fold cross-validation) improves on hold-out validation by training and evaluating across multiple splits of the data. This reduces the sensitivity of performance estimates to the specific random split chosen and provides more reliable estimates of how the model will perform on new data.
Beyond Accuracy: What Else to Validate
Accuracy is not the only important performance metric. For classification models, examine precision, recall and the tradeoff between them. For calibration, verify that predicted probabilities correspond to actual outcome rates. For fairness, examine whether the model performs comparably across relevant subgroups. For drift, test whether the model performs similarly on recent data as on older data.
See our predictive analytics guide and data signal evaluation guide for related practical guidance.