Cross-validation estimates how well a machine-learning model will perform on unseen data by training and evaluating it across several data splits. This guide explains K-fold, stratified, grouped, repeated, and time-series cross-validation in scikit-learn, with reproducible Python examples and practical rules for avoiding data leakage.
Why Use Cross-Validation?
A single train/test split can produce a misleading result. The measured score depends on which observations happen to enter the test set. An easy split may make a weak model look strong, while a difficult split may hide the value of a good model.
Cross-validation reduces this dependence on one split. In five-fold cross-validation, the data is divided into five parts. The model trains on four folds and validates on the remaining fold. This process repeats until every fold has served as validation data once.
- Divide the development data into k folds.
- Train on k − 1 folds.
- Evaluate on the remaining fold.
- Repeat until every fold has been used for validation.
- Report the mean score and its variability across folds.
Quick Regression Example with KFold
The example below evaluates Ridge regression on the diabetes dataset. Preprocessing is placed inside a scikit-learn Pipeline, ensuring that scaling is fitted separately within every training fold.
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Ridge
from sklearn.model_selection import KFold, cross_val_score
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_diabetes(return_X_y=True)
model = Pipeline([
("scale", StandardScaler()),
("ridge", Ridge(alpha=1.0))
])
cv = KFold(
n_splits=5,
shuffle=True,
random_state=42
)
scores = cross_val_score(
model,
X,
y,
cv=cv,
scoring="neg_mean_absolute_error"
)
mae = -scores
print("MAE by fold:", mae)
print("Mean MAE:", mae.mean())
print("MAE standard deviation:", mae.std())With this configuration, the mean absolute error is approximately 44.24. Your goal is not to memorize that number, but to understand that every observation is evaluated by a model that was not trained on that observation.
Why Are Error Scores Negative?
Scikit-learn follows a “higher is better” convention for scorer objects. Losses such as MAE and mean squared error are therefore returned as negative values. Multiply the scores by −1 before reporting them as familiar positive errors.
rmse_scores = cross_val_score(
model,
X,
y,
cv=cv,
scoring="neg_root_mean_squared_error"
)
rmse_scores = -rmse_scores
print("Mean RMSE:", rmse_scores.mean())Classification with StratifiedKFold
Ordinary K-fold splitting does not explicitly preserve class proportions. For binary or multiclass classification, StratifiedKFold attempts to keep approximately the same class distribution in every fold.
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_validate
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
X, y = load_breast_cancer(return_X_y=True)
model = Pipeline([
("scale", StandardScaler()),
("logistic", LogisticRegression(max_iter=2000))
])
cv = StratifiedKFold(
n_splits=5,
shuffle=True,
random_state=42
)
results = cross_validate(
model,
X,
y,
cv=cv,
scoring=["accuracy", "balanced_accuracy", "roc_auc"],
n_jobs=-1,
return_train_score=True
)
for metric in ["accuracy", "balanced_accuracy", "roc_auc"]:
values = results[f"test_{metric}"]
print(metric, values.mean(), values.std())cross_validate() is useful when you need several metrics, training scores, fit times, or scoring times. For an in-depth discussion of metric selection, read Accuracy vs Precision vs Recall vs F1 vs ROC AUC.
Choosing the Correct Cross-Validation Splitter
| Splitter | Use it when | Main protection |
|---|---|---|
KFold | Regression or approximately independent observations | Every sample is validated once |
StratifiedKFold | Binary or multiclass classification | Preserves class proportions |
RepeatedKFold | You want a more stable estimate across multiple random partitions | Reduces dependence on one fold assignment |
RepeatedStratifiedKFold | Classification with repeated partitions | Repetition plus class balance |
GroupKFold | Multiple rows belong to the same person, machine, site, or experiment | Keeps groups from crossing train and validation sets |
StratifiedGroupKFold | Grouped classification with imbalanced classes | Non-overlapping groups plus approximate class balance |
TimeSeriesSplit | Chronologically ordered observations | Prevents training on future data and testing on the past |
LeaveOneOut | Very small datasets where computational cost is acceptable | Uses all but one sample for each training run |
Cross-Validation and Data Leakage
Cross-validation is only trustworthy when every learned preprocessing step is fitted inside each training fold. Leakage occurs if information from a validation fold influences scaling, imputation, feature selection, encoding, PCA, resampling, or target-based transformations.
Wrong approach
# Do not scale the full dataset before cross-validation.
X_scaled = StandardScaler().fit_transform(X)
scores = cross_val_score(
LogisticRegression(),
X_scaled,
y,
cv=5
)Correct approach
model = Pipeline([
("scale", StandardScaler()),
("logistic", LogisticRegression(max_iter=2000))
])
scores = cross_val_score(
model,
X,
y,
cv=cv,
scoring="roc_auc"
)Grouped Cross-Validation
Random splitting is invalid when several rows come from the same entity. For example, repeated measurements from one patient, turbine, building, or manufacturing batch are correlated. If one entity appears in both training and validation data, performance can look unrealistically high.
import numpy as np
from sklearn.model_selection import GroupKFold
X = np.arange(200).reshape(100, 2)
y = np.arange(100) % 2
# Ten measurements for each of ten subjects.
groups = np.repeat(np.arange(10), 10)
cv = GroupKFold(n_splits=5)
for fold, (train_idx, valid_idx) in enumerate(
cv.split(X, y, groups=groups),
start=1
):
train_groups = set(groups[train_idx])
valid_groups = set(groups[valid_idx])
print(f"Fold {fold}")
print("Shared groups:", train_groups & valid_groups)The set of shared groups should be empty in every fold. For grouped classification where class balance also matters, consider StratifiedGroupKFold.
Time-Series Cross-Validation
Random K-fold cross-validation usually breaks chronology. It can train on future observations and validate on earlier observations, which creates temporal leakage. TimeSeriesSplit uses earlier data for training and later data for validation.
import numpy as np
from sklearn.model_selection import TimeSeriesSplit
X = np.arange(40).reshape(20, 2)
cv = TimeSeriesSplit(
n_splits=4,
gap=1
)
for fold, (train_idx, valid_idx) in enumerate(
cv.split(X),
start=1
):
print(
f"Fold {fold}:",
f"train {train_idx[0]}-{train_idx[-1]}",
f"validation {valid_idx[0]}-{valid_idx[-1]}"
)The optional gap excludes observations between the end of training and the start of validation. This is useful when features or labels are influenced by nearby time periods.
Out-of-Fold Predictions with cross_val_predict
cross_val_predict() returns one prediction for every observation from a model that did not train on that observation. These out-of-fold predictions are valuable for diagnostic plots, confusion matrices, and stacking.
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import cross_val_predict
y_pred_oof = cross_val_predict(
model,
X,
y,
cv=cv,
n_jobs=-1
)
cm = confusion_matrix(y, y_pred_oof)
print(cm)For binary or multiclass interpretation and visualization, continue with the Confusion Matrix in Python complete guide.
model, X, y, and cv should refer to the same classification workflow.How Many Folds Should You Use?
| Choice | Advantage | Trade-off |
|---|---|---|
| 5 folds | Good general-purpose balance of stability and speed | Each model trains on 80% of the data |
| 10 folds | More training data in each run | Approximately twice as many fits as five-fold CV |
| Repeated 5-fold | Measures sensitivity to several random partitions | Can become computationally expensive |
| Leave-one-out | Maximum training data per split | Very expensive and often high-variance |
Five-fold cross-validation is a strong default for many tabular machine-learning problems. Increase the number of folds only when the dataset, computational budget, and evaluation objective justify it.
Reproducibility and Shuffling
For independent tabular observations, using shuffle=True prevents fold quality from depending on the original row order. Set random_state to an integer so the split can be reproduced.
cv = StratifiedKFold(
n_splits=5,
shuffle=True,
random_state=42
)Do not shuffle time-series data. For grouped data, the group constraint matters more than random row order.
Parallel Cross-Validation
Many scikit-learn validation functions accept n_jobs. Setting n_jobs=-1 allows all available CPU cores to process independent fits, subject to memory limits and the estimator's own parallelism.
results = cross_validate(
model,
X,
y,
cv=cv,
scoring=["accuracy", "roc_auc"],
n_jobs=-1
)Avoid uncontrolled nested parallelism. If both cross-validation and the estimator use every CPU core, the machine may create too many workers. In that case, parallelize one level and limit the other.
Cross-Validation for Hyperparameter Tuning
GridSearchCV and RandomizedSearchCV evaluate candidate hyperparameters using cross-validation. Pass the correct splitter to the cv parameter and keep preprocessing inside the searched pipeline.
from sklearn.model_selection import GridSearchCV
search = GridSearchCV(
estimator=model,
param_grid={
"logistic__C": [0.01, 0.1, 1.0, 10.0]
},
scoring="roc_auc",
cv=cv,
n_jobs=-1
)
search.fit(X, y)
print("Best parameters:", search.best_params_)
print("Best CV score:", search.best_score_)When you report an unbiased estimate after intensive tuning, use a separate test set or nested cross-validation. The inner loop chooses hyperparameters; the outer loop estimates performance.
Common Cross-Validation Mistakes
- Preprocessing the full dataset before splitting. This leaks validation information.
- Using KFold for time series. Random or ordinary folds can train on the future.
- Splitting repeated entities across folds. Use grouped validation.
- Optimizing on the final test set. The test set must remain untouched.
- Using the wrong metric. Accuracy may be misleading for imbalanced classes.
- Reporting only the mean. Include standard deviation or fold-level results.
- Ignoring failed fits. Investigate warnings and use
error_score="raise"during debugging. - Comparing models on different folds. Use the same splitter and random seed for fair comparison.
- Oversampling before cross-validation. Resampling must happen inside each training fold.
- Creating too much parallelism. Coordinate
n_jobsacross validation and estimator levels.
Practical Checklist
- Identify whether observations are independent, grouped, or time-ordered.
- Select the splitter that matches the data-generating process.
- Place all learned preprocessing inside a pipeline.
- Choose metrics before comparing models.
- Use identical folds for fair model comparison.
- Report mean, variability, and individual fold results.
- Reserve a final test set whenever possible.
- Record the splitter, fold count, seed, scoring, and library version.
Frequently Asked Questions
Is five-fold cross-validation enough?
It is a strong default for many tabular problems. Repeated validation can improve stability when the dataset is small or results are sensitive to the split.
Should I stratify regression data?
Standard StratifiedKFold is designed for class labels. For regression, use KFold unless the data requires grouped or temporal splitting.
Can cross-validation overfit?
Yes. Repeatedly choosing models and hyperparameters based on the same cross-validation results can overfit the validation process. Use a final test set or nested cross-validation for a less biased estimate.
Can I use cross-validation with Random Forest?
Yes. The same principles apply. Compare ensemble methods under identical folds using the workflow in Random Forest vs Gradient Boosting in Python.
