Accuracy, precision, recall, F1, ROC AUC, and average precision measure different aspects of a classifier. A model can score well on one metric and still fail the real objective, especially when the positive class is rare or false positives and false negatives have different costs.
Short answer: use accuracy only when classes and error costs are reasonably balanced. Use precision when false positives are expensive, recall when false negatives are expensive, F1 when both matter, ROC AUC for threshold-independent ranking, and average precision or the precision-recall curve when positive cases are rare.
Classification metrics at a glance
| Metric | What it answers | Useful when |
|---|---|---|
| Accuracy | What fraction of all predictions is correct? | Classes are balanced and errors have similar costs. |
| Precision | Of predicted positives, how many are truly positive? | False alarms are expensive. |
| Recall | Of actual positives, how many did the model find? | Missing a positive is expensive. |
| F1 | What is the harmonic balance of precision and recall? | You need one threshold-dependent score that considers both. |
| ROC AUC | How well does the model rank positives above negatives across thresholds? | You want threshold-independent ranking performance. |
| Average precision | How strong is precision across recall levels? | The positive class is rare and positive retrieval matters. |
Start with the confusion matrix
For binary classification, every prediction belongs to one of four groups:
- True positive (TP): positive case correctly predicted as positive.
- True negative (TN): negative case correctly predicted as negative.
- False positive (FP): negative case incorrectly flagged as positive.
- False negative (FN): positive case incorrectly predicted as negative.
Most threshold-dependent metrics are combinations of these four counts. Inspecting the confusion matrix often reveals more than reading one summary score.
Practical tutorial: Learn to build, normalize, plot, and interpret every cell in our Confusion Matrix in Python complete guide.
Accuracy
Accuracy is the fraction of correct predictions:
Accuracy = (TP + TN) / (TP + TN + FP + FN)
It is intuitive, but it can be misleading. If only 1% of observations are positive, a classifier that always predicts the negative class achieves 99% accuracy while finding no positive cases.
Precision
Precision measures the reliability of positive predictions:
Precision = TP / (TP + FP)
Choose precision when false positives create significant cost. Examples include unnecessary manual investigations, expensive follow-up tests, or blocking legitimate transactions.
Recall
Recall, also called sensitivity for the positive class, measures coverage of actual positives:
Recall = TP / (TP + FN)
Choose recall when false negatives are especially harmful, such as failing to detect a dangerous fault, security incident, or high-risk condition.
F1 score
F1 is the harmonic mean of precision and recall:
F1 = 2 × Precision × Recall / (Precision + Recall)
The harmonic mean penalizes a large imbalance between precision and recall. F1 ignores true negatives, so it should not be treated as a complete description of performance.
ROC AUC
The ROC curve plots true-positive rate against false-positive rate across decision thresholds. ROC AUC summarizes how well the model ranks positive observations above negative observations. A score near 0.5 indicates random-like ranking, while a score near 1 indicates strong separation.
ROC AUC is threshold-independent, but it does not tell you which operating threshold to use. It can also look strong even when the positive class is rare and the absolute number of false positives is operationally unacceptable.
Precision-recall curve and average precision
The precision-recall curve shows the trade-off between positive-prediction quality and positive-case coverage across thresholds. Average precision summarizes that curve by weighting precision improvements across recall increments.
For rare-positive problems, precision-recall analysis often provides a more focused picture than ROC because it concentrates on performance for the positive class.
Complete Python comparison
The following example creates an imbalanced binary-classification problem, evaluates several metrics using identical stratified folds, then reports held-out test performance.
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
average_precision_score,
classification_report,
confusion_matrix,
roc_auc_score,
)
from sklearn.model_selection import (
StratifiedKFold,
cross_validate,
train_test_split,
)
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
X, y = make_classification(
n_samples=6_000,
n_features=25,
n_informative=10,
n_redundant=5,
weights=[0.95, 0.05],
class_sep=1.0,
random_state=42,
)
X_train, X_test, y_train, y_test = train_test_split(
X,
y,
test_size=0.20,
stratify=y,
random_state=42,
)
model = Pipeline(
steps=[
("scaler", StandardScaler()),
(
"classifier",
LogisticRegression(
class_weight="balanced",
max_iter=2000,
random_state=42,
),
),
]
)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scoring = {
"accuracy": "accuracy",
"balanced_accuracy": "balanced_accuracy",
"precision": "precision",
"recall": "recall",
"f1": "f1",
"roc_auc": "roc_auc",
"average_precision": "average_precision",
}
scores = cross_validate(
model,
X_train,
y_train,
cv=cv,
scoring=scoring,
n_jobs=-1,
)
columns = [name for name in scores if name.startswith("test_")]
summary = pd.DataFrame(scores)[columns].agg(["mean", "std"]).T
print(summary.round(4))
model.fit(X_train, y_train)
test_probability = model.predict_proba(X_test)[:, 1]
test_prediction = model.predict(X_test)
print("Confusion matrix:")
print(confusion_matrix(y_test, test_prediction))
print(classification_report(y_test, test_prediction, digits=4))
print("ROC AUC:", roc_auc_score(y_test, test_probability))
print(
"Average precision:",
average_precision_score(y_test, test_probability),
)Plot ROC and precision-recall curves
Scikit-learn display objects can create both curves directly from predictions.
import matplotlib.pyplot as plt
from sklearn.metrics import (
PrecisionRecallDisplay,
RocCurveDisplay,
)
fig, axes = plt.subplots(1, 2, figsize=(12, 5))
RocCurveDisplay.from_predictions(
y_test,
test_probability,
ax=axes[0],
name="Logistic regression",
)
PrecisionRecallDisplay.from_predictions(
y_test,
test_probability,
ax=axes[1],
name="Logistic regression",
)
axes[0].set_title("ROC curve")
axes[1].set_title("Precision-recall curve")
plt.tight_layout()
plt.show()The decision threshold changes precision and recall
predict_proba returns scores, while predict converts them into class labels using the estimator's default decision rule. Changing the threshold changes the confusion matrix, precision, recall, and F1 without changing ROC AUC for the same scores.
import numpy as np
from sklearn.metrics import precision_score, recall_score, f1_score
for threshold in [0.2, 0.4, 0.5, 0.7, 0.9]:
prediction = (test_probability >= threshold).astype(int)
print(
{
"threshold": threshold,
"precision": precision_score(y_test, prediction),
"recall": recall_score(y_test, prediction),
"f1": f1_score(y_test, prediction),
}
)A lower threshold usually increases recall and decreases precision. A higher threshold usually increases precision and decreases recall. Do not choose a threshold on the final test set. Use training data with internal validation or cross-validation, then evaluate the chosen rule once on untouched test data.
How to choose the right metric
Fraud or anomaly alerts
When analysts can inspect only a limited number of alerts, precision and precision at a fixed alert budget matter. Recall shows how much harmful activity is still missed.
Safety or fault detection
If missing a dangerous event is unacceptable, prioritize recall while enforcing an operationally acceptable false-positive rate.
Marketing response
If outreach is inexpensive but opportunities are valuable, recall may dominate. If contacts are expensive or tightly limited, precision becomes more important.
Balanced benchmark classification
Accuracy can be useful when class sizes and error costs are similar, but report the confusion matrix and per-class metrics as a check.
Model ranking before threshold selection
ROC AUC and average precision compare probability rankings without fixing one threshold. Final deployment still requires an operating threshold based on costs and constraints.
Multiclass averaging
For multiclass problems, precision, recall, and F1 require an averaging strategy:
- Macro: calculates each class separately and gives every class equal weight.
- Weighted: averages class scores using class support.
- Micro: aggregates all class decisions before calculating the metric.
Macro F1 is especially useful when minority classes matter. Weighted F1 can remain high when the majority class dominates, so examine per-class results as well.
Common mistakes
- Reporting only accuracy for highly imbalanced data.
- Selecting the threshold using the test set.
- Comparing models on different folds or different test samples.
- Ignoring class prevalence when comparing results across datasets.
- Using ROC AUC as proof that deployed precision and recall will be acceptable.
- Optimizing one metric without considering operational costs.
- Reporting only an average for a multiclass problem and hiding weak classes.
Recommended reporting set
For most binary-classification projects, report:
- Class prevalence and test-set size.
- Confusion matrix at the chosen threshold.
- Precision, recall, and F1 for the positive class.
- ROC AUC and average precision from probability scores.
- The decision threshold and how it was selected.
- Cross-validation mean and variability.
- Results on the untouched final test set.
To compare tree ensembles, read Random Forest vs Gradient Boosting in Python. Browse the Machine Learning Tutorials hub for more workflows.
Frequently asked questions
Is F1 better than accuracy?
Not universally. F1 is often more informative when the positive class matters and classes are imbalanced, but it ignores true negatives. The right choice depends on the decision.
Does a high ROC AUC guarantee high precision?
No. Precision depends on the threshold and class prevalence. Always inspect precision-recall behavior and the confusion matrix at the intended operating threshold.
Should I use macro or weighted F1?
Use macro F1 when each class should count equally. Use weighted F1 when class frequency should influence the average, but also report per-class scores.
Can I optimize multiple metrics?
Yes. Cross-validation can return several metrics. Choose one explicit refit objective or use a decision rule that respects operational constraints.
Official references: model evaluation guide, precision-recall example, and decision-threshold guide.
No comments:
Post a Comment