The Evaluation Checklist

For imbalanced classification, accuracy lies. For regression with outliers, MSE may overweight them. Match the metric to the problem.

R² alone isn't enough. Report AIC or BIC when comparing models of different complexity.

Look at residual plots. Check normality. Check homoscedasticity. Fix problems you find before trusting coefficients.

Don't just report point estimates. Quantify uncertainty.

Look for Simpson's Paradox. Check whether your model's performance is consistent across important subgroups — fairness considerations live here too.

Single-split evaluation is noisy. K-fold cross-validation gives you a distribution of performance estimates.

When you claim Model A beats Model B, can you back it up? Use a paired test on cross-validation fold performance, with Bonferroni correction if testing many models.

Even if a difference is statistically significant, is it large enough to matter operationally? A 0.001 RMSE improvement after a month of engineering is statistically real and practically irrelevant.