A Comparative Study of Machine Learning and Classical Statistical Models for Predictive Data Analytics
Abstract
Predictive data analytics increasingly combines classical statistical reasoning with flexible machine learning algorithms, yet model selection is often driven by novelty rather than evidence. This study compares representative statistical and machine learning models on two reproducible structured-data tasks: diabetes progression regression (n = 442, 10 predictors) and breast-cancer diagnosis classification (n = 569, 30 predictors). Ordinary least squares, ridge regression, logistic regression, and linear discriminant analysis were evaluated against radial-basis-function support vector machines, random forests, and gradient boosting. All pipelines used repeated 10-fold cross-validation with three repetitions, standardized preprocessing where required, and identical folds within each task. On the regression benchmark, ridge regression achieved the strongest mean result (RMSE = 54.73; R² = 0.465), narrowly ahead of ordinary least squares and the RBF support-vector regressor. On the classification benchmark, RBF-SVM achieved the highest ROC-AUC (0.996), while logistic regression provided nearly equivalent discrimination (0.995) with simpler coefficients and lower computational burden. Tree ensembles did not dominate either small structured dataset. The findings demonstrate that predictive superiority is conditional on sample size, signal geometry, objective function, interpretability needs, and validation discipline. A practical model-selection framework is proposed that treats classical models as strong baselines and escalates complexity only when verified out-of-sample improvement justifies the additional operational cost.
Keywords
Machine Learning, Classical Statistical Models, Predictive Data Analytics