The Impact of Feature Selection on Machine Learning Model Performance
DOI:
https://doi.org/10.38035/gcir.v1i2.882Keywords:
Feature Selection, Machine Learning, Data Science, Classification, Model EvaluationAbstract
Feature selection can reduce input dimensionality and improve interpretability, but its effect on predictive performance depends on the data and learning algorithm. This study experimentally compares three classifiers Logistic Regression, Support Vector Machine (SVM) with a radial basis function kernel, and Random Forest under three feature configurations using the Wisconsin Breast Cancer Diagnostic dataset. A stratified five-fold cross-validation design was used, with scaling and feature selection fitted only within each training fold to reduce data leakage. The conditions included all 30 predictors, the 10 highest-ranked predictors by ANOVA F-score, and the 10 highest-ranked predictors by mutual information. Mean accuracy, precision, recall, F1-score, and ROC-AUC were recorded, treating malignant cases as the positive class. SVM with all features achieved the highest mean accuracy (97.72%) and F1-score (0.9688), while Logistic Regression with all features achieved the highest ROC-AUC (0.9953). Reducing the input to 10 features lowered mean accuracy for all three classifiers, although the magnitude of the change varied by model. The findings indicate that feature selection should be evaluated jointly with the classifier and the intended performance objective rather than assumed to improve prediction automatically. Because the experiment uses one benchmark dataset, the results are methodological evidence rather than clinical validation.
References
Aamir, S., et al. (2022). Predicting breast cancer leveraging supervised machine learning techniques. Computational and Mathematical Methods in Medicine, 2022, Article 5869529. https://doi.org/10.1155/2022/5869529
Bommert, A., Sun, X., Bischl, B., Rahnenführer, J., & Lang, M. (2020). Benchmark for filter methods for feature selection in high-dimensional classification data. Computational Statistics & Data Analysis, 143, 106839. https://doi.org/10.1016/j.csda.2019.106839
Büyükkeçeci, M., & Okur, M. C. (2023). A comprehensive review of feature selection and feature selection stability in machine learning. Gazi University Journal of Science, 36(4), 1506–1520. https://doi.org/10.35378/gujs.993763
Cawley, G. C., & Talbot, N. L. C. (2010). On over-fitting in model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research, 11, 2079–2107. https://www.jmlr.org/papers/v11/cawley10a.html
Chandrashekar, G., & Sahin, F. (2014). A survey on feature selection methods. Computers & Electrical Engineering, 40(1), 16–28. https://doi.org/10.1016/j.compeleceng.2013.11.024
Chicco, D., & Jurman, G. (2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 21, Article 6. https://doi.org/10.1186/s12864-019-6413-7
Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20, 273–297. https://doi.org/10.1007/BF00994018
Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. https://doi.org/10.1016/j.patrec.2005.10.010
Guyon, I., & Elisseeff, A. (2003). An introduction to variable and feature selection. Journal of Machine Learning Research, 3, 1157–1182. https://www.jmlr.org/papers/v3/guyon03a.html
Kohavi, R., & John, G. H. (1997). Wrappers for feature subset selection. Artificial Intelligence, 97(1–2), 273–324. https://doi.org/10.1016/S0004-3702(97)00043-X
Kuzudisli, C., Bakir-Gungor, B., Bulut, N., Qaqish, B. F., & others. (2023). Review of feature selection approaches based on grouping of features. PeerJ, 11, e15666. https://doi.org/10.7717/peerj.15666
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, É. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830. https://www.jmlr.org/papers/v12/pedregosa11a.html
Pudjihartono, N., Fadason, T., Kempa-Liehr, A. W., & O’Sullivan, J. M. (2022). A review of feature selection methods for machine learning-based disease risk prediction. Frontiers in Bioinformatics, 2, Article 927312. https://doi.org/10.3389/fbinf.2022.927312
Saeys, Y., Inza, I., & Larrañaga, P. (2007). A review of feature selection techniques in bioinformatics. Bioinformatics, 23(19), 2507–2517. https://doi.org/10.1093/bioinformatics/btm344
Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE, 10(3), e0118432. https://doi.org/10.1371/journal.pone.0118432
Sharma, A., & Mishra, P. K. (2022). Performance analysis of machine learning based optimized feature selection approaches for breast cancer diagnosis. International Journal of Information Technology, 14, 1603–1611. https://doi.org/10.1007/s41870-021-00671-5
Street, W. N., Wolberg, W. H., & Mangasarian, O. L. (1993). Nuclear feature extraction for breast tumor diagnosis. In Proceedings of the SPIE Symposium on Electronic Imaging: Science and Technology (Vol. 1905, pp. 861–870). SPIE. https://doi.org/10.1117/12.148698
Strelcenia, E., & Prakoonwit, S. (2023). Effective feature engineering and classification of breast cancer diagnosis: A comparative study. BioMedInformatics, 3(3), 42. https://doi.org/10.3390/biomedinformatics3030042
Urbanowicz, R. J., Meeker, M., La Cava, W., Olson, R. S., & Moore, J. H. (2018). Relief-based feature selection: Introduction and review. Journal of Biomedical Informatics, 85, 189–203. https://doi.org/10.1016/j.jbi.2018.07.014
Varma, S., & Simon, R. (2006). Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics, 7, Article 91. https://doi.org/10.1186/1471-2105-7-91
Breiman, L. (2001). Random forests. Machine Learning, 45, 5–32. https://doi.org/10.1023/A:1010933404324
Yu, L., & Liu, H. (2004). Efficient feature selection via analysis of relevance and redundancy. Journal of Machine Learning Research, 5, 1205–1224. https://www.jmlr.org/papers/v5/yu04a.html
Refaeilzadeh, P., Tang, L., & Liu, H. (2009). Cross-validation. In L. Liu & M. T. Özsu (Eds.), Encyclopedia of Database Systems (pp. 532–538). Springer. https://doi.org/10.1007/978-0-387-39940-9_565
