Beyond Accuracy: Demographic Feature Sufficiency in Imbalanced Transmission Preference Prediction
Abstract
Predicting customer preference from limited demographics can appear successful when target classes are highly skewed. This study evaluates whether normalized age and gender indicators are sufficient to predict vehicle transmission preference and whether conventional accuracy masks minority-class failure. A customer dataset of 223,042 records contains MANUAL 178,943, AUTOMATIC 35,810, CVT 8,283, and E-CVT 6. Because E-CVT is too sparse for reliable stratified validation, it is retained for descriptive analysis but excluded from the three-class predictive comparison. We combine association tests, mutual information, an empirical pattern-oracle analysis, and five-fold stratified cross-validation of a majority baseline, multinomial logistic regression, cost-sensitive logistic regression, and class-weighted random forest. Standard logistic regression reproduces the majority baseline, with accuracy 0.8023 but balanced accuracy 0.3333 and MCC 0.0000. Class-weighted random forest lowers accuracy to 0.4946 but improves balanced accuracy to 0.4535, Macro-F1 to 0.3399, and MCC to 0.0969. Age and gender jointly provide only 0.0300 bits of information, approximately 3.5 percent of target entropy. The findings show that high accuracy primarily reflects class prevalence; richer behavioral, product, and transaction features are required for practically useful preference prediction.
References
Abedin, M. Z., Guotai, C., Hajek, P., & Zhang, T. (2023). Combining weighted SMOTE with ensemble learning for the class-imbalanced prediction of small business credit risk. Complex & Intelligent Systems, 9, 3559–3579. https://doi.org/10.1007/s40747-021-00614-4
Araf, I., Idri, A., & Chairi, I. (2024). Cost-sensitive learning for imbalanced medical data: A review. Artificial Intelligence Review, 57(4), 80. https://doi.org/10.1007/s10462-023-10652-8
Chen, S.-X., Wang, X.-K., Zhang, H.-Y., & Wang, J.-Q. (2021). Customer purchase prediction from the perspective of imbalanced data: A machine learning framework based on factorization machine. Expert Systems with Applications, 173, 114756. https://doi.org/10.1016/j.eswa.2021.114756
Chen, W., Yang, K., Yu, Z., Shi, Y., & Chen, C. L. P. (2024). A survey on imbalanced learning: Latest research, applications and future directions. Artificial Intelligence Review, 57, 137. https://doi.org/10.1007/s10462-024-10759-6
Chicco, D., & Jurman, G. (2023). The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification. BioData Mining, 16, 4. https://doi.org/10.1186/s13040-023-00322-4
Chicco, D., Tötsch, N., & Jurman, G. (2021). The Matthews correlation coefficient (MCC) is more reliable than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix evaluation. BioData Mining, 14, 13. https://doi.org/10.1186/s13040-021-00244-z
He, J., & Cheng, M. X. (2021). Weighting methods for rare event identification from imbalanced datasets. Frontiers in Big Data, 4, 715320. https://doi.org/10.3389/fdata.2021.715320
Hicks, S. A., Strümke, I., Thambawita, V., Hammou, M., Riegler, M. A., Halvorsen, P., & Parasa, S. (2022). On evaluation metrics for medical applications of artificial intelligence. Scientific Reports, 12, 5979. https://doi.org/10.1038/s41598-022-09954-8
Jeffry, J., Usman, S., & Aziz, F. (2023). Analisis perilaku pelanggan menggunakan metode ensemble logistic regression. Jurnal Teknologi dan Ilmu Komputer Prima (JUTIKOMP), 6(2), 90–97. https://doi.org/10.34012/jutikomp.v6i2.4258
Jeffry, J., Azis, A. I. S., & Kandakon, E. T. J. (2025). Integrating Bayesian Optimization into Ensemble Logistic Regression for Explainable AI-Based Customer Behavior Analysis. Sinkron: Jurnal dan Penelitian Teknik Informatika, 9(4), 1900–1911. https://doi.org/10.33395/sinkron.v9i4.15219
Khatri, J., Marín-Morales, J., Moghaddasi, M., Guixeres, J., Giglioli, I. A. C., & Alcañiz, M. (2022). Recognizing personality traits using consumer behavior patterns in a virtual retail store. Frontiers in Psychology, 13, 752073. https://doi.org/10.3389/fpsyg.2022.752073
Kim, S., Shin, W., & Kim, H. W. (2024). Predicting online customer purchase: The integration of customer characteristics and browsing patterns. Decision Support Systems, 177, 114105. https://doi.org/10.1016/j.dss.2023.114105
Li, Y., Jia, X., Wang, R., Qi, J., Jin, H., Chu, X., & Mu, W. (2022). A new oversampling method and improved radial basis function classifier for customer consumption behavior prediction. Expert Systems with Applications, 199, 116982. https://doi.org/10.1016/j.eswa.2022.116982
Liu, Y., Li, B., Yang, S., & Li, Z. (2024). Handling missing values and imbalanced classes in machine learning to predict consumer preference: Demonstrations and comparisons to prominent methods. Expert Systems with Applications, 237, 121694. https://doi.org/10.1016/j.eswa.2023.121694
Naglik, I., & Lango, M. (2024). GMMSampling: A new model-based, data difficulty-driven resampling method for multi-class imbalanced data. Machine Learning, 113, 5183–5202. https://doi.org/10.1007/s10994-023-06416-8
Opitz, J. (2024). A closer look at classification evaluation metrics and a critical reflection of common evaluation practice. Transactions of the Association for Computational Linguistics, 12, 820–836. https://doi.org/10.1162/tacl_a_00675
Predić, B., Ćirić, M., & Stoimenov, L. (2023). Business purchase prediction based on XAI and LSTM neural networks. Electronics, 12(21), 4510. https://doi.org/10.3390/electronics12214510
Tanvir, A.-A., Khandokar, I. A., Islam, A. K. M. M., Islam, S., & Shatabda, S. (2023). A gradient boosting classifier for purchase intention prediction of online shoppers. Heliyon, 9(4), e15163. https://doi.org/10.1016/j.heliyon.2023.e15163
Vieira, P. M., & Rodrigues, F. (2024). An automated approach for binary classification on imbalanced data. Knowledge and Information Systems, 66, 2747–2767. https://doi.org/10.1007/s10115-023-02046-7
Zhao, D., Wang, X., Mu, Y., & Wang, L. (2021). Experimental study and comparison of imbalance ensemble classifiers with dynamic selection strategy. Entropy, 23(7), 822. https://doi.org/10.3390/e23070822.
