Confidence-Aware Machine Learning for Imbalanced and Noisy Datasets: A Joint Framework for Calibration, Minority-Class Recall, and Label-Noise Robustness

Author

Neetu

Abstract

 Real-world classification tasks rarely provide the clean, balanced training data assumed by standard supervised learning pipelines; instead, they combine skewed class distributions with unreliable, noisily annotated labels. Existing research has largely treated class imbalance, confidence calibration, and label-noise robustness as separate problems, producing methods that improve one property while silently degrading another a re-weighted loss that improves minority-class recall, for instance, can simultaneously amplify the influence of mislabeled examples and worsen calibration. This paper proposes Confidence-Aware Adaptive Learning (CAAL), a unified classification framework that jointly estimates prediction confidence, routes uncertain or suspect-noisy samples to a noise-adaptive loss re-weighting mechanism, and applies a dedicated minority-class recall booster within a single, iteratively updated training loop. The framework couples a shared feature encoder with a lightweight confidence-estimation head, enabling per-sample decisions about how much a given label should be trusted during optimization. A structured literature review spanning imbalance-handling, calibration, and noisy-label research identifies a concrete gap: no existing framework jointly evaluates confidence calibration, minority-class recall, and robustness to label noise under a shared experimental protocol. Building on this gap, the paper presents a proposed system architecture, a mathematical formulation of the training objective, and a detailed research methodology  including controlled imbalance-noise experimental grids, component-wise ablation design, and calibration-aware evaluation metrics such as Expected Calibration Error and Area Under the ROC Curve  intended to guide empirical validation of the framework on benchmark and real-world datasets.

Keywords

confidence calibration; class imbalance; label noise; robust learning; uncertainty estimation; deep neural networks.

Full Text:

Download Paper PDF

References

[1] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, "SMOTE: Synthetic minority over-sampling technique," Journal of Artificial Intelligence Research, vol. 16, pp. 321–357, 2002.

[2] H. He and E. A. Garcia, "Learning from imbalanced data," IEEE Transactions on Knowledge and Data Engineering, vol. 21, no. 9, pp. 1263–1284, Sep. 2009.

[3] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, "Focal loss for dense object detection," in Proc. IEEE Int. Conf. Computer Vision (ICCV), Venice, Italy, 2017, pp. 2980–2988.

[4] Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie, "Class-balanced loss based on effective number of samples," in Proc. IEEE/CVF Conf. Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 9268–9277.

[5] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, "On calibration of modern neural networks," in Proc. 34th Int. Conf. Machine Learning (ICML), Sydney, Australia, 2017, pp. 1321–1330.

[6] A. Kendall and Y. Gal, "What uncertainties do we need in Bayesian deep learning for computer vision?," in Advances in Neural Information Processing Systems (NeurIPS), vol. 30, Long Beach, CA, USA, 2017.

[7] B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama, "Co-teaching: Robust training of deep neural networks with extremely noisy labels," in Advances in Neural Information Processing Systems (NeurIPS), vol. 31, Montréal, Canada, 2018, pp. 8527–8537.

[8] Z. Zhang and M. R. Sabuncu, "Generalized cross entropy loss for training deep neural networks with noisy labels," in Advances in Neural Information Processing Systems (NeurIPS), vol. 31, Montréal, Canada, 2018, pp. 8778–8788.

[9] C. G. Northcutt, L. Jiang, and I. L. Chuang, "Confident learning: Estimating uncertainty in dataset labels," Journal of Artificial Intelligence Research, vol. 70, pp. 1373–1411, 2021.

[10] H. Song, M. Kim, D. Park, Y. Shin, and J.-G. Lee, "Learning from noisy labels with deep neural networks: A survey," IEEE Transactions on Neural Networks and Learning Systems, vol. 34, no. 11, pp. 8135–8153, Nov. 2023.

Share your valuable work from Social Media Buttons