Journal of Systems Engineering and Electronics ›› 2026, Vol. 37 ›› Issue (1): 184-196.doi: 10.23919/JSEE.2026.000053

• SYSTEMS ENGINEERING • Previous Articles     Next Articles

Adversarial robustness evaluation based on classification confidence-based confusion matrix

Xuemei YAO(), Jianbin SUN(), Zituo LI(), Kewei YANG()   

  • Received:2024-04-19 Online:2026-02-18 Published:2026-03-09
  • Contact: Jianbin SUN E-mail:yaoxuemei28@126.com;sunjianbin@nudt.edu.cn;lizituo0926@163.com;kayyang27@nudt.edu.cn
  • About author:
    YAO Xuemei was born in 2000. She received her B.E. degree from National University of Defense Technology in 2023. Now she is pursuing her Ph.D. degree at National University of Defense Technology. Her main research interests include adversarial robustness evaluation, test and evaluation, and reinforcement learning. E-mail: yaoxuemei28@126.com

    SUN Jianbin was born in 1989. He received his B.E., M.E. and Ph.D. degrees from National University of Defense Technology in 2012, 2014, and 2018 respectively. He is currently an associate professor and M.E. supervisor with National University of Defense Technology. His research interests include system test and evaluation, decision and analysis under uncertainty. E-mail: sunjianbin@nudt.edu.cn

    LI Zituo was born in 1999. He received his B.E. degree from Huazhong Agricultural University in 2021, and M.E. degree from National University of Defense Technology in 2024. Now he is pursuing his Ph.D. degree at National University of Defense Technology. His main research interests include adversarial robustness evaluation, test and evaluation, and deep learning. E-mail: lizituo0926@163.com

    YANG Kewei was born in 1977. He received his B.E., M.E. and Ph.D. degrees from National University of Defense Technology in 1999, 2001, and 2004 respectively. He is currently a professor and Ph.D. supervisor with National University of Defense Technology. His main research interests include test and evaluation, defense acquisition and system of systems requirement modeling. E-mail: kayyang27@nudt.edu.cn

Abstract:

Evaluating the adversarial robustness of classification algorithms in machine learning is a crucial domain. However, current methods lack measurable and interpretable metrics. To address this issue, this paper introduces a visual evaluation index named confidence centroid skewing quadrilateral, which is based on a classification confidence-based confusion matrix, offering a quantitative and visual comparison of the adversarial robustness among different classification algorithms, and enhances intuitiveness and interpretability of attack impacts. We first conduct a validity test and sensitive analysis of the method. Then, prove its effectiveness through the experiments of five classification algorithms including artificial neural network (ANN), logistic regression (LR), support vector machine (SVM), convolutional neural network (CNN) and transformer against three adversarial attacks such as fast gradient sign method (FGSM), DeepFool, and projected gradient descent (PGD) attack.

Key words: adversarial robustness evaluation, visual evaluation, classification confidence-based confusion matrix, centroid skewing