
Journal of Systems Engineering and Electronics ›› 2026, Vol. 37 ›› Issue (1): 184-196.doi: 10.23919/JSEE.2026.000053
• SYSTEMS ENGINEERING • Previous Articles Next Articles
Xuemei YAO(
), Jianbin SUN(
), Zituo LI(
), Kewei YANG(
)
Received:2024-04-19
Online:2026-02-18
Published:2026-03-09
Contact:
Jianbin SUN
E-mail:yaoxuemei28@126.com;sunjianbin@nudt.edu.cn;lizituo0926@163.com;kayyang27@nudt.edu.cn
About author:Xuemei YAO, Jianbin SUN, Zituo LI, Kewei YANG. Adversarial robustness evaluation based on classification confidence-based confusion matrix[J]. Journal of Systems Engineering and Electronics, 2026, 37(1): 184-196.
Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks
Table 1
Basic information of the testing dataset"
| Dataset | Number of samples | Number of categories | Number of attributes | Attribute type |
| Cancer | 569 | 2 | 30 | Continuous |
| Spambase | 2 | 57 | Continuous | |
| Magic | 2 | 10 | Continuous | |
| Setap | 768 | 2 | 84 | Mixing |
| Iris | 150 | 3 | 4 | Discrete |
| Wine | 178 | 3 | 13 | Continuous |
| Robot | 4 | 24 | Continuous | |
| Absenteeism | 669 | 6 | 20 | Mixing |
Table 2
Visual evaluation index based on classification confidence-based confusion matrix"
| Classification algorithm | Attack algorithm | Indicator | Cancer | Spambase | Magic | Setap | Iris | Wine | Robot | Absenteeism |
| ANN | FGSM | 0.701 | 0.953 | 0.439 | 0.957 | 0.331 | 0.264 | 0.287 | 0.101 | |
| 0.701 | 0.953 | 0.439 | 0.957 | 0.498 | 0.437 | 1.443 | 1.468 | |||
| DeepFool | 0.981 | 0.953 | 0.359 | 0.973 | 0.820 | 0.933 | 0.418 | 0.104 | ||
| 0.981 | 0.953 | 0.359 | 0.973 | 2.115 | 2.595 | 2.148 | 1.394 | |||
| PGD | 0.385 | 0.180 | 0.000 | 0.183 | 0.220 | 0.290 | 0.201 | 0.115 | ||
| 0.385 | 0.180 | 0.000 | 0.183 | 0.340 | 0.519 | 0.759 | 0.949 | |||
| LR | FGSM | 0.389 | 0.133 | 0.169 | 0.143 | 0.262 | 0.359 | 0.389 | 0.220 | |
| 0.389 | 0.133 | 0.169 | 0.143 | 0.458 | 0.688 | 1.658 | 2.564 | |||
| DeepFool | 0.393 | 0.345 | 0.128 | 0.178 | 0.390 | 0.367 | 0.422 | 0.221 | ||
| 0.393 | 0.345 | 0.128 | 0.178 | 0.709 | 0.697 | 1.794 | 2.569 | |||
| PGD | 0.413 | 0.105 | 0.149 | 0.147 | 0.262 | 0.367 | 0.297 | 0.220 | ||
| 0.413 | 0.105 | 0.149 | 0.147 | 0.458 | 0.797 | 1.174 | 2.557 | |||
| SVM | FGSM | 0.377 | 0.364 | 0.165 | 0.137 | 0.310 | 0.355 | 0.385 | 0.167 | |
| 0.377 | 0.364 | 0.165 | 0.137 | 0.632 | 0.734 | 1.794 | 2.453 | |||
| DeepFool | 0.393 | 0.322 | 0.117 | 0.153 | 0.431 | 0.374 | 0.401 | 0.175 | ||
| 0.393 | 0.322 | 0.117 | 0.153 | 0.313 | 0.338 | 0.233 | 0.174 | |||
| PGD | 0.331 | 0.342 | 0.146 | 0.130 | 0.854 | 0.784 | 1.862 | 2.555 | ||
| 0.331 | 0.342 | 0.146 | 0.130 | 0.686 | 0.809 | 1.022 | 2.545 | |||
| CNN | FGSM | 0.463 | 0.166 | 0.244 | 0.412 | 0.316 | 0.325 | 0.364 | 0.270 | |
| 0.463 | 0.166 | 0.244 | 0.412 | 0.964 | 0.983 | 1.434 | 2.803 | |||
| DeepFool | 0.555 | 0.882 | 0.465 | 0.763 | 0.407 | 0.531 | 0.587 | 0.375 | ||
| 0.555 | 0.882 | 0.465 | 0.763 | 1.158 | 1.464 | 3.280 | 6.187 | |||
| PGD | 0.454 | 0.052 | 0.214 | 0.171 | 0.309 | 0.431 | 0.266 | 0.220 | ||
| 0.454 | 0.052 | 0.214 | 0.171 | 0.959 | 0.845 | 0.917 | 2.068 | |||
| Transformer | FGSM | 0.468 | 0.246 | 0.153 | 0.451 | 0.312 | 0.479 | 0.388 | 0.319 | |
| 0.468 | 0.246 | 0.153 | 0.451 | 0.587 | 1.003 | 1.703 | 3.559 | |||
| DeepFool | 0.666 | 0.819 | 0.437 | 0.686 | 0.922 | 0.782 | 0.736 | 0.579 | ||
| 0.666 | 0.819 | 0.437 | 0.686 | 2.486 | 2.115 | 4.430 | 9.368 | |||
| PGD | 0.442 | 0.109 | 0.107 | 0.089 | 0.213 | 0.210 | 0.314 | 0.276 | ||
| 0.442 | 0.109 | 0.107 | 0.089 | 0.321 | 0.409 | 1.574 | 2.938 |
Table 3
Different $ {\bf{esp}} $ corresponding to ${\boldsymbol{ \Delta}} {{\boldsymbol{S}}}_{\bf{conf}} $ and ${\boldsymbol{ \Delta}} {{\bf{acc}}}_{\bf{conf}} $"
| 0.01 | 0.203 | 0.304 |
| 0.05 | 0.220 | 0.330 |
| 0.10 | 0.252 | 0.379 |
| 0.20 | 0.331 | 0.498 |
| 0.25 | 0.376 | 0.567 |
| 0.50 | 0.601 | 0.917 |
| 0.75 | 0.731 | 1.319 |
Table 4
$ {\boldsymbol{\Delta}} {{\boldsymbol{S}}} _{\bf{conf}} $ and $ {\boldsymbol{\Delta}} {{\bf{acc}}}_{\bf{conf}} $ of the cancer dataset"
| Classification algorithm | FGSM | DeepFool | PGD | |||||
| ANN | 0.701 | 0.701 | 0.981 | 0.981 | 0.385 | 0.385 | ||
| LR | 0.389 | 0.389 | 0.393 | 0.393 | 0.413 | 0.413 | ||
| SVM | 0.377 | 0.377 | 0.393 | 0.393 | 0.331 | 0.331 | ||
| CNN | 0.463 | 0.463 | 0.555 | 0.555 | 0.454 | 0.454 | ||
| Transformer | 0.468 | 0.468 | 0.666 | 0.666 | 0.442 | 0.442 | ||
| 7 |
ZHOU J P, LI W, XIA Q L, et al Robust missile autopilot design based on dynamic surface control. Journal of Systems Engineering and Electronics, 2023, 34 (1): 160- 171.
doi: 10.23919/JSEE.2022.000154 |
| 8 | SZEGEDY C, ZAREMBA W, SUTSKEVER I, et al. Intriguing properties of neural networks. https://arxiv.org/abs/1312.6199. |
| 9 | GOODFELLOW I J, SHLENS J, SZEGEDY C. Explaining and harnessing adversarial examples. https://arxiv.org/abs/1412.6572. |
| 10 | KURAKI A, GOODFELLOW I, BENGIO S. Adversarial examples in the physical world. https://arxiv.org/abs/1607.02533. |
| 11 | MOOSAVI-DEZFOOLI S M, FAWZI A, FROSSARD P. DeepFool: a simple and accurate method to fool deep neural networks. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 2574−2582. |
| 12 |
SU J, VARGAS D V, SAKURAI K One pixel attack for fooling deep neural networks. IEEE Trans. on Evolutionary Computation, 2019, 23 (5): 828- 841.
doi: 10.1109/TEVC.2019.2890858 |
| 13 | JIA X J, WEI X X, CAO X C, et al. Adv-watermark: a novel watermark perturbation for adversarial examples. Proc. of the 28th ACM International Conference on Multimedia, 2020: 1579−1587. |
| 14 | JIANG H, YANG J T, HUA G, et al FAWA: fast adversarial watermark attack. IEEE Trans. on Computers, 2021, 1 (1): 1- 14. |
| 15 |
ROSHAN M K, ZAFAR A Boosting robustness of network intrusion detection systems: a novel two phase defense strategy against untargeted white-box optimization adversarial attack. Expert Systems with Applications, 2024, 249, 123567.
doi: 10.1016/j.eswa.2024.123567 |
| 16 | DONG Y F. Research on robustness of intrusion detection method in adversarial environment. Changsha: National University of Defense Technology, 2018. (in Chinese) |
| 17 |
LIU A S, LIU X L, YU H, et al Training robust deep neural networks via adversarial noise propagation. IEEE Trans. on Image Processing, 2021, 30, 5769- 5781.
doi: 10.1109/TIP.2021.3082317 |
| 18 |
PAN W W, WANG X Y, SONG M L, et al Review of adversarial sample generation techniques. Journal of Software, 2020, 31 (1): 67- 81.
doi: 10.1051/itmconf/20257802013 |
| 19 | XU Y J, SUN H, LEI L, et al Robustness of convolutional neural network for SAR ship recognition based on adversarial attack. Signal Processing, 2020, 36 (12): 1965- 1978. |
| 20 | CHEN S H, SHEN H J, WANG R, et al Research on the relationship between prediction uncertainty and adversarial robustness. Journal of Software, 2022, 33 (2): 524- 538. |
| 21 |
LIU A S, LIU X L, GUO J, et al A comprehensive evaluation framework for deep model robustness. Pattern Recognition, 2023, 137, 109308.
doi: 10.1016/j.patcog.2023.109308 |
| 22 | LUO B, LIU Y N, WEI L X, et al. Towards imperceptible and robust adversarial sample attacks against neural networks. https://arxiv.org/abs/1801.04693. |
| 23 | DONG Y P, LIAO F Z, PANG T Y, et al. Boosting adversarial attacks with momentum. Proc. of the IEEE /CVF Conference on Computer Vision and Pattern Recognition, 2018: 9185−9193. |
| 24 |
YUAN X Y, HE P, ZHU Q L, et al Adversarial examples: attacks and defenses for deep learning. IEEE Trans. on Neural Networks and Learning Systems, 2019, 30 (9): 2805- 2824.
doi: 10.1109/TNNLS.2018.2886017 |
| 25 | CARLINI N, WAGNER D. Towards evaluating the robustness neural networks. Proc. of the Symposium on Security and Privacy, 2017: 39−57. |
| 26 | ZHANG S C, LI J Y KNN classification with one-step computation. IEEE Trans. on Knowledge and Data Engineering, 2023, 35 (3): 2711- 2723. |
| 27 | WANG W W, DUAN Y, CAO L H, et al. Application of improved Naive Bayes classification algorithm in 5G signaling analysis. The Journal of Supercomputing, 2023, 79(6): 6941−6964. |
| 28 |
SUN Z G, WANG G T, LI P F, et al An improved random forest based on the classification accuracy and correlation measurement of decision trees. Expert Systems with Applications, 2024, 237, 121549.
doi: 10.1016/j.eswa.2023.121549 |
| 29 |
XIAO L, ZHANG Z L, HUANG K, et al Noise optimization in artificial neural networks. IEEE Trans. on Automation Science and Engineering, 2024, 22, 2780- 2793.
doi: 10.1109/case49997.2022.9926712 |
| 30 | CORNILLY D, TUBEX L, VAN AELST S, et al Robust and sparse logistic regression. Advances in Data Analysis and Classification, 2023, 18 (3): 663- 679. |
| 1 | LI Z T, SUN J B, YANG K W, et al Review on adversarial robustness evaluation for image classification. Journal of Computer Research and Development, 2022, 59 (10): 2164- 2189. |
| 2 | LING X, JI S L, ZOU J X, et al. DEEPSEC: a uniform platform for security analysis of deep learning model. Proc. of the IEEE Symposium on Security and Privacy, 2019: 673−690. |
| 3 | ZHANG C Z, LIU A S, LIU X L, et al Interpreting and improving adversarial robustness of deep neural networks with neuron sensitivity. IEEE Trans. on Image Processing, 2020, 30, 1291- 1304. |
| 4 | WENG T W, ZHANG H, CHEN P Y, et al. Evaluating the robustness of neural networks: an extreme value theory approach. https://arxiv.org/abs/1801.10578. |
| 5 | MA L, XU J F, ZHANG F Y, et al. DeepGauge: multi-granularity testing criteria for deep learning systems. Proc. of the 33rd ACM/IEEE International Conference on Automated Software Engineering, 2018: 120−131. |
| 6 |
WANG Z, BOVIK A C, SHEIKH H R, et al Image quality assessment: from error visibility to structural similarity. IEEE Trans. on Image Processing, 2004, 13 (4): 600- 612.
doi: 10.1109/TIP.2003.819861 |
| 31 |
CHEN C, XU J T, NI J, et al An intelligent broaching tool design method based on CBR and support vector machine. Advanced Engineering Informatics, 2024, 60, 102447.
doi: 10.1016/j.aei.2024.102447 |
| 32 | LEC Y, BOSER B, DENKER J S, et al. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1989, 1(4): 541–551. |
| 33 | VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need. Proc. of the Advances in Neural Information Processing Systems, 2017: 5998−6008. |
| 34 | DUA D, GRAFF C. UCI machine learning repository. http://archive.ics.uci.edu/ml. |
| No related articles found! |
| Viewed | ||||||
|
Full text |
|
|||||
|
Abstract |
|
|||||