Disintegration of heterogeneous combat network based on double deep Q-learning

doi:10.23919/JSEE.2024.000063

Journal of Systems Engineering and Electronics ›› 2025, Vol. 36 ›› Issue (5): 1235-1246.doi: 10.23919/JSEE.2024.000063

• SYSTEMS ENGINEERING • Previous Articles

Disintegration of heterogeneous combat network based on double deep Q-learning

Wenhao CHEN(), Gang CHEN(), Jichao LI(), Jiang JIANG()

College of Systems Engineering, National University of Defense Technology, Changsha 410073, China

Received:2022-11-04 Online:2025-10-18 Published:2025-10-24
Contact: Jiang JIANG E-mail:chenwenhao14a@163.com;chengang@nudt.edu.cn;ljcnudt@hotmail.com;jiangjiangnudt@163.com
About author:
CHEN Wenhao was born in 1996. He received his M.E. degree in management science and engineering from National University of Defense Technology, Changsha, China, in 2021. He is an assistant engineer in PLA. His research interest focuses on mathematical modeling of heterogeneous information networks. E-mail: chenwenhao14a@163.com

CHEN Gang was born in 1997. He received his bachelor’s degree from China University of Mining and Technology, Xuzhou, China, in 2019. He is currently pursuing his Ph.D. degree at National University of Defense Technology, Changsha, China. His research interests focus on applying deep reinforcement learning to study game methods and applications of intelligent system-of-systems confrontation, intelligent optimization methods, and complex networks. E-mail: chengang@nudt.edu.cn

LI Jichao was born in 1990. He received his B.E. degree in management science, M.E. and Ph.D. degrees in management science and engineering from National University of Defense Technology, Changsha, China, in 2013, 2015, and 2019, respectively. He is currently an associate professor of management science and engineering at National University of Defense Technology. His research interests focus on studying complex systems with a combination of theoretical tool and data analysis, including mathematical modeling of heterogeneous information networks, applying network methodologies to analyze the development of complex system-of-systems, and data-driven studying of the collective behavior of humans. E-mail: ljcnudt@hotmail.com

JIANG Jiang was born in 1981. He received his Ph.D. degree in management science and engineering from National University of Defense Technology, Changsha, China, in 2011. He is an associate professor with the College of Systems Engineering at National University of Defense Technology. His research interests include evidential reasoning, uncertainty decision-making, and risk analysis. E-mail: jiangjiangnudt@163.com
Supported by:
This work was supported by the National Natural Science Foundation of China (72001209;72231011;72071206), the Science and Technology Innovative Research Team in Higher Educational Institutions of Hunan Province (2020RC4046), and the Science Foundation for Outstanding Youth Scholars of Hunan Province (2022JJ20047).

Abstract

Abstract:

The rapid development of military technology has prompted different types of equipment to break the limits of operational domains and emerged through complex interactions to form a vast combat system of systems (CSoS), which can be abstracted as a heterogeneous combat network (HCN). It is of great military significance to study the disintegration strategy of combat networks to achieve the breakdown of the enemy’s CSoS. To this end, this paper proposes an integrated framework called HCN disintegration based on double deep $Q$-learning (HCN-DDQL). Firstly, the enemy’s CSoS is abstracted as an HCN, and an evaluation index based on the capability and attack costs of nodes is proposed. Meanwhile, a mathematical optimization model for HCN disintegration is established. Secondly, the learning environment and double deep $Q$-network model of HCN-DDQL are established to train the HCN’s disintegration strategy. Then, based on the learned HCN-DDQL model, an algorithm for calculating the HCN’s optimal disintegration strategy under different states is proposed. Finally, a case study is used to demonstrate the reliability and effectiveness of HCN-DDQL, and the results demonstrate that HCN-DDQL can disintegrate HCNs more effectively than baseline methods.

Key words: heterogeneous combat network (HCN), combat system of systems (CSoS), network disintegration, reinforcement learning

Wenhao CHEN, Gang CHEN, Jichao LI, Jiang JIANG. Disintegration of heterogeneous combat network based on double deep Q-learning[J]. Journal of Systems Engineering and Electronics, 2025, 36(5): 1235-1246.

Figures/Tables 11

Table 1

Fig 1

Fig 2

Table 2

Fig 3

Fig 4

Table 3

Fig 5

Fig 6

Fig 7

Fig 8

References 41

1	JORDAN D, KIRAS J D, LONSDALE D J, et al. Understanding modern warfare. Cambridge: Cambridge University Press, 2016.
2	MOFFAT J. Complexity theory and network centric warfare. Philadelphia: Diane Publishing, 2010.
3	O’DONOUGHUE N A, MCBIRNEY S, PERSONS B. Distributed kill chains: drawing insights for mosaic warfare from the immune system and from the navy. Santa Monica: RAND Corporation, 2021.
4	CLARK B, PATT D, SCHRAMM H. Mosaic warfare: exploiting artificial intelligence and autonomous systems to implement decision-centric operations. Washington D.C.: Center for Strategic and Budgetary Assessments, 2020.
5	CHENG C, LI J C, ZHAO Q S, et al. Research on weapon system portfolio selection based on combat network modeling. Proc. of the Annual IEEE International Systems Conference, 2017. DOI: 10.1109/SYSCON.2017.7934733.
6	GAO C X, SU B, SUN M, et al Interprovincial transfer of embodied primary energy in China: a complex network approach. Applied Energy, 2018, 215, 792- 807. doi: 10.1016/j.apenergy.2018.02.075
7	LIU C, ARUNKUMAR N Risk prediction and evaluation of transnational transmission of financial crisis based on complex network. Cluster Computing, 2019, 22, 4307- 4313. doi: 10.1007/s10586-018-1870-3
8	LI J C, GE B F, ZHAO D L, et al Meta-path-based weapon-target recommendation in heterogeneous combat network. IEEE Systems Journal, 2019, 13 (4): 4433- 4141. doi: 10.1109/JSYST.2018.2890090
9	DING J Y, ZHAO Q S, LI J C, et al. Temporal constraint modeling and conflict resolving based on the combat process of air and missile defense system. Proc. of the IEEE International Conference on Systems, Man and Cybernetics, 2019. DOI: 10.1109/SMC.2019.8914484.
10	WAN Z Y, LI Y G, ZHANG Z Z. Modeling and analysis of integrated combat network system based on VOODAC. Proc. of the International Conference on Industrial Control Network and System Engineering Research, 2019: 81−86.
11	YANG Z F, LI Y G, LIU J Y. A method of node importance measurement base on community structure in heterogeneous combat networks. Proc. of the International Conference on Communications and Networking in China, 2021: 766−775.
12	CHEN Z H, WU J J, XIA Y X, et al. Robustness of interdependent power grids and communication networks: a complex network perspective. IEEE Trans. on Circuits and Systems II: Express Briefs, 2017, 65(1): 115−119.
13	FU C, WANG Y, WANG X Y, et al. Multi-node attack strategy of complex networks due to cascading breakdown. Chaos, Solitons & Fractals, 2018, 106: 61–66.
14	LI J C, ZHAO D L, GE B F, et al. Disintegration of operational capability of heterogeneous combat networks under incomplete information. IEEE Trans. on Systems, Man, and Cybernetics: Systems, 2018, 50(12): 5172–5179.
15	MALIK H A M, ABID F, WAHIDDIN M R, et al Robustness of dengue complex network under targeted versus random attack. Complexity, 2017, 2017, 2515928.
16	QIN L, WANG Y D, SUN Q, et al A network analysis of COVID-19 epidemic in mainland China by K-Core decomposition. JMIR Public Health and Surveillance, 2020, 6 (4): e24291. doi: 10.2196/24291
17	YANG L X, YANG X F, WU Y B The impact of patch forwarding on the prevalence of computer virus: a theoretical assessment approach. Applied Mathematical Modelling, 2017, 43, 110- 125. doi: 10.1016/j.apm.2016.10.028
18	ENGSTROM J. How the Chinese People’s Liberation Army seeks to wage modern warfare. Washington D.C.: RAND Corporation, 2018.
19	MALIK H A M, ABID F, WAHIDDIN M R, et al. Modeling of internal and external factors affecting a complex dengue network. Chaos, Solitons & Fractals, 2021, 144: 110694.
20	PRASSE B, VAN MIEGHEM P Network reconstruction and prediction of epidemic outbreaks for general group-based compartmental epidemic models. IEEE Trans. on Network Science and Engineering, 2020, 7 (4): 2755- 2764. doi: 10.1109/TNSE.2020.2987771
21	SINGH J A new analysis for fractional rumor spreading dynamical model in a social network with Mittag-Leffler law. Chaos: An Interdisciplinary Journal of Nonlinear Science, 2019, 29 (1): 013137. doi: 10.1063/1.5080691
22	YANG L, LI Z W, GIUA A Containment of rumor spread in complex social networks. Information Sciences, 2020, 506, 113- 130. doi: 10.1016/j.ins.2019.07.055
23	SMITH P, HUTCHISON D, STERBENZ J P G, et al Network resilience: a systematic approach. IEEE Communications Magazine, 2011, 49 (7): 88- 97. doi: 10.1109/MCOM.2011.5936160
24	LIU X M, LI D Q, MA M Q, et al Network resilience. Physics Reports, 2022, 971, 1- 108. doi: 10.1016/j.physrep.2022.04.002
25	DENG Y, WU J, XIAO Y, et al Efficient disintegration strategies with cost constraint in complex networks: the crucial role of nodes near average degree. Chaos: An Interdisciplinary Journal of Nonlinear Science, 2018, 28 (6): 061101. doi: 10.1063/1.5029984
26	KORKALI M, VENEMAN J G, TIVNAN B F, et al Reducing cascading failure risk by increasing infrastructure network interdependence. Scientific Reports, 2017, 7 (1): 44499. doi: 10.1038/srep44499
27	DOBSON S, HUTCHISON D, MAUTHE A, et al Self-organization and resilience for networked systems: design principles and open research issues. Proceedings of the IEEE, 2019, 107 (4): 819- 834. doi: 10.1109/JPROC.2019.2894512
28	ZHANG J L, LUO Y. Degree centrality, betweenness centrality, and closeness centrality in social network. Proc. of the 2nd International Conference on Modelling, Simulation and Applied Mathematics, 2017: 300−303.
29	ZHOU J, YU X H, LU J A. Node importance in controlled complex networks. IEEE Trans. on Circuits and Systems II: Express Briefs, 2018, 66(3): 437−441.
30	HALIM A H, ISMAIL I Combinatorial optimization: comparison of heuristic algorithms in travelling salesman problem. Archives of Computational Methods in Engineering, 2019, 26, 367- 380. doi: 10.1007/s11831-017-9247-y
31	YU Y, DENG Y, TAN S Y, et al Efficient disintegration strategy in directed networks based on tabu search. Physica A: Statistical Mechanics and Its Applications, 2018, 507, 435- 442. doi: 10.1016/j.physa.2018.05.079
32	LOZANO M, GARCIA-MARTINEZ C, RODRIGUEZ F J, et al Optimizing network attacks by artificial bee colony. Information Sciences, 2017, 377, 30- 50. doi: 10.1016/j.ins.2016.10.014
33	CARES J R. An information age combat model. Newport: Alidade Incorporated, 2004: 85−90.
34	TIRPAK J A Find, fix, track, target, engage, assess. Air Force Magazine, 2000, 83 (7): 24- 29.
35	ZHAO D L, TAN Y J, LI J C, et al Research on structural robustness of weapon system-of-systems based on heterogeneous network. Systems Engineering-Theory & Practice, 2019, 39 (12): 3197- 3207.
36	LUONG N C, HOANG D T, GONG S, et al Applications of deep reinforcement learning in communications and networking: a survey. IEEE Communications Surveys & Tutorials, 2019, 21 (4): 3133- 3174.
37	MAZYAVKINA N, SVIRIDOV S, IVANOV S, et al Reinforcement learning for combinatorial optimization: a survey. Computers & Operations Research, 2021, 134, 105400.
38	HASSELT H V, GUEZ A, SILVER D. Deep reinforcement learning with double Q-learning. Proc. of the 30th AAAI Conference on Artificial Intelligence. 2016. DOI: 10.1609/aaai.v30i1.10295.
39	MO H. M, GAO C, DENG Y. Evidential method to identify influential nodes in complex networks. Journal of Systems Engineering and Electronics, 2015, 26 (2): 381- 387.
40	BAGHEL M, AGRAWAL S, SILAKARI S Survey of metaheuristic algorithms for combinatorial optimization. International Journal of Computer Applications, 2012, 58 (19): 21- 31. doi: 10.5120/9391-3813
41	ZHANG X K, WU J, WANG H, et al. Optimization of disintegration strategy for multi-edges complex networks. Proc. of the IEEE Congress on Evolutionary Computation, 2016: 522−528.

Link type	Physical meaning
$S \to D$	Reconnaissance intelligence upload
$D \to I$	Fire strike order issued
$I \to S$	Fire strike feedback
${C_D} \to {C_D}$	Information sharing
${C_S} \to {C_S}$	Information sharing
${C_D} \to {C_S}$	Information sharing

Kill chain	Physical meaning
$S \to D \to I \to S$	A typical kill chain that does not involve information transmission and processing
$S \to {C_S} \to D \to I \to S$	Kill chain containing the information transmission and processing of sensor entity
$S \to {C_D} \to D \to I \to S$	Kill chain containing information transmission and processing of decision entity
$ S \to {C_D} \to {C_S} \to D \to I \to S $	Kill chain with multiple information transmission and processing

Method		Strategy
Random		Attack is carried out by randomly selecting nodes
Based on network static indices	DC	Preferentially attack nodes with greater degree centrality
	BC	Preferentially attack nodes with greater closeness centrality
	CC	Preferentially attack nodes with greater closeness centrality
	Cluster	Preferentially attack nodes with greater clustering coefficient
Based on heuristic algorithm	PSO	Consider HCN state vectors as particles, share information among particles, and find the problem’s solution by iteration. However, the disintegration order of the nodes cannot be determined.
Based on heuristic algorithm	GA	Considering the state vector of the HCN as the population’s genome, the solution is found by mechanisms such as selection, crossover, and mutation. However, the disintegration order of the nodes cannot be determined.

[1]	Siyu HENG, Ting CHENG, Zishu HE, Yuanqing WANG, Luqing LIU. Adaptive dwell scheduling based on Q-learning for multifunctional radar system [J]. Journal of Systems Engineering and Electronics, 2025, 36(4): 985-993.
[2]	Yifan ZHANG, Tao DONG, Zhihui LIU, Shichao JIN. Multi-QoS routing algorithm based on reinforcement learning for LEO satellite networks [J]. Journal of Systems Engineering and Electronics, 2025, 36(1): 37-47.
[3]	Nanxun DUO, Qinzhao WANG, Qiang LYU, Wei WANG. Tactical reward shaping for large-scale combat by multi-agent reinforcement learning [J]. Journal of Systems Engineering and Electronics, 2024, 35(6): 1516-1529.
[4]	Guofei LI, Shituo LI, Bohao LI, Yunjie WU. Deep reinforcement learning guidance with impact time control [J]. Journal of Systems Engineering and Electronics, 2024, 35(6): 1594-1603.
[5]	Qi WANG, Zhizhong LIAO. Computational intelligence interception guidance law using online off-policy integral reinforcement learning [J]. Journal of Systems Engineering and Electronics, 2024, 35(4): 1042-1052.
[6]	Guang ZHAN, Kun ZHANG, Ke LI, Haiyin PIAO. UAV maneuvering decision-making algorithm based on deep reinforcement learning under the guidance of expert experience [J]. Journal of Systems Engineering and Electronics, 2024, 35(3): 644-665.
[7]	Yaozhong ZHANG, Zhuoran WU, Zhenkai XIONG, Long CHEN. A UAV collaborative defense scheme driven by DDPG algorithm [J]. Journal of Systems Engineering and Electronics, 2023, 34(5): 1211-1224.
[8]	Jiawei XIA, Xufang ZHU, Zhong LIU, Qingtao XIA. LSTM-DPPO based deep reinforcement learning controller for path following optimization of unmanned surface vehicle [J]. Journal of Systems Engineering and Electronics, 2023, 34(5): 1343-1358.
[9]	Kewei YANG, Jichao LI, Maidi LIU, Tianyang LEI, Xueming XU, Hongqian WU, Jiaping CAO, Gaoxin QI. Complex systems and network science: a survey [J]. Journal of Systems Engineering and Electronics, 2023, 34(3): 543-573.
[10]	Yunxiu ZENG, Kai XU. Recognition and interfere deceptive behavior based on inverse reinforcement learning and game theory [J]. Journal of Systems Engineering and Electronics, 2023, 34(2): 270-288.
[11]	Yaozhong ZHANG, Yike LI, Zhuoran WU, Jialin XU. Deep reinforcement learning for UAV swarm rendezvous behavior [J]. Journal of Systems Engineering and Electronics, 2023, 34(2): 360-373.
[12]	Lu DONG, Zichen HE, Chunwei SONG, Changyin SUN. A review of mobile robot motion planning methods: from classical motion planning workflows to reinforcement learning-based architectures [J]. Journal of Systems Engineering and Electronics, 2023, 34(2): 439-459.
[13]	Guangran CHENG, Lu DONG, Xin YUAN, Changyin SUN. Reinforcement learning-based scheduling of multi-battery energy storage system [J]. Journal of Systems Engineering and Electronics, 2023, 34(1): 117-128.
[14]	Peng LIU, Boyuan XIA, Zhiwei YANG, Jichao LI, Yuejin TAN. A deep reinforcement learning method for multi-stage equipment development planning in uncertain environments [J]. Journal of Systems Engineering and Electronics, 2022, 33(6): 1159-1175.
[15]	Bohao LI, Yunjie WU, Guofei LI. Hierarchical reinforcement learning guidance with threat avoidance [J]. Journal of Systems Engineering and Electronics, 2022, 33(5): 1173-1185.

Disintegration of heterogeneous combat network based on double deep Q-learning

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

Share this article

Figures/Tables 11

References 41

Related Articles 15

Recommended Articles

Metrics

Comments