A formation pursuit method integrated coordinated reciprocity for enhanced capture

doi:10.23919/JSEE.2026.000016

Abstract

Abstract:

Cooperative pursuit poses challenges across natural, social, and technical systems, particularly when decentralized, slow-speed pursuers attempt to capture a high-speed evader with limited observation. Most existing contributions place the focus on the greedy pursuit of the evader, overlooking potential collaborations among pursuers. To tackle this issue, a decision-making framework of multi-agent coordinated reciprocity formation pursuit (MACRFP) via deep reinforcement learning is introduced. This framework integrates the actor-critic algorithm with the coordinated reciprocity mechanism to enhance the capability of capturing a faster evader. Initially, a local perception model is created by utilizing a cellular network to simulate limitations caused by obstacles. Next, the formation coalition of pursuit is guided by the Cartesian Oval, enabling dispersed pursuers to create a siege against the faster evader. Furthermore, a coordinated reciprocity model based on the coordination graph and the attention-based graph neural networks is developed, addressing the global coordination problem by estimating a reciprocity coefficient to adjust agents’ rewards. Numerical simulations demonstrate the emergence of cooperative behaviors in cooperative besiegement, target tracking, and intelligent interception during the pursuit, indicating that the proposed algorithm enhances the feasibility and effectiveness of capturing a fast-escaping target by integrating coordinated reciprocity and coalition formation.

Key words: multi-agent system, reinforcement learning, cooperative pursuit, coordinated reciprocity

Xiaoyu XING, Haoxiang XIA. A formation pursuit method integrated coordinated reciprocity for enhanced capture[J]. Journal of Systems Engineering and Electronics, 2026, 37(1): 211-224.

Figures/Tables 12

Fig 1

Fig 2

Fig 3

Fig 4

Fig 5

Fig 6

Fig 7

Fig 8

Fig 9

Fig 10

Fig 11

Fig 12

References 45

1	WANG H B, ZHANG Y Impulsive maneuver strategy for multi-agent orbital pursuit-evasion game under sparse rewards. Aerospace Science and Technology, 2024, 155, 109618. doi: 10.1016/j.ast.2024.109618
2	ZHANG C, LIU T, BAI G H, et al. A dynamic resilience evaluation method for cross-domain swarms in confrontation. Reliability Engineering & System Safety, 2023: 109904.
3	SUN H, YAN S, LIANG Y, et al Memory-extraction-based DRL cooperative guidance against the maneuvering target protected by interceptors. Aerospace Science and Technology, 2024, 155, 109575. doi: 10.1016/j.ast.2024.109575
4	WANG X T, YANG M, WANG S Y, et al Linear-quadratic and norm-bounded combined differential game guidance scheme with obstacle avoidance for attacking defended aircraft in three-player engagement. Defence Technology, 2024, 42, 136- 155. doi: 10.1016/j.dt.2024.06.018
5	HUA X, LIU J, ZHANG J J, et al An apollonius circle based game theory and Q-learning for cooperative hunting in unmanned aerial vehicle cluster. Computers and Electrical Engineering, 2023, 110, 108876. doi: 10.1016/j.compeleceng.2023.108876
6	GURUMURTHY V, MOHANTY N, SUNDARAM S, et al An efficient reinforcement learning scheme for the confinement escape problem. Applied Soft Computing, 2024, 152, 111248. doi: 10.1016/j.asoc.2024.111248
7	GAO M J, YAN T, HAN B J, et al Cooperative guidance law based on super-twisting observer for target maneuvering. Journal of Systems Engineering and Electronics, 2024, 35 (5): 1304- 1314.
8	BAI Y, ZHOU D, ZHANG B L, et al Two-to-one differential game via improved MOGWO. Journal of Systems Engineering and Electronics, 2025, 36 (1): 233- 255.
9	GARCIA E, BOPARDIKAR S D. Cooperative containment of a high-speed evader. Proc. of the American Control Conference, 2021: 4698−4703.
10	XIONG H, ZHANG Y Reinforcement learning-based formation-surrounding control for multiple quadrotor UAVs pursuit-evasion games. ISA Transactions, 2024, 145, 205- 224. doi: 10.1016/j.isatra.2023.12.006
11	CHEN J, ZHA W Z, PENG Z H, et al Multi-player pursuit-evasion games with one superior evader. Automatica, 2016, 71, 24- 32.
12	YAO M, DENG H G, FENG X Y, et al Improved dynamic windows approach based on energy consumption management and fuzzy logic control for local path planning of mobile robots. Computers & Industrial Engineering, 2023, 187, 109767. doi: 10.1016/j.cie.2023.109767
13	SHENG H L, ZHANG J, YAN Z Y, et al New multi-UAV formation keeping method based on improved artificial potential field. Chinese Journal of Aeronautics, 2023, 36 (11): 249- 270. doi: 10.1016/j.cja.2023.07.030
14	CAO P, LEI L, CAI S S, et al Computational intelligence algorithms for UAV swarm networking and collaboration: a comprehensive survey and future directions. IEEE Communications Surveys & Tutorials, 2024, 26 (4): 2684- 2728. doi: 10.1109/COMST.2024.3395358
15	ZHANG C Q, ZHOU W J, QIN W D, et al A novel UAV path planning approach: heuristic crossing search and rescue optimization algorithm. Expert Systems with Applications, 2023, 215, 119243. doi: 10.1016/j.eswa.2022.119243
16	ZHAO Y H, QU Z S, LIU H C, et al Bio-inspired robot swarm path formation with local sensor scope. Applied Intelligence, 2023, 53, 17310- 17326. doi: 10.1007/s10489-022-04356-9
17	XIE J L, MA K M Suboptimal guidance law against maneuvering target with time and angle constraints. Aerospace Science and Technology, 2024, 148, 109111. doi: 10.1016/j.ast.2024.109111
18	JIN W Q, TIAN X W, SHI B H, et al Enhanced UAV pursuit-evasion using boids modelling: a synergistic integration of bird swarm intelligence and DRL. Computers, Materials and Continua, 2024, 80 (3): 1546- 2218. doi: 10.32604/cmc.2024.055125
19	SCALA F, GAIAS G, COLOMBO C, et al Design of optimal low-thrust manoeuvres for remote sensing multi-satellite formation flying in low earth orbit. Advances in Space Research, 2021, 68 (11): 4359- 4378. doi: 10.1016/j.asr.2021.09.030
20	YANG Y J, EL MOCTAR O A mathematical model for ships maneuvering in deep and shallow waters. Ocean Engineering, 2024, 295, 116927. doi: 10.1016/j.oceaneng.2024.116927
21	SHAMSOSHOARA A, LOTFI F, MOUSAVI S, et al Joint path planning and power allocation of a cellular-connected UAV using apprenticeship learning via deep inverse reinforcement learning. Computer Networks, 2024, 254, 110789. doi: 10.1016/j.comnet.2024.110789
22	ZHANG J D, YANG Q M, SHI G Q, et al UAV cooperative air combat maneuver decision based on multi-agent reinforcement learning. Journal of Systems Engineering and Electronics, 2021, 32 (6): 1421- 1438. doi: 10.23919/jsee.2021.000121
23	WANG Y D, DONG L, SUN C Y Cooperative control for multi-player pursuit-evasion games with reinforcement learning. Neurocomputing, 2020, 412, 101- 114. doi: 10.1016/j.neucom.2020.06.031
24	DU W B, GUO T, CHEN J, et al Cooperative pursuit of unauthorized UAVs in urban airspace via multi-agent reinforcement learning. Transportation Research Part C: Emerging Technologies, 2021, 128, 103122. doi: 10.1016/j.trc.2021.103122
25	XIA J W, ZHU X F, ZHANG J Q, et al Research on cooperative hunting method of unmanned surface vehicle based on multi-agent reinforcement learning. Control and Decision, 2023, 38 (5): 1438- 1447.
26	TANG N B, WANG X L, GAO S, et al Collaborative ship scheduling decision model for green tide salvage based on evolutionary population dynamics. Ocean Engineering, 2024, 304, 117796. doi: 10.1016/j.oceaneng.2024.117796
27	ZHANG Z, WANG X H, ZHANG Q R, et al. Multi-robot cooperative pursuit via potential field-enhanced reinforcement learning. Proc. of the International Conference on Robotics and Automation, 2022: 8808−8814.
28	XIE S Y, ZHANG A, BI W H, et al Multi-UAV mission allocation under constraint. Applied Science, 2019, 9 (11): 2184. doi: 10.3390/app9112184
29	ZENG Y, WU Q Q, ZHANG R R Accessing from the sky: a tutorial on UAV communications for 5G and beyond. Proceedings of the IEEE, 2019, 107 (12): 2327- 2375. doi: 10.1109/JPROC.2019.2952892
30	ZENG Y, RUI Z, LIM T J Throughput maximization for UAV-enabled mobile relaying systems. IEEE Trans. on Communications, 2016, 64 (12): 4983- 4996. doi: 10.1109/TCOMM.2016.2611512
31	ELNABTY I A, FAHMY Y, KAFAFY M A survey on UAV placement optimization for UAV-assisted communication in 5G and beyond networks. Physical Communication, 2022, 51, 101564. doi: 10.1016/j.phycom.2021.101564
32	ARIF M, SHAKOOR A Clustered jamming and antenna beam-width fluctuations for UAV-assisted cellular networks. Computer Networks, 2024, 240, 110171. doi: 10.1016/j.comnet.2024.110171
33	WAN F Y, YASEEN M B, RIAZ M B, et al Advancements and challenges in UAV-based communication networks: a comprehensive scholarly analysis. Results in Engineering, 2024, 24, 103271. doi: 10.1016/j.rineng.2024.103271
34	YANG J N, CHEN J J, YANG Z L. Energy-efficient UAV communication with trajectory optimization. Proc. of the 2nd International Conference on Big Data & Artificial Intelligence & Software Engineering, 2021: 508−514.
35	ZHANG S W, ZENG Y Z, ZHANG R. Cellular-enabled UAV communication: Trajectory optimization under connectivity constraint. Proc. of the IEEE International Conference on Communications, 2018. DOI: 10.1109/ICC.2018.8422584.
36	ESRAFILIAN O, GANGULA R, GESBERT D. 3D-map assisted UAV trajectory design under cellular connectivity constraints. Proc. of the IEEE International Conference on Communications, 2020. DOI: 10.1109/ICC40277.2020.9149190.
37	CHEN J T, YATNALLI U, GESBERT D. Learning radio maps for UAV-aided wireless networks: a segmented regression approach. Proc. of the IEEE International Conference on Communications, 2017. DOI: 10.1109/ICC.2017.7997333.
38	CHEN J T, GESBERT D. Optimal positioning of flying relays for wireless networks: a LOS map approach. Proc. of the IEEE International Conference on Communications, 2017. DOI: 10.1109/ICC.2017.7996921.
39	FU X W, WANG H, XU Z Research on cooperative pursuit strategy for multi-UAVs based on DE-MADDPG algorithm. Acta Aeronautica et Astronautica Sinica, 2021, 42, 325311.
40	FU X W, XU Z, ZHU J D, et al Research on maneuvering decision-making of multi-UAV attack-defence confrontation based on PER-MATD3. Acta Aeronautica et Astronautica Sinica, 2023, 43 (2): 327083.
41	LOWE R, WU Y, TAMAR A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments. Proc. of the 31st International Conference on Neural Information Processing Systems, 2017: 6379−6390.
42	RASHID T, SAMVELYAN M, SCHROEDER C, et al. QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning. Proc. of the 35th International Conference on Machine Learning, 2018: 4295−304.
43	LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning. Proc. of the 3rd International Conference on Learning Representations, 2015. DOI: 10.1016/S1098-3015(10)67722−4.
44	FUJIMOTO S, VAN HOOF H, MEGER D. Addressing function approximation error in actor-critic methods. Proc. of the 35th International Conference on Machine Learning, 2018: 1587−1596.
45	HANEDA K, TIAN L, ZHENG Y. 5G 3GPP-like channel models for outdoor urban microcellular and macrocellular environments. Proc. of the IEEE 83rd Vehicular Technology Conference, 2016. DOI: 10.1109/VTCSpring.2016.7503971.

[1]	Shijie DENG, Yingxin KOU, Maolong LYU, Zhanwu LI, An XU. λ-return-based aircraft maneuvering for terminal defense and positioning guidance strategies [J]. Journal of Systems Engineering and Electronics, 2025, 36(6): 1692-1708.
[2]	Wenhao CHEN, Gang CHEN, Jichao LI, Jiang JIANG. Disintegration of heterogeneous combat network based on double deep Q-learning [J]. Journal of Systems Engineering and Electronics, 2025, 36(5): 1235-1246.
[3]	Rui ZHOU, Weichao ZHONG, Wenlong LI, Hao ZHANG. Self-play training and analysis for GEO inspection game with modular actions [J]. Journal of Systems Engineering and Electronics, 2025, 36(5): 1353-1373.
[4]	Siyu HENG, Ting CHENG, Zishu HE, Yuanqing WANG, Luqing LIU. Adaptive dwell scheduling based on Q-learning for multifunctional radar system [J]. Journal of Systems Engineering and Electronics, 2025, 36(4): 985-993.
[5]	Yifan ZHANG, Tao DONG, Zhihui LIU, Shichao JIN. Multi-QoS routing algorithm based on reinforcement learning for LEO satellite networks [J]. Journal of Systems Engineering and Electronics, 2025, 36(1): 37-47.
[6]	Yu BAI, Di ZHOU, Bolun ZHANG, Zhen HE, Ping HE. Two-to-one differential game via improved MOGWO [J]. Journal of Systems Engineering and Electronics, 2025, 36(1): 233-255.
[7]	Xueqiang GU, Lina LU, Fengtao XIANG, Wanpeng ZHANG. Formation-containment control for nonholonomic multi-agent systems with a desired trajectory constraint [J]. Journal of Systems Engineering and Electronics, 2025, 36(1): 256-268.
[8]	Nanxun DUO, Qinzhao WANG, Qiang LYU, Wei WANG. Tactical reward shaping for large-scale combat by multi-agent reinforcement learning [J]. Journal of Systems Engineering and Electronics, 2024, 35(6): 1516-1529.
[9]	Guofei LI, Shituo LI, Bohao LI, Yunjie WU. Deep reinforcement learning guidance with impact time control [J]. Journal of Systems Engineering and Electronics, 2024, 35(6): 1594-1603.
[10]	Qi WANG, Zhizhong LIAO. Computational intelligence interception guidance law using online off-policy integral reinforcement learning [J]. Journal of Systems Engineering and Electronics, 2024, 35(4): 1042-1052.
[11]	Guang ZHAN, Kun ZHANG, Ke LI, Haiyin PIAO. UAV maneuvering decision-making algorithm based on deep reinforcement learning under the guidance of expert experience [J]. Journal of Systems Engineering and Electronics, 2024, 35(3): 644-665.
[12]	Donghao QIN, Le WANG, Jiuan GAO, Jianxiang XI. Minimum-energy leader-following formation of distributed multi-agent systems with communication constraints [J]. Journal of Systems Engineering and Electronics, 2023, 34(6): 1419-1431.
[13]	Yaozhong ZHANG, Zhuoran WU, Zhenkai XIONG, Long CHEN. A UAV collaborative defense scheme driven by DDPG algorithm [J]. Journal of Systems Engineering and Electronics, 2023, 34(5): 1211-1224.
[14]	Jiawei XIA, Xufang ZHU, Zhong LIU, Qingtao XIA. LSTM-DPPO based deep reinforcement learning controller for path following optimization of unmanned surface vehicle [J]. Journal of Systems Engineering and Electronics, 2023, 34(5): 1343-1358.
[15]	Zhengyu YE, Bin JIANG, Yuehua CHENG, Ziquan YU, Yang YANG. Distributed fault diagnosis observer for multi-agent system against actuator and sensor faults [J]. Journal of Systems Engineering and Electronics, 2023, 34(3): 766-774.