λ-return-based aircraft maneuvering for terminal defense and positioning guidance strategies

doi:10.23919/JSEE.2025.000112

Journal of Systems Engineering and Electronics ›› 2025, Vol. 36 ›› Issue (6): 1692-1708.doi: 10.23919/JSEE.2025.000112

• CONTROL THEORY AND APPLICATION • Previous Articles

λ-return-based aircraft maneuvering for terminal defense and positioning guidance strategies

Shijie DENG¹^,*(), Yingxin KOU¹(), Maolong LYU²(), Zhanwu LI¹(), An XU¹()

¹ Aeronautics Engineering School, Air Force Engineering University, Xi’an 710038, China
² Air Traffic Control and Navigation College, Air Force Engineering University, Xi’an 710038, China

Received:2024-03-06 Online:2025-12-18 Published:2026-01-07
Contact: Shijie DENG E-mail:dengshijiesilence@163.com;kgykyx@hotmail.com;maolonglv@163.com;afeulzw@189.cn;18157494594@163.com
About author:
DENG Shijie was born in 1997. He received his M.S. degree in armament science and technology from Air Force Engineering University, Xi’an, China in 2021. He is currently pursuing his Ph.D. degree at the School of Aeronautical Engineering, Air Force Engineering University. His research interests include the application of reinforcement learning in air combat maneuver decision-making, air combat decision-making cognition and environmental model learning based on artificial neural networks. Additionally, he has also dabbled in research fields such as airborne firepower command and control, missile guidance, and hyperspectral data target detection. E-mail: dengshijiesilence@163.com

KOU Yingxin was born in 1965. He received his M.S. and Ph.D. degrees from Air Force Engineering University, Xi’an, China, in 1997 and 2010, respectively. He is currently a professor with the Aeronautics Engineering School, Air Force Engineering University. His research interests include air combat, nonlinear and adaptive control, artificial intelligence, multi-sensor data fusion, and optimized target assignment. E-mail: kgykyx@hotmail.com

LYU Maolong was born in 1993. He received his B.S. degree in electrical engineering and automation and M.S. degree in control science and engineering from Air Force Engineering University, Xi’an, China, in 2014 and 2016, respectively, and Ph.D. degree in intelligent control from the Delft Center for Systems and Control, Delft University of Technology, Delft, the Netherlands, in 2021. He is currently an associate professor with the College of Air Traffic Control and Navigation, Air Force Engineering University. His research interests include adaptive learning control, deep reinforcement learning, and intelligent decision with applications in multiagent systems, hypersonic flight, vehicles, unmanned autonomous system. E-mail: maolonglv@163.com

LI Zhanwu was born in 1978. He received his M.S. degree from Air Force Engineering University, Xi’an, China, in 2007. He is currently a professor with the Aeronautics Engineering School, Air Force Engineering University. His research mainly focuses on multi-sensor data fusion, machine intelligent and artificial intelligence. E-mail: afeulzw@189.cn

XU An was born in 1984. He received his M.S. and Ph.D. degrees in armament science and technology from Air Force Engineering University, Xi’an, China, in 2009 and 2012, respectively. He is currently a postdocotoral researcher at the Electronic Information College of Northwestern Polytechnical University, Xi’an, China, as well as an associate professor with the Aeronautics Engineering School, Air Force Engineering University. His research interests include nonlinear and adaptive control, artificial intelligence and pattern recognition. E-mail: 18157494594@163.com

Abstract

Abstract:

Aiming at the terminal defense problem of aircraft, this paper proposes a method to simultaneously achieve terminal defense and seize the dominant position. The method employs a λ-return based reinforcement learning algorithm, which can be applied to the flight assistance decision-making system to improve the pilot’s survivability. First, we model the environment to simulate the interaction between air-to-air missiles and aircraft. Subsequently, we propose a λ-return based approach to improve the deep Q learning network (DQN), deep advantageous actor criticism (A2C), and proximity policy optimization (PPO) algorithms used to train manoeuvre strategies. The method employs an action space containing nine manoeuvres and defines the off-target distance at the end of the scene as a sparse reward for algorithm training. Simulation results show that the convergence speed of the three improved algorithms is significantly improved when using the λ-return method. Moreover, the effect of the fetch value on the convergence speed is verified by ablation experiments. In order to solve the illegal behavior problem in the training process, we also design a backtracking-based illegal behavior masking mechanism, which improves the data generation efficiency of the environment model and promotes effective algorithm training.

Key words: λ-return, terminal defense, positioning guidance, reinforcement learning

Shijie DENG, Yingxin KOU, Maolong LYU, Zhanwu LI, An XU. λ-return-based aircraft maneuvering for terminal defense and positioning guidance strategies[J]. Journal of Systems Engineering and Electronics, 2025, 36(6): 1692-1708.

Figures/Tables 19

Fig 1

Table 1

Table 2

Fig 2

Fig 3

Table 3

Table 4

Table 5

Table 6

Fig 4

Fig 5

Table 7

Table 8

Fig 6

Fig 7

Fig 8

Fig 9

Fig 10

Fig 11

References 33

1	YANG X Z, AI J L Evasive maneuvers against missiles for unmanned combat aerial vehicle in autonomous air combat. Journal of System Simulation, 2018, 30 (5): 1957- 1966.
2	ZHAN G, ZHANG K, LI K, et al UAV maneuvering decision-making algorithm based on deep reinforcement learning under the guidance of expert experience. Journal of Systems Engineering and Electronics, 2024, 35 (3): 644- 665. doi: 10.23919/JSEE.2024.000022
3	DUO N X, WANG Q Z, LV Q, et al Tactical reward shaping for large-scale combat by multi-agent reinforcement learning. Journal of Systems Engineering and Electronics, 2024, 35 (6): 1516- 1529. doi: 10.23919/JSEE.2024.000062
4	IMADO F Some aspects of a realistic three-dimensional pursuit-evasion game. Journal of Guidance, Control, and Dynamics, 1993, 16 (2): 289- 293. doi: 10.2514/3.21002
5	IMADO F, KURODA T Family of local solutions in a missile-aircraft differential game. Journal of Guidance, Control, and Dynamics, 2011, 34 (2): 583- 591. doi: 10.2514/1.48345
6	ALKAHER D, MOSHAIOV A Game-based safe aircraft navigation in the presence of energy-bleeding coasting missile. Journal of Guidance, Control, and Dynamics, 2016, 39 (7): 1539- 1550. doi: 10.2514/1.G001676
7	KARELAHTI J, VIRTANEN K, RAIVIO T Near-optimal missile avoidance trajectories via receding horizon control. Journal of Guidance, Control, and Dynamics, 2007, 30 (5): 1287- 1298. doi: 10.2514/1.26024
8	CARR R W, COBB R G, PACHTER M, et al Solution of a pursuit-evasion game using a near-optimal strategy. Journal of Guidance, Control, and Dynamics, 2018, 41 (4): 841- 850. doi: 10.2514/1.G002911
9	SINGH L. Autonomous missile avoidance using nonlinear model predictive control. Proc. of the AIAA Guidance, Navigation, and Control Conference and Exhibit, 2004. DOI: 10.2514/6.2004−4910.
10	ZHANG N, CHEN C S, SUN J G, et al Strategy of barrel roll and decoy deployment against infrared air-to-air missile. Infrared Technology, 2022, 44 (3): 236- 248.
11	YANG Z, ZHOU D Y, KONG W R, et al Nondominated manoeuvre strategy set with tactical requirements for a fighter against missiles in a dogfight. IEEE Access, 2020, 8, 117298- 117312. doi: 10.1109/ACCESS.2020.3004864
12	YAGCI O, NIKBAY M. Evasive maneuver trajectory optimization of an UCAV against an air to air missile. Proc. of the AIAA Aviation Forum, 2022: 3791.
13	WANG X P, LIN Q Y, DONG X M. Aircraft evasive maneuver trajectory optimization based on QPSO. Proc. of the International Congress on Ultra Modern Telecommunications and Control Systems, 2010: 416−420.
14	LEE G T, KIM C O Autonomous control of combat unmanned aerial vehicles to evade surface-to-air missiles using deep reinforcement learning. IEEE Access, 2020, 8, 226724- 226736. doi: 10.1109/ACCESS.2020.3046284
15	HUANG C H, WANG C Z, CHAI S J, et al. Research on evasion policy of UCAV against infrared air-to-air missile based on soft actor critic algorithm. Proc. of the 14th International Conference on Computer Modeling and Simulation, 2022: 153−159.
16	MNIH V, KAVUKCUOGLU K, SILVER D, et al Human-level control through deep reinforcement learning. Nature, 2015, 518 (7540): 529- 533. doi: 10.1038/nature14236
17	DEGRAVE J, FELICI F, BUCHLI J, et al Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 2022, 602 (7897): 414- 419. doi: 10.1038/s41586-021-04301-9
18	FAWZI A, BALOG M, HUANG A, et al Discovering faster matrix multiplication algorithms with reinforcement learning. Nature, 2022, 610 (7930): 47- 53. doi: 10.1038/s41586-022-05172-4
19	WATKINS C J C H. Learning from delayed rewards. Cambridge: Cambridge University, 1989.
20	SUTTON R S Learning to predict by the methods of temporal differences. Machine Learning, 1988, 3, 9- 44.
21	WANG B, LI X L, GAO Z Q, et al Gradient compensation traces based temporal difference learning. Neurocomputing, 2021, 442, 221- 235. doi: 10.1016/j.neucom.2021.02.042
22	LI T, YANG D S, XIE X P, et al Event-triggered control of nonlinear discrete-time system with unknown dynamics based on HDP (λ). IEEE Trans. on Cybernetics, 2021, 52 (7): 6046- 6058.
23	LI B, LI T, LIU M H, et al. A non-asymptotic analysis of adaptive TD (λ) learning in wireless sensor networks. International Journal of Distributed Sensor Networks, 2022. DOI: 10.1177/15501329221114546.
24	NAIK A, SUTTON R S. Multi-step average-reward prediction via differential TD (λ). Proc. of the Conference on Reinforcement Learning and Decision Making. http://incompleteideas.net/papers/RLDM22-NS-Differential_{T}Dlambda.pdf.
25	WAN Y, NAIK A, SUTTON R S. Learning and planning in average-reward Markov decision processes. Proc. of the International Conference on Machine Learning, 2021. DOI: 10.48550/arXiv.2006.16318.
26	MADAN K, RECTOR-BROOKS J, KORABLYOV M, et al Learning GflowNets from partial episodes for improved convergence and stability. Proc. of the International Conference on Machine Learning, 2023, 23467- 23483.
27	CHEN Z, MAGULURI S T, SHAKKOTTAI S, et al A Lyapunov theory for finite-sample guarantees of Markovian stochastic approximation. Operations Research, 2024, 72 (4): 1352- 1367. doi: 10.1287/opre.2022.0249
28	KOU Y X, DENG S J, LI Z W, et al. Collaborative air warfare game environment for multi-agent in 3D space. Proc. of the IEEE International Conference on Unmanned Systems, 2023. DOI: 10.1109/ICUS58632.2023.10318365.
29	FU S T, ZHANG H J A method of high-precision and fast calculation for attack area of air-to-air missile. Science Technology and Engineering, 2012, 12 (27): 7074- 7077.
30	YANG W Y, BAI C J, CAI C, et al Survey on sparse reward in deep reinforcement learning. Computer Science, 2020, 47 (3): 182- 191.
31	SUTTON R S. Reinforcement learning: an introduction. Cambridge: MIT Press, 2018.
32	FUJIMOTO S, HOOF H, MEGER D. Addressing function approximation error in actor-critic methods. Proc. of the International Conference on Machine Learning, 2018. DOI: 10.48550/arXiv.1802.09477.
33	CHEN G, PENG Y M, ZHANG M J. An adaptive clipping approach for proximal policy optimization. https://arXiv.org/abs/180406461.

Number	Maneuver mode	Control quantity
Number	Maneuver mode	$ {n_x} $	$ {n_y} $	${\gamma _\text{T}}$
1	Constant speed forward flight	0	1	0
2	Accelerated forward flight	4	1	0
3	Slow-speed forward flight	−4	1	0
4	Turn right	0	2	$ {\text{π} \mathord{\left/ {\vphantom {\text{π} 3}} \right. } 3} $
5	Turn left	0	2	$ - {\text{π} \mathord{\left/ {\vphantom {\text{π} 3}} \right. } 3} $
6	Accelerated dive	4	−5	0
7	Accelerated climb	4	7	0
8	Deceleration dive	−4	−5	0
9	Slow down climb	−4	7	0

Parameter	Value	Illustration
$ {{{N}}_{{y}}} $	50	Pitch overload limit
$ {{{N}}_{{z}}} $	50	Yaw overload limit
$ {{K}} $	4	Proportional guidance coefficient
$ {{{\tau}}_{{0}}} $	0.3	Indicating the maximum first-order time constant

Object	X/m	Y/m	Z/m	V/(m/s)
Fighter	0	5000	0	300
Missile	−20000	5000	0	1000

Object	$ \theta $/(°)	$ \varphi $/(°)	$ \gamma $/(°)	−
Fighter	1	0	0	−
Missile	1	0	0	−

Hyperparameter	Value	Description
Input layer	9	Input state space dimension
Hidden layer 1	1e2	Number of neurons in hidden layer 1
Hidden layer 1	50	Number of neurons in hidden layer 2
Out layer	9	Output action space action number
Memory capacity	2e4	Only when the memory bank is full of data can the network start training
Minibatch size	32	The number of samples used during each training session
Learning rate	1e-5	The magnitude of parameter updates each time
Init exploration	0.9	Initial the value of ε in ε-greedy exploration
Final exploration	0.3	Final the value of ε in ε-greedy exploration
Gamma	0.99	Discount factor gamma used in the Q-learning update
Target network update frequency	1e2	The target network is updated once every 100 times the evaluation network is updated
Back step	2	The length of the storage queue for the illegal action blocking mechanism
λ	0.8	The λ value of the λ-return method

Hyperparameter	Value	Description
Actor net input layer	9	“Actor” network input state space dimension
Actor net hidden layer	1e2	The number of hidden neurons in the “actor” network
Actor net out layer	9	Dimension of “actor” network output action probability
Critic net input layer	9	Dimension of network input state space for “critic”
Critic net hidden layer 1	1e2	Number of neurons in the hidden layer 1 of the “commentator” network
Critic net hidden layer 2	50	Number of neurons in the hidden layer 2 of the “commentator” network
Critic net out layer	1	Dimension of “critic” network output state estimation
Actor net learning rate	1e-3	Learning rate of the actor network
Critic net learning rate	1e-3	Learning rate of the “commentator” network
Back step	2	The length of the storage queue for the illegal action blocking mechanism
λ	0.8	The λ value of the λ-return method

λ-return-based aircraft maneuvering for terminal defense and positioning guidance strategies

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

Share this article

Figures/Tables 19

References 33

Related Articles 15

Recommended Articles

Metrics

Comments

Algorithm	λ=0.2	λ=0.4	λ=0.6	λ=0.8
λ-DQN	3775	2050	1975	1775
λ-A2C	−	−	950	600
λ-PPO	−	1500	1182	590

Algorithm	Scenes’ number	Back step
Algorithm	Scenes’ number	0	1	2	3
λ-DQN	4000	690	3717	3863	3886
λ-A2C	1500	221	1231	1327	1410
λ-PPO	2000	410	1725	1919	1948

[1]	Wenhao CHEN, Gang CHEN, Jichao LI, Jiang JIANG. Disintegration of heterogeneous combat network based on double deep Q-learning [J]. Journal of Systems Engineering and Electronics, 2025, 36(5): 1235-1246.
[2]	Rui ZHOU, Weichao ZHONG, Wenlong LI, Hao ZHANG. Self-play training and analysis for GEO inspection game with modular actions [J]. Journal of Systems Engineering and Electronics, 2025, 36(5): 1353-1373.
[3]	Siyu HENG, Ting CHENG, Zishu HE, Yuanqing WANG, Luqing LIU. Adaptive dwell scheduling based on Q-learning for multifunctional radar system [J]. Journal of Systems Engineering and Electronics, 2025, 36(4): 985-993.
[4]	Yifan ZHANG, Tao DONG, Zhihui LIU, Shichao JIN. Multi-QoS routing algorithm based on reinforcement learning for LEO satellite networks [J]. Journal of Systems Engineering and Electronics, 2025, 36(1): 37-47.
[5]	Nanxun DUO, Qinzhao WANG, Qiang LYU, Wei WANG. Tactical reward shaping for large-scale combat by multi-agent reinforcement learning [J]. Journal of Systems Engineering and Electronics, 2024, 35(6): 1516-1529.
[6]	Guofei LI, Shituo LI, Bohao LI, Yunjie WU. Deep reinforcement learning guidance with impact time control [J]. Journal of Systems Engineering and Electronics, 2024, 35(6): 1594-1603.
[7]	Qi WANG, Zhizhong LIAO. Computational intelligence interception guidance law using online off-policy integral reinforcement learning [J]. Journal of Systems Engineering and Electronics, 2024, 35(4): 1042-1052.
[8]	Guang ZHAN, Kun ZHANG, Ke LI, Haiyin PIAO. UAV maneuvering decision-making algorithm based on deep reinforcement learning under the guidance of expert experience [J]. Journal of Systems Engineering and Electronics, 2024, 35(3): 644-665.
[9]	Yaozhong ZHANG, Zhuoran WU, Zhenkai XIONG, Long CHEN. A UAV collaborative defense scheme driven by DDPG algorithm [J]. Journal of Systems Engineering and Electronics, 2023, 34(5): 1211-1224.
[10]	Jiawei XIA, Xufang ZHU, Zhong LIU, Qingtao XIA. LSTM-DPPO based deep reinforcement learning controller for path following optimization of unmanned surface vehicle [J]. Journal of Systems Engineering and Electronics, 2023, 34(5): 1343-1358.
[11]	Lu DONG, Zichen HE, Chunwei SONG, Changyin SUN. A review of mobile robot motion planning methods: from classical motion planning workflows to reinforcement learning-based architectures [J]. Journal of Systems Engineering and Electronics, 2023, 34(2): 439-459.
[12]	Yaozhong ZHANG, Yike LI, Zhuoran WU, Jialin XU. Deep reinforcement learning for UAV swarm rendezvous behavior [J]. Journal of Systems Engineering and Electronics, 2023, 34(2): 360-373.
[13]	Yunxiu ZENG, Kai XU. Recognition and interfere deceptive behavior based on inverse reinforcement learning and game theory [J]. Journal of Systems Engineering and Electronics, 2023, 34(2): 270-288.
[14]	Guangran CHENG, Lu DONG, Xin YUAN, Changyin SUN. Reinforcement learning-based scheduling of multi-battery energy storage system [J]. Journal of Systems Engineering and Electronics, 2023, 34(1): 117-128.
[15]	Peng LIU, Boyuan XIA, Zhiwei YANG, Jichao LI, Yuejin TAN. A deep reinforcement learning method for multi-stage equipment development planning in uncertain environments [J]. Journal of Systems Engineering and Electronics, 2022, 33(6): 1159-1175.