Deep reinforcement learning and its application in autonomous fitting optimization for attack areas of UCAVs

doi:10.23919/JSEE.2020.000048

Journal of Systems Engineering and Electronics ›› 2020, Vol. 31 ›› Issue (4): 734-742.doi: 10.23919/JSEE.2020.000048

• Systems Engineering • Previous Articles Next Articles

Deep reinforcement learning and its application in autonomous fitting optimization for attack areas of UCAVs

Yue LI^1,²(), Xiaohui QIU^2,*(), Xiaodong LIU³(), Qunli XIA¹()

¹ School of Aerospace Engineering, Beijing Institute of Technology, Beijing 100081, China
² Science and Technology on Electro-Optic Control Laboratory, Luoyang 471000, China
³ Beijing Aerospace Automatic Control Research Institute, Beijing 100854, China

Received:2019-11-12 Online:2020-08-25 Published:2020-08-25
Contact: Xiaohui QIU E-mail:liyue627167955@163.com;qiuxh759@163.com;k.start@163.com;1010@bit.edu.cn
About author:LI Yue was born in 1995. He received his B.E. degree from Beijing Institute of Technology in 2016. He is currently a doctoral student in School of Aerospace Engineering, Beijing Institute of Technology. His main research interests include flight vehicle design, guidance and control. E-mail: liyue627167955@163.com|QIU Xiaohui was born in 1975. He received hisB.E. degree in industrial automation from Xi'an Jiaotong Universityin 1997, and M.E. degree in electronic information engineering fromNorthwestern Polytechnical University in 2008. He is asenior engineer in Science and Technology on Electro-Optic ControlLaboratory. He is currently a doctoral student in School ofAerospace Engineering, Beijing Institute of Technology. Hisresearch interest is system simulation.E-mail: qiuxh759@163.com|LIU Xiaodong was born in 1987. He received his B.E. degree from Qingdao University in 2008, and Ph.D. degree in automation control science and engineering from Beihang University in 2013. From 2013 to 2015, he was a post-doctoral researcher with Beijing Aerospace Automatic Control Institute, Beijing, China. Currently, he is a senior engineer of the Beijing Aerospace Automatic Control Institute. His research interests include adaptive control, integrated guidance and control for aircraft. E-mail: k.start@163.com|XIA Qunli was born in 1971. He received his B.E. degree in launcher and design, M.E. degree in flight mechanics and Ph.D. degree in aircraft design from Beijing Institute of Technology in 1993, 1996 and 1999, respectively. He is currently an adjunct professor with Beijing Institute of Technology. His research interests are control and guidance technology. E-mail: 1010@bit.edu.cn
Supported by:
the Key Laboratory of Defense Science and Technology Foundation of Luoyang Electro-optical Equipment Research Institute(6142504200108);This work was supported by the Key Laboratory of Defense Science and Technology Foundation of Luoyang Electro-optical Equipment Research Institute (6142504200108)

Abstract

Abstract:

The ever-changing battlefield environment requires the use of robust and adaptive technologies integrated into a reliable platform. Unmanned combat aerial vehicles (UCAVs) aim to integrate such advanced technologies while increasing the tactical capabilities of combat aircraft. As a research object, common UCAV uses the neural network fitting strategy to obtain values of attack areas. However, this simple strategy cannot cope with complex environmental changes and autonomously optimize decision-making problems. To solve the problem, this paper proposes a new deep deterministic policy gradient (DDPG) strategy based on deep reinforcement learning for the attack area fitting of UCAVs in the future battlefield. Simulation results show that the autonomy and environmental adaptability of UCAVs in the future battlefield will be improved based on the new DDPG algorithm and the training process converges quickly. We can obtain the optimal values of attack areas in real time during the whole flight with the well-trained deep network.

Key words: attack area, neural network, deep deterministic policy gradient (DDPG), unmanned combat aerial vehicle (UCAV)

Yue LI, Xiaohui QIU, Xiaodong LIU, Qunli XIA. Deep reinforcement learning and its application in autonomous fitting optimization for attack areas of UCAVs[J]. Journal of Systems Engineering and Electronics, 2020, 31(4): 734-742.

Figures/Tables 12

Fig 1

Fig 2

Fig 3

Table 1

Table 2

Table 3

Fig 4

Fig 5

Fig 6

Fig 7

Fig 8

Fig 9

References 29

1	SIBLEY A K, JAIN T N, BUTLER M, et al. Remote scene size-up using an unmanned aerial vehicle in a simulated mass casualty incident. Prehospital Emergency Care, 2019, 23 (3): 332- 339. doi: 10.1080/10903127.2018.1511765
2	JAIN T, SIBLEY A, STRYHN H, et al. Comparison of unmanned aerial vehicle technology-assisted triage versus standard practice in triaging casualties by paramedic students in a mass-casualty incident scenario. Prehospital and Disaster Medicine, 2018, 33 (4): 335- 380.
3	HANDFORD C, REEVES F, PARKER P. Prospective use of unmanned aerial vehicles for military medical evacuation in future conflicts. Journal of the Royal Army Medical Corps, 2018, 164 (4): 293- 296. doi: 10.1136/jramc-2017-000890
4	RICHARD M, DANIEL S, VITALI V, et al. A third-party casualty risk model for unmanned aircraft system operations. Reliability Engineering and System Safety, 2014, 124, 105- 116. doi: 10.1016/j.ress.2013.11.016
5	MATTHEW M. The drone debate: a primer on the U.S. use of unmanned aircraft outside of conventional battlefields. Joint Force Quarterly, 2019, 92, 83- 84.
6	SEPULVEDA E, SMITH H. Technology challenges of stealth unmanned combat aerial vehicles. The Aeronautical Journal, 2017, 121 (1243): 1261- 1295. doi: 10.1017/aer.2017.53
7	SUN F Y, LI Y J, DU Y, et al. A study on the high stability control for the integrated aero-propulsion system under supersonic state. Aerospace Science and Technology, 2018, 76, 350- 360. doi: 10.1016/j.ast.2018.02.017
8	YOSHIYASU H. Composing supersonic materials. Flight International, 2015, 188 (5510): 43.
9	YAZAN M, AMRO N, ALAA D, et al. Agent-based simulation of unmanned aerial vehicles in civilian applications: a systematic literature review and research directions. Future Generation Computer Systems, 2019, 100, 344- 364. doi: 10.1016/j.future.2019.04.051
10	WEI X Q, YANG J Y, FAN X R. Distributed guidance law design for multi-UAV multi-direction attack based on reducing surrounding area. Aerospace Science and Technology, 2020, 99, 105571. doi: 10.1016/j.ast.2019.105571
11	KERRY N, ROBERT R, PAVEL F. Goal model analysis of autonomy requirements for unmanned aircraft systems. Requirements Engineering, 2018, 23 (4): 509- 555. doi: 10.1007/s00766-017-0278-6
12	PENG K M, LIN F, CHEN B M. Online schedule for autonomy of multiple unmanned aerial vehicles. Science China Information Sciences, 2017, 60 (7): 217- 229.
13	HUI Y L, NAN Y, CHEN S D, et al. Dynamic attack zone of air-to-air missile after being launched in random wind field. Chinese Journal of Aeronautics, 2015, 28 (5): 1519- 1528. doi: 10.1016/j.cja.2015.08.013
14	ASHARUL I K, YASEEN A. Unmanned aerial vehicle in the machine learning environment. Procedia Computer Science, 2019, 160, 46- 53. doi: 10.1016/j.procs.2019.09.442
15	LUCAS P O, MAURO S A, MARCATO J, et al. A convolutional neural network approach for counting and geolocating citrus-trees in UAV multispectral imagery. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 160, 97- 106. doi: 10.1016/j.isprsjprs.2019.12.010
16	LI Y W, CAO K. Establishment and application of intelligent city building information model based on BP neural network model. Computer Communications, 2020, 153, 382- 389. doi: 10.1016/j.comcom.2020.02.013
17	WU G X, MIAO Y M, ZHANG Y, et al. Energy efficient for UAV-enabled mobile edge computing networks: intelligent task prediction and offloading. Computer Communications, 2020, 150, 556- 562. doi: 10.1016/j.comcom.2019.11.037
18	QIU H X, DUAN H B. A multi-objective pigeon-inspired optimization approach to UAV distributed flocking among obstacles. Information Sciences, 2020, 509, 515- 529. doi: 10.1016/j.ins.2018.06.061
19	LIU Q, LI M, YANG J, et al. Joint power and time allocation in energy harvesting of UAV operating system. Computer Communications, 2020, 150, 811- 817. doi: 10.1016/j.comcom.2019.12.009
20	FALLATI L, POLIDORI A, SALVATORE C, et al. Anthropogenic marine debris assessment with unmanned aerial vehicle imagery and deep learning: a case study along the beaches of the Republic of Maldives. Science of the Total Environment, 2019, 693, 133581. doi: 10.1016/j.scitotenv.2019.133581
21	NEUPANE B, HORANONT T, HUNG N D. Deep learning based banana plant detection and counting using high-resolution red-green-blue images collected from unmanned aerial vehicle. PLoS One, 2019, 14 (10): e0223906. doi: 10.1371/journal.pone.0223906
22	JIAO Z Y, JIA G Z, CAI Y J. A new approach to oil spill detection that combines deep learning with unmanned aerial vehicles. Computers and Industrial Engineering, 2019, 135, 1300- 1311. doi: 10.1016/j.cie.2018.11.008
23	ZHAO X, YUAN Y T, SONG M D, et al. Use of unmanned aerial vehicle imagery and deep learning UNet to extract rice lodging. Sensors, 2019, 19 (18): 3859. doi: 10.3390/s19183859
24	QU C Z, GAI W D, ZHONG M Y, et al. A novel reinforcement learning based grey wolf optimizer algorithm for unmanned aerial vehicles path planning. Applied Soft Computing Journal, 2020, 89, 106099. doi: 10.1016/j.asoc.2020.106099
25	ZHAO X Y, ZONG Q, TIAN B L, et al. Fast task allocation for heterogeneous unmanned aerial vehicles through reinforcement learning. Aerospace Science and Technology, 2019, 92, 588- 594. doi: 10.1016/j.ast.2019.06.024
26	YANG J, YOU X H, WU G X, et al. Application of reinforcement learning in UAV cluster task scheduling. Future Generation Computer Systems, 2019, 95, 140- 148. doi: 10.1016/j.future.2018.11.014
27	MA Y, ZHU W B, MICHAEL G B, et al. Continuous control of a polymerization system with deep reinforcement learning. Journal of Process Control, 2019, 75, 40- 47. doi: 10.1016/j.jprocont.2018.11.004
28	MA Z W, WANG C, NIU Y F, et al. A saliency-based reinforcement learning approach for a UAV to avoid flying obstacles. Robotics and Autonomous Systems, 2018, 100, 108- 118. doi: 10.1016/j.robot.2017.10.009
29	LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning. Computer Science, 2015, 8 (6): 187- 200.

Parameter	Range	Value
Height of missile/m	0-18 000	0, 3 000, 6 000, 9 000, 12 000, 15 000, 18 000
Height of target/m	0-18 000	0, 3 000, 6 000, 9 000, 12 000, 15 000, 18 000
Velocity of missile	Ma = 0-1	0.7, 0.8, 0.9
Velocity of target	Ma = 0-1	0.60, 0.64, 0.68
Ballistic inclination angle/(°)	-30-+45	-30, -15, 0, 15, 30, 45
Entry angle/(°)	-180-+180	-180, -150, -120, -90, -60, -30, 0, 30, 60, 90, 120, 150, 180
Off-axis angle/(°)	-60-+60	-60, -45, -30, -15, 0, 15, 30, 45, 60

Algorithm parameter	Value in program
Rounds of learning	2 000
Steps of learning	200
Learning rate of actor-network	0.001
Learning rate of critic-network	0.001
Reward discount	0.9
Memory pool capacity	30 000
Size of batch	32

Environ-mental state	Atmospheric pressure/Pa	Atmospheric temperature/K	Atmospheric density/(kg/m$^3$)	Wind speed/(m/s)
1	26 500	216.65	0.088 9	0
2	26 500	216.65	0.413 5	0
3	26 500	223.252 1	0.088 9	0
4	26 500	223.252 1	0.413 5	0
5	5 529.1	216.65	0.088 9	0
6	5 529.1	216.65	0.413 5	0
7	5 529.1	223.252 1	0.088 9	0
8	5 529.1	223.252 1	0.413 5	0
9	26 500	216.65	0.088 9	3
10	5 529.1	223.252 1	0.413 5	3

[1]	Deping XIA, Liang ZHANG, Tao WU, Wenjun HU. An interference suppression algorithm for cognitive bistatic airborne radars [J]. Journal of Systems Engineering and Electronics, 2022, 33(3): 585-593.
[2]	Fuyunxiang YANG, Leping YANG, Yanwei ZHU, Xin ZENG. A DNN based trajectory optimization method for intercepting non-cooperative maneuvering spacecraft [J]. Journal of Systems Engineering and Electronics, 2022, 33(2): 438-446.
[3]	Fan WANG, Pengfei FAN, Yonghua FAN, Bin XU, Jie YAN. Robust adaptive control of hypersonic vehicle considering inlet unstart [J]. Journal of Systems Engineering and Electronics, 2022, 33(1): 188-196.
[4]	Kwame Bensah KULEVOME Delanyo, Hong WANG, Xuegang WANG. Deep neural network based classification of rolling element bearings and health degradation through comprehensive vibration signal analysis [J]. Journal of Systems Engineering and Electronics, 2022, 33(1): 233-246.
[5]	Jiandong ZHANG, Qiming YANG, Guoqing SHI, Yi LU, Yong WU. UAV cooperative air combat maneuver decision based on multi-agent reinforcement learning [J]. Journal of Systems Engineering and Electronics, 2021, 32(6): 1421-1438.
[6]	Zhengliang ZHU, Degui YANG, Junchao ZHANG, Feng TONG. Dataset of human motion status using IR-UWB through-wall radar [J]. Journal of Systems Engineering and Electronics, 2021, 32(5): 1083-1096.
[7]	Xin ZENG, Yanwei ZHU, Leping YANG, Chengming ZHANG. A guidance method for coplanar orbital interception based on reinforcement learning [J]. Journal of Systems Engineering and Electronics, 2021, 32(4): 927-938.
[8]	Zhifei XI, An XU, Yingxin KOU, Zhanwu LI, Aiwu YANG. Target maneuver trajectory prediction based on RBF neural network optimized by hybrid algorithm [J]. Journal of Systems Engineering and Electronics, 2021, 32(2): 498-516.
[9]	Wantian WANG, Ziyue TANG, Yichang CHEN, Yongjian SUN. Parity recognition of blade number and manoeuvre intention classification algorithm of rotor target based on micro-Doppler features using CNN [J]. Journal of Systems Engineering and Electronics, 2020, 31(5): 884-889.
[10]	Chuan LIN, Qing CHANG, Xianxu LI. Uplink NOMA signal transmission with convolutional neural networks approach [J]. Journal of Systems Engineering and Electronics, 2020, 31(5): 890-898.
[11]	Qingguo LIU, Xinxue LIU, Jian WU, Yaxiong LI. A fast computational method for the landing footprints of space-to-ground vehicles [J]. Journal of Systems Engineering and Electronics, 2020, 31(5): 1062-1076.
[12]	Caihua WU, Jianchao MA, Xiuwei ZHANG, Dang XIE. User space transformation in deep learning based recommendation [J]. Journal of Systems Engineering and Electronics, 2020, 31(4): 674-684.
[13]	Rui SUN, Qiheng HUANG, Wei FANG, Xudong ZHANG. Attributes-based person re-identification via CNNs with coupled clusters loss [J]. Journal of Systems Engineering and Electronics, 2020, 31(1): 45-55.
[14]	Baiquan LU, Chenlong NI, Zhongwei ZHENG, Tingzhang LIU. A global optimization algorithm based on multi-loop neural network control [J]. Journal of Systems Engineering and Electronics, 2019, 30(5): 1007-1024.
[15]	Junhua YAN, Xuehan BAI, Wanyi ZHANG, Yongqi XIAO, Chris CHATWIN, Rupert YOUNG, Phil BIRCH. No-reference image quality assessment based on AdaBoost BP neural network in wavelet domain [J]. Journal of Systems Engineering and Electronics, 2019, 30(2): 223-237.

Deep reinforcement learning and its application in autonomous fitting optimization for attack areas of UCAVs

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

Share this article

Figures/Tables 12

References 29

Related Articles 15

Recommended Articles

Metrics

Comments