Autonomous landing scene recognition based on transfer learning for drones

doi:10.23919/JSEE.2023.000031

Journal of Systems Engineering and Electronics ›› 2023, Vol. 34 ›› Issue (1): 28-35.doi: 10.23919/JSEE.2023.000031

• REMOTE SENSING • Previous Articles Next Articles

Autonomous landing scene recognition based on transfer learning for drones

Hao DU¹^,²(), Wei WANG²^,³(), Xuerao WANG¹(), Yuanda WANG¹^,*()

¹ School of Automation, Southeast University, Nanjing 210096, China
² Autonomous Control Robot Laboratory, Jiangsu Zhongke Institute of Applied Research on Intelligent Science and Technology, Changzhou 213164, China
³ Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology, Nanjing University of Information Science and Technology, Nanjing 210044, China

Received:2022-06-30 Accepted:2022-12-30 Online:2023-02-18 Published:2023-03-03
Contact: Yuanda WANG E-mail:du-hao@seu.edu.cn;wwcb@nuist.edu.cn;wangxuerao@seu.edu.cn;wangyd@seu.edu.cn
About author:
DU Hao was born in 1987. He received his B.E. and M.S. degrees in electronic information engineering and system analysis and integration from Nanjing University of Information Science and Technology, Nanjing, China, in 2009 and 2012, respectively. He is pursuing his Ph.D. degree in the School of Automation, Southeast University, Nanjing, China. His current research interests include multi-sensor fusion navigation for the drone, computer vision, and scene recognition. E-mail: du-hao@seu.edu.cn

WANG Wei was born in 1972. He received his B.E., M.E., and Ph.D. degrees from Chiba University, Chiba, Japan, in 2004, 2006, and 2009, respectively. He is a professor at the School of Automation, Nanjing University of Information Science and Technology, Nanjing, China. His current research interests include nonlinear system control and drone intelligent control. E-mail: wwcb@nuist.edu.cn

WANG Xuerao was born in 1996. She received her B.S. degree in engineering from Qingdao University of Technology, Qingdao, China, in 2016, and M.S. degree in engineering from University of Science and Technology Beijing, Beijing, China, in 2019. She is pursuing her Ph.D. degree in control science and engineering in the School of Automation, Southeast University, Nanjing, China. Her research interests include intelligent control, nonlinear system control, and reinforcement learning. E-mail: wangxuerao@seu.edu.cn

WANG Yuanda was born in 1993. He received his B.S. degree in automation from Nanjing University of Information Science and Technology, Nanjing, China in 2014, and Ph.D. degree in control science and engineering from Southeast University, Nanjing, China, in 2020. He is working as a postdoctoral researcher with the School of Automation, Southeast University, Nanjing, China. He was a visiting Ph.D. student with the Department of Electrical, Computer, and Biomedical Engineering, University of Rhode Island, Kingston, RI, USA, from 2016 to 2018. His current research interests include deep reinforcement learning, neural networks, and multi-agent systems. E-mail: wangyd@seu.edu.cn
Supported by:
This work was supported by the National Natural Science Foundation of China (62103104) and the China Postdoctoral Science Foundation (2021M690615).

Abstract

Abstract:

In this paper, we study autonomous landing scene recognition with knowledge transfer for drones. Considering the difficulties in aerial remote sensing, especially that some scenes are extremely similar, or the same scene has different representations in different altitudes, we employ a deep convolutional neural network (CNN) based on knowledge transfer and fine-tuning to solve the problem. Then, LandingScenes-7 dataset is established and divided into seven classes. Moreover, there is still a novelty detection problem in the classifier, and we address this by excluding other landing scenes using the approach of thresholding in the prediction stage. We employ the transfer learning method based on ResNeXt-50 backbone with the adaptive momentum (ADAM) optimization algorithm. We also compare ResNet-50 backbone and the momentum stochastic gradient descent (SGD) optimizer. Experiment results show that ResNeXt-50 based on the ADAM optimization algorithm has better performance. With a pre-trained model and fine-tuning, it can achieve 97.8450% top-1 accuracy on the LandingScenes-7 dataset, paving the way for drones to autonomously learn landing scenes.

Key words: landing scene recognition, convolutional neural network (CNN), transfer learning, remote sensing image

Hao DU, Wei WANG, Xuerao WANG, Yuanda WANG. Autonomous landing scene recognition based on transfer learning for drones[J]. Journal of Systems Engineering and Electronics, 2023, 34(1): 28-35.

Figures/Tables 9

Fig 1

Fig 2

Table 1

Fig 3

Fig 4

Fig 5

Table 2

Fig 6

Fig 7

References 20

1	LI B Q, HU X H Effective distributed convolutional neural network architecture for remote sensing images target classification with a pre-training approach. Journal of Systems Engineering and Electronics, 2019, 30 (2): 238- 244. doi: 10.21629/JSEE.2019.02.02
2	XIA G S, BAI X, DING J, et al DOTA: a large-scale dataset for object detection in aerial images. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, 3974- 3983.
3	CHENG G, XIE X X, HAN J W, et al Remote sensing image scene classification meets deep learning: challenges, methods, benchmarks, and opportunities. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13, 3735- 3756. doi: 10.1109/JSTARS.2020.3005403
4	ANAND A K, BARMAN S, PRAKASH N S, et al Vision based automatic landing of unmanned aerial vehicle. Proc. of the International Conference on Information Technology and Applied Mathematics, 2019, 102- 113.
5	TIAN C, HUANG C B An algorithm for unmanned aerial vehicle landing scene recognition based on deep learning and computational verbs. Proc. of the IEEE International Conference on Civil Aviation Safety and Information Technology, 2019, 180- 184.
6	LU C W, TSOUGENIS E, TANG C K Improving object recognition with the l-channel. Pattern Recognition, 2016, 49, 187- 197. doi: 10.1016/j.patcog.2015.06.007
7	ZHOU B L, LAPEDRIZA A, KHOSLA A, et al Places: a 10 million image database for scene recognition. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2018, 40 (6): 1452- 1464. doi: 10.1109/TPAMI.2017.2723009
8	XIAO J X, HAYS J, EHINGER K A, et al SUN database: large-scale scene recognition from abbey to zoo. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2010, 3485- 3492.
9	XIAO J X, EHINGER K A, HAYS J, et al SUN database: exploring a large collection of scene categories. International Journal of Computer Vision, 2016, 119 (1): 3- 22. doi: 10.1007/s11263-014-0748-y
10	PATTERSON G, XU C, SU H, et al The SUN attribute database: beyond categories for deeper scene understanding. International Journal of Computer Vision, 2014, 108 (1/2): 59- 81.
11	YANG T, LI P Q, ZHANG H M, et al. Monocular vision SLAM-based UAV autonomous landing in emergencies and unknown environments. Electronics, 2018, 7(5): 73.
12	HE K M, ZHANG X Y, REN S Q, et al Deep residual learning for image recognition. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, 770- 778.
13	XIE S N, GIRSHICK R, DOLLAR P, et al Aggregated residual transformations for deep neural networks. Proc. of the 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017, 5987- 5995.
14	XIE L, LEE F F, LIU L, et al Scene recognition: a comprehensive survey. Pattern Recognition, 2020, 102, 107205.
15	NEYSHABUR B, SEDGHI H, ZHANG C Y What is being transferred in transfer learning? Proc. of the 34th Conference on Neural Information Processing Systems, 2020, 512- 523.
16	WU X Z, LIU R Z, YANG H Q, et al An Xception based convolutional neural network for scene image classification with transfer learning. Proc. of the 2nd International Conference on Information Technology and Computer Application, 2020, 262- 267.
17	STEGER C, UIRICH M, WIEDEMANN C. Machine vision algorithms and applications. Berlin: Wiley-VCH, 2018.
18	KINGMA D P, BA J L. Adam: a method for stochastic optimization. Proc. of the 3rd International Conference on Learning Representations. https://arxiv.org/abs/1412.6980.
19	QIAN N On the momentum term in gradient descent learning algorithms. Neural Networks, 1999, 12 (1): 145- 151. doi: 10.1016/S0893-6080(98)00116-6
20	DUCHI J, HAZAN E, SINGER Y Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 2011, 12 (61): 2121- 2159.

Grade	Presentation	Comment
Safe	00	lawn
General	01	wilderness, wheat_field
Dangerous	11	crowded_place, water_ area, vehicle_ intensive_place, road

Method	Batch size	Learning rate	Error	Top-1 accuracy/%
ResNeXt-50 with ADAM	32	0.0001	0.0622	97.8450
ResNeXt-50 with SGD	32	0.0001	0.1508	95.3875
ResNet-50 with ADAM	32	0.0001	0.0935	96.7675

[1]	Tingting WEI, Weilin YUAN, Junren LUO, Wanpeng ZHANG, Lina LU. VLCA: vision-language aligning model with cross-modal attention for bilingual remote sensing image captioning [J]. Journal of Systems Engineering and Electronics, 2023, 34(1): 9-18.
[2]	Zhengliang ZHU, Degui YANG, Junchao ZHANG, Feng TONG. Dataset of human motion status using IR-UWB through-wall radar [J]. Journal of Systems Engineering and Electronics, 2021, 32(5): 1083-1096.
[3]	Wantian WANG, Ziyue TANG, Yichang CHEN, Yongjian SUN. Parity recognition of blade number and manoeuvre intention classification algorithm of rotor target based on micro-Doppler features using CNN [J]. Journal of Systems Engineering and Electronics, 2020, 31(5): 884-889.
[4]	Binquan LI, Xiaohui HU. Effective distributed convolutional neural network architecture for remote sensing images target classification with a pre-training approach [J]. Journal of Systems Engineering and Electronics, 2019, 30(2): 238-244.
[5]	Baojun ZHAO, Boya ZHAO, Linbo TANG, Wenzheng WANG, Chen WU. Multi-scale object detection by top-down and bottom-up feature pyramid network [J]. Journal of Systems Engineering and Electronics, 2019, 30(1): 1-12.
[6]	Liangkui LIN, Shaoyou WANG, Zhongxing TANG. Using deep learning to detect small targets in infrared oversampling images [J]. Journal of Systems Engineering and Electronics, 2018, 29(5): 947-952.
[7]	Bendong Zhao, Huanzhang Lu, Shangfeng Chen, Junliang Liu, and Dongya Wu. Convolutional neural networks for time series classification [J]. Systems Engineering and Electronics, 2017, 28(1): 162-.
[8]	Gensheng Hu, Xiaoqi Sun, Dong Liang, and Yingying Sun. Cloud removal of remote sensing image based on multi-output support vector regression [J]. Journal of Systems Engineering and Electronics, 2014, 25(6): 1082-1088.
[9]	Qunming Wang, Liguo Wang, and Danfeng Liu. Integration of spatial attractions between and within pixels for sub-pixel mapping [J]. Journal of Systems Engineering and Electronics, 2012, 23(2): 293-303.

Autonomous landing scene recognition based on transfer learning for drones

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

Share this article

Figures/Tables 9

References 20

Related Articles 9

Recommended Articles

Metrics

Comments