Vision-based aerial image mosaicking algorithm with object detection

doi:10.23919/JSEE.2022.000026

Journal of Systems Engineering and Electronics ›› 2022, Vol. 33 ›› Issue (2): 259-268.doi: 10.23919/JSEE.2022.000026

• ELECTRONICS TECHNOLOGY • Previous Articles Next Articles

Vision-based aerial image mosaicking algorithm with object detection

Jun HAN(), Weixing LI*(), Kai FENG(), Feng PAN()

¹ School of Automation, Beijing Institute of Technology, Beijing 100081, China

Received:2020-09-25 Accepted:2022-02-15 Online:2022-05-06 Published:2022-05-06
Contact: Weixing LI E-mail:3120190878@bit.edu.cn;liweixing@bit.edu.cn;fengkai_bit@outlook.com;andropanfeng@126.com
About author:|HAN Jun was born in 1997. He received his B.S. degree from Beijing Institute of Technology (BIT) in 2019. He is a postgraduate student majoring in control science and engineering in BIT. His research interests include computer vision and multi-object tracking. E-mail: 3120190878@bit.edu.cn||LI Weixing was born in 1976. He is a senior experimentalist at the School of Automation in Beijing Institute of Technology (BIT). He received his M.S. degree in pattern recognition and intelligent control from BIT. His research interests include pattern recognition, video analysis, and machine learning. E-mail: liweixing@bit.edu.cn||FENG Kai was born in 1996. He is a postgraduate student majoring in control science and engineering in Beijing Institute of Technology (BIT). He received his B.S. degree from BIT in 2018. His research interests include computer vision and object detection. E-mail: fengkai_bit@outlook.com||PAN Feng was born in 1978. He is an associate professor at the School of Automation in Beijing Institute of Technology (BIT). He received his M.S. and Ph.D. degrees in pattern recognition and intelligent control from BIT. His research interests include intelligent computing and robotics and control. E-mail: andropanfeng@126.com
Supported by:
This work was supported by the National Natural Science Foundation of China (61603040; 61973036)

Abstract

Abstract:

Aerial image sequence mosaicking is one of the challenging research fields in computer vision. To obtain large-scale orthophoto maps with object detection information, we propose a vision-based image mosaicking algorithm without any extra location data. According to object detection results, we define a complexity factor to describe the importance of each input image and dynamically optimize the feature extraction process. The feature points extraction and matching processes are mainly guided by the speeded-up robust features (SURF) and the grid motion statistic (GMS) algorithm respectively. A robust reference frame selection method is proposed to eliminate the transformation distortion by searching for the center area based on overlaps. Besides, the sparse Levenberg-Marquardt (LM) algorithm and the heavy occluded frames removal method are applied to reduce accumulated errors and further improve the mosaicking performance. The proposed algorithm is performed by using multithreading and graphics processing unit (GPU) acceleration on several aerial image datasets. Extensive experiment results demonstrate that our algorithm outperforms most of the existing aerial image mosaicking methods in visual quality while guaranteeing a high calculation speed.

Key words: image mosaicking, object detection, grid motion statistic (GMS), mapping

Jun HAN, Weixing LI, Kai FENG, Feng PAN. Vision-based aerial image mosaicking algorithm with object detection[J]. Journal of Systems Engineering and Electronics, 2022, 33(2): 259-268.

Figures/Tables 13

Fig 1

Fig 2

Fig 3

Fig 4

Fig 5

Fig 6

Fig 7

Fig 8

Fig 9

Fig 10

Fig 9

Table 1

Fig 11

References 28

1	CUI J G, LIU M, ZHANG Z T, et al Robust UAV thermal infrared remote sensing images stitching via overlap-prior-based global similarity prior model. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14, 270- 282. doi: 10.1109/JSTARS.2020.3032011
2	LIU C, ZHANG S H, AKBAR A Ground feature oriented path planning for unmanned aerial vehicle mapping. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12 (4): 1175- 1187. doi: 10.1109/JSTARS.2019.2899369
3	BU S H, ZHAO Y, WAN G, et al Map2DFusion: real-time incremental UAV image mosaicing based on monocular SLAM. Proc. of the IEEE International Conference on Intelligent Robots and Systems, 2016, 4564- 4571.
4	ZHANG J M, ZHU H Q, WANG P Y, et al ATT squeeze U-Net: a lightweight network for forest fire detection and recognition. IEEE Access, 2021, 9, 10858- 10870. doi: 10.1109/ACCESS.2021.3050628
5	CAO Y C, YANG F, TANG Q F, et al An attention enhanced bidirectional LSTM for early forest fire smoke recognition. IEEE Access, 2019, 7, 154732- 154742. doi: 10.1109/ACCESS.2019.2946712
6	AVOLA D, CINQUE L, FORESTI G L, et al A UAV video dataset for mosaicking and change detection from low-altitude flights. IEEE Trans. on Systems, Man, and Cybernetics: Systems, 2020, 50 (6): 2139- 2149. doi: 10.1109/TSMC.2018.2804766
7	SZELISKI R Image alignment and stitching: a tutorial. Boston: Now Foundations and Trends, 2006.
8	ZITOVA B, FLUSSER J Image registration methods: a survey. Image and Vision Computing, 2003, 21 (11): 977- 1000. doi: 10.1016/S0262-8856(03)00137-9
9	LOWE D G Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60 (2): 91- 110. doi: 10.1023/B:VISI.0000029664.99615.94
10	BAY H, ESS A, TUYTELAARS T, et al Speeded-up robust features (SURF). Computer Vision and Image Understanding, 2008, 110 (3): 346- 359. doi: 10.1016/j.cviu.2007.09.014
11	RUBLEE E, RABAUD V, KONOLIGE K, et al ORB: an efficient alternative to SIFT or SURF. Proc. of the IEEE International Conference on Computer Vision, 2011, 2564- 2571.
12	EDWARD R, REID B P, TOM D Faster and better: a machine learning approach to corner detection. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2010, 32 (1): 105- 119.
13	BIAN J W, LIN W Y, LIU Y, et al GMS: grid-based motion statistics for fast, ultra-robust feature correspondence. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, 2828- 2837.
14	AN M, JIANG Z G, ZHAO D P High speed robust image registration and localization using optimized algorithm and its performances evaluation. Journal of Systems Engineering and Electronics, 2010, 21 (3): 520- 526. doi: 10.3969/j.issn.1004-4132.2010.03.026
15	AMIRI A J, MORADI H Real-time video stabilization and mosaicking for monitoring and surveillance. Proc. of the 4th International Conference on Robotics and Mechatronics, 2016, 613- 618.
16	LI C, GUO B L, GUO X X, et al Real-time UAV imagery stitching based on grid-based motion statistics. Journal of Physics: Conference Series, 2018, (1069): 012163.
17	MCLAUCHLAN P F, JAENICKE A Image mosaicing using sequential bundle adjustment. Image and Vision Computing, 2002, 20 (9/10): 751- 759. doi: 10.1016/S0262-8856(02)00064-1
18	XU Q, CHEN J, LUO L B, et al UAV image mosaicking based on multiregion guided local projection deformation. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13, 3844- 3855. doi: 10.1109/JSTARS.2020.3006289
19	WU F L, GUAN H N, YAN D J, et al Precise geometric correction and robust mosaicking for airborne lightweight optical butting infrared imaging system. IEEE Access, 2019, 7, 93569- 93579. doi: 10.1109/ACCESS.2019.2928380
20	LI M, LI D R, GUO B X, et al Automatic seam-line detection in UAV remote sensing image mosaicking by use of graph cuts. International Journal of Geo-Information, 2018, 7 (9): 361. doi: 10.3390/ijgi7090361
21	YUAN Y T, FANG F M, ZHANG G X Superpixel-based seamless image stitching for UAV images. IEEE Trans. on Geoscience and Remote Sensing, 2021, 59 (2): 1565- 1576. doi: 10.1109/TGRS.2020.2999404
22	GE Y, WEN G J, YANG X L A fast mosaicking method for small UAV image sequence using a small number of ground control points. Proc. of the 8th International Conference on Intelligent Human-Machine Systems and Cybernetics, 2016, 90- 94.
23	REDMON J, FARHADI A. YOLOv3: an incremental improvement. arXiv preprint arXiv: 1804.02767, 2018.
24	XU Y H, OU J L, HE H, et al Mosaicking of unmanned aerial vehicle imagery in the absence of camera poses. Remote Sensing, 2016, 8 (3): 204. doi: 10.3390/rs8030204
25	HU J W, ZHOU Y H, ZHAO C H, et al An application of panoramic mosaic in UAV aerial image. Proc. of the IEEE 13th International Conference on Control and Automation, 2017, 1049- 1053.
26	QU Z, LI J, BAO K H An unordered image stitching method based on binary tree and estimated overlapping area. IEEE Trans. on Image Processing, 2020, 29, 6734- 6744. doi: 10.1109/TIP.2020.2993134
27	JI Y F, LI W X, FENG K, et al Automatic video mosaicking algorithm via dynamic key-frame. Journal of Systems Engineering and Electronics, 2020, 31 (2): 272- 278. doi: 10.23919/JSEE.2020.000005
28	HSIEH M R, LIN Y L, HSU W H Drone-based object counting by spatially regularized regional proposal network. Proc. of the IEEE International Conference on Computer Vision, 2017, 4165- 4173.

Image data	Sequence length	Mosaicking time/s
Image data	Sequence length	Pix4Dmapper	Ours
Dataset 1	71	169	14
Dataset 2	37	92	8
Dataset 3	40	113	7
Dataset 4	24	58	4

[1]	Xiaoli WU, Wentao WEI, Sabrina CALDWELL, Chengqi XUE, Linlin WANG. Optimization method for a radar situation interface from error-cognition to information feature mapping [J]. Journal of Systems Engineering and Electronics, 2022, 33(4): 924-937.
[2]	Yangyang JIANG, Yan GAO, Wenqi SONG, Yue LI, Quan QUAN. Bibliometric analysis of UAV swarms [J]. Journal of Systems Engineering and Electronics, 2022, 33(2): 406-425.
[3]	Tao YE, Zongyang ZHAO, Jun ZHANG, Xinghua CHAI, Fuqiang ZHOU. Low-altitude small-sized object detection using lightweight feature-enhanced convolutional neural network [J]. Journal of Systems Engineering and Electronics, 2021, 32(4): 841-853.
[4]	Yufeng JI, Weixing LI, Kai FENG, Boyang XING, Feng PAN. Automatic video mosaicking algorithm via dynamic key-frame [J]. Journal of Systems Engineering and Electronics, 2020, 31(2): 272-278.
[5]	Zhenxing ZHANG, Rennong YANG, Huanyu LI, Yuhuan FANG, Zhenyu HUANG, Ying ZHANG. Antlion optimizer algorithm based on chaos search and its application [J]. Journal of Systems Engineering and Electronics, 2019, 30(2): 352-365.
[6]	Baojun ZHAO, Boya ZHAO, Linbo TANG, Wenzheng WANG, Chen WU. Multi-scale object detection by top-down and bottom-up feature pyramid network [J]. Journal of Systems Engineering and Electronics, 2019, 30(1): 1-12.
[7]	Jinbo CHEN, Zhiheng WANG, Hengyu LI. Real-time object segmentation based on convolutional neural network with saliency optimization for picking [J]. Journal of Systems Engineering and Electronics, 2018, 29(6): 1300-1307.
[8]	Shuxia Guo, Yafeng,Ruibing Liu, and Ying Gao. Multi-dimensional and complicated electromagnetic interference hardware-in-the-loop simulation method [J]. Systems Engineering and Electronics, 2015, 26(6): 1142-1148.
[9]	Rui Yao and Yanning Zhang. Compressive sensing for small moving space object detection in astronomical images [J]. Journal of Systems Engineering and Electronics, 2012, 23(3): 378-384.
[10]	Qunming Wang, Liguo Wang, and Danfeng Liu. Integration of spatial attractions between and within pixels for sub-pixel mapping [J]. Journal of Systems Engineering and Electronics, 2012, 23(2): 293-303.
[11]	Qinkun Xiao, Nan Zhang, Fei Li, and Yue Gao. Object detection based on combination of local and spatial information [J]. Journal of Systems Engineering and Electronics, 2011, 22(4): 715-720.
[12]	Fuzhen Zhu, Jinzong Li, Bing Zhu, and Dongdong Ma. Super-resolution image reconstruction based on three-step-training neural networks [J]. Journal of Systems Engineering and Electronics, 2010, 21(6): 934-940.
[13]	Jing Li, Junzheng Wang, and Wei Shen. Moving object detection in framework of compressive sampling [J]. Journal of Systems Engineering and Electronics, 2010, 21(5): 740-745.
[14]	Jianjun Zhang and Hong Yuan. Analysis of unmanned aerial vehicle navigation and height control system based on GPS [J]. Journal of Systems Engineering and Electronics, 2010, 21(4): 643-649.
[15]	Zheng Liping, Li Guangyao, Liang Yongquan & Sha Jing. Design of ontology mapping framework and improvement of similarity computation [J]. Journal of Systems Engineering and Electronics, 2007, 18(3): 641-645.

Vision-based aerial image mosaicking algorithm with object detection

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

Share this article

Figures/Tables 13

References 28

Related Articles 15

Recommended Articles

Metrics

Comments