Automatic video mosaicking algorithm via dynamic key-frame

doi:10.23919/JSEE.2020.000005

Journal of Systems Engineering and Electronics ›› 2020, Vol. 31 ›› Issue (2): 272-278.doi: 10.23919/JSEE.2020.000005

• Electronics Technology • Previous Articles Next Articles

Automatic video mosaicking algorithm via dynamic key-frame

Yufeng JI¹(), Weixing LI^1,*(), Kai FENG¹(), Boyang XING¹(), Feng PAN^1,²()

¹ School of Automation, Beijing Institute of Technology, Beijing 100081, China
² Kunming-BIT Industry Technology Research Institute Co., Ltd, Kunming 650106, China

Received:2019-03-26 Online:2020-04-30 Published:2020-04-30
Contact: Weixing LI E-mail:jyf131462@163.com;liweixing@bit.edu.cn;bitfengkai@163.com;golaced@163.com;andropanfeng@126.com
About author:JI Yufeng was born in 1994. He received his B.S. degree from Beijing Institute of Technology in 2016. Since 2016, he has been a postgraduate student majoring in control science and engineering in Beijing Institute of Technology. His research interests include machine learning and objects detecting, and tracking of UAVs. E-mail: jyf131462@163.com|LI Weixing was born in 1976. He received his M.S. degree in pattern recognition and intelligent control from Beijing Institute of Technology. He is an associate professor at School of Automation in Beijing Institute of Technology, China. His research interests include pattern recognition, video analysis and machine learning. E-mail: liweixing@bit.edu.cn|FENG Kai was born in 1996. He received his B.S. degree from Beijing Institute of Technology in 2018. Since 2018, he has been a postgraduate student majoring in control science and engineering in Beijing Institute of Technology. His research interests include computer vision and deep learning. E-mail: bitfengkai@163.com|XING Boyang was born in 1990. He received his M.S. degree from Beijing Institute of Technology in 2012. Since 2015, he has been a doctoral student majoring in control science and engineering in Beijing Institute of Technology. His research interests include robotics and control, and computer vision. E-mail: golaced@163.com|PAN Feng was born in 1978. He received his M.S. and Ph.D. degrees in pattern recognition and intelligent control from Beijing Institute of Technology. He is an associate professor at School of Automation in Beijing Institute of Technology. His research interests include intelligent computing, robotics and control, and computer vision. E-mail: andropanfeng@126.com
Supported by:
the National Science Foundation of China(61603040);the National Science Foundation of China(61973036);the National Science Foundation of China(61433003);This work was supported by the National Science Foundation of China (61603040; 61973036; 61433003)

Abstract

Abstract:

Automatic video mosaicking is a challenging task in computer vision. Current researches consider either panoramic or mapping tasks on short videos. In this paper, an automatic mosaicking algorithm is proposed for both mapping and panoramic tasks based on the adapted key-frame on videos of any length. The speeded up robust features (SURF) and the grid motion statistic (GMS) algorithm are used for feature extraction and matching between consecutive frames, which are used to compute the transformation. In order to reduce the influence of the accumulated error during image stitching, an evaluation metric is put forward for the transformation matrix. Besides, a self-growth method is employed to stitch the global image for long videos. The algorithm is evaluated by using aerial-view and panoramic videos respectively on the graphic processing unit (GPU) device, which can satisfy the real-time requirement. The experimental results demonstrate that the proposed algorithm is able to achieve a better performance than the state-of-art.

Key words: video mosaicking, image stitching, speeded up robust feature (SURF), panorama, mapping

Yufeng JI, Weixing LI, Kai FENG, Boyang XING, Feng PAN. Automatic video mosaicking algorithm via dynamic key-frame[J]. Journal of Systems Engineering and Electronics, 2020, 31(2): 272-278.

Figures/Tables 8

Fig 1

Fig 2

Fig 3

Fig 4

Fig 5

Fig 6

Fig 7

Table 1

References 16

1	ZITOVA B, FLUSSER J. Image registration methods: a survey. Image and Vision Computing, 2003, 21 (11): 977- 1000.
2	LI J, YANG T, YU J Y, et al. Fast aerial video stitching. International Journal of Advanced Robotic Systems, 2014, 11 (10): 167- 177.
3	AMIRI A J, MORADI H. Real-time video stabilization and mosaicking for monitoring and surveillance. Proc. of the 4th International Conference on Robotics and Mechatronics, 2016, 613- 618.
4	BROWN M, LOWE D G. Automatic panoramic image stitching using invariant features. International Journal of Computer Vision, 2007, 74 (1): 59- 73. doi: 10.1007/s11263-006-0002-3
5	HEINZE N, ESSWEIN M, KRUGER W, et al. Automatic image exploitation system for small UAVs. Proc. of the International Society for Optics and Photonics, 2008, 1- 10.
6	BAY H, ESS A, TUYTELAARS T, et al. Speeded-up robust features (SURF). Computer Vision and Image Understanding, 2008, 110 (3): 346- 359. doi: 10.1016/j.cviu.2007.09.014
7	DAVID G L. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60 (2): 91- 110. doi: 10.1023/B:VISI.0000029664.99615.94
8	HARRIS C, STEPHENS M J. A combined corner and edge detector. Proc. of the Alvey Vision Conference, 1988, 147- 152.
9	RUBLEE E, RABAUD V, KONOLIGE K, et al. ORB: an efficient alternative to SIFT or SURF. Proc. of the International Conference on Computer Vision, 2011, 2564- 2571.
10	ROSTEN E, DRUMMOND T. Machine learning for high-speed corner detection. Proc. of the European Conference on Computer Vision, 2006, 430- 443.
11	ZHU J, REN M. Image mosaic method based on SIFT features of line segment. Computational and Mathematical Methods in Medicine, 2014, 1- 11.
12	BU S, ZHAO Y, WAN G, et al. Map2DFusion: real-time incremental UAV image mosaicking based on monocular SLAM. Proc. of the International Conference on Intelligent Robots & Systems, 2016, 1464- 1571.
13	LIN C C, PANKANTI S U, RAMAMURTHY K N, et al. Adaptive as-natural-as-possible image stitching. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 1155- 1163.
14	YANG Y, WANG D S, SONG W J, et al. Dynamic video stitching method embracing image semantics. Systems Engineering and Electronics, 2018, 40 (12): 2845- 2854.
15	SZELISKI R, SHUM H Y. Creating full view panoramic image mosaics and environment maps. Proc. of the 24th Annual Conference on Computer Graphics and Interactive Techniques, 1997, 251- 258.
16	BIAN J, LIN W Y, MATSUSHITA Y, et al. GMS: grid-based motion statistics for fast, ultra-robust feature correspondence. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, 4181- 4190.

Device	SURF+GMS/s	FPS
CPU	0.128	8
GPU	0.027	22

[1]	Xiaoli WU, Wentao WEI, Sabrina CALDWELL, Chengqi XUE, Linlin WANG. Optimization method for a radar situation interface from error-cognition to information feature mapping [J]. Journal of Systems Engineering and Electronics, 2022, 33(4): 924-937.
[2]	Jun HAN, Weixing LI, Kai FENG, Feng PAN. Vision-based aerial image mosaicking algorithm with object detection [J]. Journal of Systems Engineering and Electronics, 2022, 33(2): 259-268.
[3]	Yangyang JIANG, Yan GAO, Wenqi SONG, Yue LI, Quan QUAN. Bibliometric analysis of UAV swarms [J]. Journal of Systems Engineering and Electronics, 2022, 33(2): 406-425.
[4]	Zhenxing ZHANG, Rennong YANG, Huanyu LI, Yuhuan FANG, Zhenyu HUANG, Ying ZHANG. Antlion optimizer algorithm based on chaos search and its application [J]. Journal of Systems Engineering and Electronics, 2019, 30(2): 352-365.
[5]	Shuxia Guo, Yafeng,Ruibing Liu, and Ying Gao. Multi-dimensional and complicated electromagnetic interference hardware-in-the-loop simulation method [J]. Systems Engineering and Electronics, 2015, 26(6): 1142-1148.
[6]	Qunming Wang, Liguo Wang, and Danfeng Liu. Integration of spatial attractions between and within pixels for sub-pixel mapping [J]. Journal of Systems Engineering and Electronics, 2012, 23(2): 293-303.
[7]	Fuzhen Zhu, Jinzong Li, Bing Zhu, and Dongdong Ma. Super-resolution image reconstruction based on three-step-training neural networks [J]. Journal of Systems Engineering and Electronics, 2010, 21(6): 934-940.
[8]	Jianjun Zhang and Hong Yuan. Analysis of unmanned aerial vehicle navigation and height control system based on GPS [J]. Journal of Systems Engineering and Electronics, 2010, 21(4): 643-649.
[9]	Meng An*, Zhiguo Jiang, Danpei Zhao. High speed robust image registration and localization using optimized algorithm and its performances evaluation [J]. Journal of Systems Engineering and Electronics, 2010, 21(3): 520-526.
[10]	Zheng Liping, Li Guangyao, Liang Yongquan & Sha Jing. Design of ontology mapping framework and improvement of similarity computation [J]. Journal of Systems Engineering and Electronics, 2007, 18(3): 641-645.
[11]	Yang Guowei, Tu Xuyan & Wang Shoujue. Block and parallel modelling of broad domain nonlinear continuous mapping based on NN [J]. Journal of Systems Engineering and Electronics, 2006, 17(3): 586-592.

Automatic video mosaicking algorithm via dynamic key-frame

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

Share this article

Figures/Tables 8

References 16

Related Articles 11

Recommended Articles

Metrics

Comments