Relational graph location network for multi-view image localization

doi:10.23919/JSEE.2023.000050

Abstract

Abstract:

In multi-view image localization task, the features of the images captured from different views should be fused properly. This paper considers the classification-based image localization problem. We propose the relational graph location network (RGLN) to perform this task. In this network, we propose a heterogeneous graph construction approach for graph classification tasks, which aims to describe the location in a more appropriate way, thereby improving the expression ability of the location representation module. Experiments show that the expression ability of the proposed graph construction approach outperforms the compared methods by a large margin. In addition, the proposed localization method outperforms the compared localization methods by around 1.7% in terms of meter-level accuracy.

Key words: multi-view image localization, graph construction, heterogeneous graph, graph neural network

Yukun YANG, Xiangdong LIU. Relational graph location network for multi-view image localization[J]. Journal of Systems Engineering and Electronics, 2023, 34(2): 460-468.

Figures/Tables 8

Fig 1

Fig 2

Fig 3

Fig 4

Table 1

Table 2

Fig 5

Fig 6

References 46

1	HU H J, WANG H S, LIU Z, et al Domain-invariant similarity activation map contrastive learning for retrieval-based long-term visual localization. IEEE/CAA Journal of Automatica Sinica, 2021, 9 (2): 313- 328.
2	ARANDJELOVIC R, GRONAT P, TORII A, et al. NetVLAD: CNN architecture for weakly supervised place recognition. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 5297−5307.
3	CHENG W T, LIN W S, CHEN K, et al. Cascaded parallel filtering for memory-efficient image-based localization. Proc. of the IEEE/CVF International Conference on Computer Vision, 2019: 1032−1041.
4	CHIOU M J, LIU Z, YIN Y, et al Zero-shot multi-view indoor localization via graph location networks. Proc. of the 28th ACM International Conference on Multimedia, 2020, 3431- 3440.
5	MILFORD M J, WYETH G F SeqSLAM: visual routebased navigation for sunny summer days and stormy winter nights. Proc. of the IEEE International Conference on Robotics and Automation, 2012, 1643- 1649.
6	FACIL J M, OLID D, MONTESANO L, et al. Conditioninvariant multiview place recognition. https://arxiv.53yu.com/abs/1902.09516.
7	LIU Z G, CHENG L, LIU A A, et al Multiview and multimodal pervasive indoor localization. Proc. of the 25th ACM international Conference on Multimedia, 2017, 109- 117.
8	KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks. https://arxiv.org/abs/1609.02907.
9	VELICKOVIC P, CUCURULL G, CASANOVA A, et al. Graph attention networks. https://arxiv.org/abs/1710.10903.
10	SCHLICHTKRULL M, KIPF T N, BLOEM P, et al Modeling relational data with graph convolutional networks. Proc. of the European Semantic Web Conference, 2018, 593- 607.
11	ZHANG C X, SONG D J, HUANG C, et al Heterogeneous graph neural network. Proc. of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019, 793- 803.
12	WANG X, JI H Y, SHI C, et al Heterogeneous graph attention network. Proc. of the World Wide Web Conference, 2019, 2022- 2032.
13	FAN S H, ZHU J X, HAN X T, et al Metapath-guided heterogeneous graph neural network for intent recommendation. Proc. of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019, 2478- 2486.
14	LUO W J, ZHANG H, YANG X D, et al Dynamic heterogeneous graph neural network for real-time event prediction. Proc. of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020, 3213- 3223.
15	ARANDJELOVIC R, GRONAT P, TORII A, et al. NetVLAD: CNN architecture for weakly supervised place recognition. Proc of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 5297−5307.
16	SARLIN P E, CADENA C, SIEGWART R, et al From coarse to fine: robust hierarchical localization at large scale. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, 12716- 12725.
17	XU K Y L, HU W H, LESKOVEC J, et al. How powerful are graph neural networks? https://arxiv.org/abs/1810.00826.
18	GILMER J, SCHOENHOLZ S S, RILEY P F, et al Neural message passing for quantum chemistry. Proc. of the International Conference on International Conference on Machine Learning, 2017, 1263- 1272.
19	WANG X L, GIRSHICK R, GUPTA A, et al Non-local neural networks. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, 7794- 7803.
20	BATTAGLIA P W, HAMRICK J B, BAPST V, et al. Relational inductive biases, deep learning, and graph networks. https://arxiv.org/abs/1806.01261.
21	WANG W T, TANG Z Y, CHEN Y C, et al Parity recognition of blade number and manoeuvre intention classification algorithm of rotor target based on micro-Doppler features using CNN. Journal of Systems Engineering and Electronics, 2020, 31 (5): 884- 889. doi: 10.23919/JSEE.2020.000062
22	SUN R, HUANG Q H, FANG W, et al Attributes-based person re-identification via CNNs with coupled clusters loss. Journal of Systems Engineering and Electronics, 2020, 31 (1): 45- 55.
23	LIN L K, WANG S Y, TANG Z X Using deep learning to detect small targets in infrared oversampling images. Journal of Systems Engineering and Electronics, 2018, 29 (5): 947- 952. doi: 10.21629/JSEE.2018.05.07
24	LI B Q, HU X H Effective distributed convolutional neural network architecture for remote sensing images target classification with a pre-training approach. Journal of Systems Engineering and Electronics, 2019, 30 (2): 238- 244. doi: 10.21629/JSEE.2019.02.02
25	YE T, ZHAO Z Y, ZHANG J, et al Low-altitude small-sized object detection using lightweight feature-enhanced convolutional neural network. Journal of Systems Engineering and Electronics, 2021, 32 (4): 841- 853. doi: 10.23919/JSEE.2021.000073
26	CHEN J B, WANG Z H, LI H Y Real-time object segmentation based on convolutional neural network with saliency optimization for picking. Journal of Systems Engineering and Electronics, 2018, 29 (6): 1300- 1307. doi: 10.21629/JSEE.2018.06.17
27	ZHANG C S, WANG P Y, CHEN K, et al Identity-aware convolutional neural networks for facial expression recognition. Journal of Systems Engineering and Electronics, 2017, 28 (4): 784- 792. doi: 10.21629/JSEE.2017.04.18
28	ZHAO B D, LU H Z, CHEN S F, et al Convolutional neural networks for time series classification. Journal of Systems Engineering and Electronics, 2017, 28 (1): 162- 169. doi: 10.21629/JSEE.2017.01.18
29	LIN C, CHANG Q, LI X X Uplink NOMA signal transmission with convolutional neural networks approach. Journal of Systems Engineering and Electronics, 2020, 31 (5): 890- 898. doi: 10.23919/JSEE.2020.000068
30	ZHAO B J, ZHAO B Y, TANG L B, et al Multi-scale object detection by top-down and bottom-up feature pyramid network. Journal of Systems Engineering and Electronics, 2019, 30 (1): 1- 12. doi: 10.21629/JSEE.2019.01.01
31	KAUR H, KOUNDAL D, KADYAN V Image fusion techniques: a survey. Archives of Computational Methods in Engineering, 2021, 28 (7): 4425- 4447. doi: 10.1007/s11831-021-09540-7
32	LECUN Y, BOTTOU L, BENGIO Y, et al Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86 (11): 2278- 2324. doi: 10.1109/5.726791
33	KRIZHEVSKY A, SUTSKEVER I, HINTON G E Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 2012, 25, 1097- 1105.
34	HE K M, ZHANG X Y, REN S Q, et al Deep residual learning for image recognition. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, 770- 778.
35	HE K M, ZHANG X Y, REN S Q, et al Identity mappings in deep residual networks. Proc. of the European Conference on Computer Vision, 2016, 630- 645.
36	SANDLER M, HOWARD A, ZHU M, et al Mobilenetv2: Inverted residuals and linear bottlenecks. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, 4510- 4520.
37	HOWARD A, SANDLER M, CHU G, et al Searching for mobilenetv3. Proc. of the IEEE/CVF International Conference on Computer Vision, 2019, 1314- 1324.
38	HE C, XIA G S, SUN H SAR images classification method based on Dempster-Shafer theory and kernel estimate. Journal of Systems Engineering and Electronics, 2007, 18 (2): 210- 216. doi: 10.1016/S1004-4132(07)60076-0
39	FENG L, WANG J, LIU S L, et al Multi-label dimensionality reduction and classification with extreme learning machines. Journal of Systems Engineering and Electronics, 2014, 25 (3): 502- 513. doi: 10.1109/JSEE.2014.00058
40	LI F, ZHAO C S, DING G Z, et al A reliable and accurate indoor localization method using phone inertial sensors. Proc. of the ACM Conference on Ubiquitous Computing, 2012, 421- 430.
41	SHU Y C, BO C, SHEN G B, et al Magicol: indoor localization using pervasive magnetic field and opportunistic WiFi sensing. IEEE Journal on Selected Areas in Communications, 2015, 33 (7): 1443- 1457. doi: 10.1109/JSAC.2015.2430274
42	RAVI N, SHANKAR P, FRANKEL A, et al Indoor localization using camera phones. Proc. of the 7th IEEE Workshop on Mobile Computing Systems & Applications, 2005, 1- 7.
43	GAO R P, TIAN Y, YE F, et al Sextant: towards ubiquitous indoor localization service by photo-taking of the environment. IEEE Trans. on Mobile Computing, 2015, 15 (2): 460- 474.
44	LIANG J Z, CORSO N, TURNER E, et al Image based localization in indoor environments. Proc. of the 4th International Conference on Computing for Geospatial Research and Application, 2013, 70- 75.
45	LIU Z G, CHENG L, LIU A A, et al Multi-view and multimodal pervasive indoor localization. Proc. of the 25th ACM International Conference on Multimedia, 2017, 109- 117.
46	LIU L, LI H D, DAI Y C Stochastic attraction-repulsion embedding for large scale image localization. Proc. of the IEEE/CVF International Conference on Computer Vision, 2019, 2570- 2579.

Dataset	Method	Meter-level accuracy
ICUBE	Pedes	58.30
	Magicol	69.20
	Matching	75.00
	MVG	82.50
	SARE	84.4
	GLN	93.92
	GLN-ATT	90.88
	RGLN	96.92
MALL-1	Sextant	47
MALL-2	GeoImage	53
WCP	SARE	78.80
	GLN	79.88
	GLN-ATT	79.88
	RGLN	80.49

Method	Training Recall@N					Training CDF@k
Method	N=1	N=2	N=3	N=5	N=10	k=1	k=2	k=3	k=5	k=10
H+R-GCN	0.997	0.998	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000
I+GCN	0.950	0.970	0.979	0.988	0.992	0.971	0.977	0.979	0.982	0.983
I+GAT	0.966	0.980	0.985	0.989	0.995	0.980	0.983	0.985	0.988	0.988
I+GIN	0.983	0.995	0.998	0.998	1.000	0.997	0.998	0.998	0.998	0.998