Multi-QoS routing algorithm based on reinforcement learning for LEO satellite networks

doi:10.23919/JSEE.2024.000041

Journal of Systems Engineering and Electronics ›› 2025, Vol. 36 ›› Issue (1): 37-47.doi: 10.23919/JSEE.2024.000041

• ELECTRONICS TECHNOLOGY • Previous Articles

Multi-QoS routing algorithm based on reinforcement learning for LEO satellite networks

Yifan ZHANG¹^,²(), Tao DONG²(), Zhihui LIU²^,*(), Shichao JIN²()

¹ School of Integrated Circuit and Electronics, Beijing Institute of Technology, Beijing 100081, China
² State Key Laboratory of Space-Ground Integrated Information Technology, Space Star Technology Co., Ltd., Beijing 100095, China

Received:2023-04-06 Online:2025-02-18 Published:2025-03-18
Contact: Zhihui LIU E-mail:1793107437@qq.com;dongtaoandy@163.com;liuzhihui1225@163.com;jinshch@spacestar.com.cn
About author:
ZHANG Yifan was born in 1999. He received his B.S. degree from Beijing Institute of Technology, Beijing, China, in 2021, where he is currently pursuing his M.S. degree in the School of Integrated Circuit and Electronics, Beijing Institute of Technology. His research interests include network traffic optimization research and network routing algorithms. E-mail: 1793107437@qq.com

DONG Tao was born in 1975. He received his B.S. and Ph.D. degrees in electronics science and technology from Beijing Institute of Technology in 1999 and 2004, respectively. He is currently a professor with State Key Laboratory of Space-Ground Integrated Information Technology, Space Star Technology Co., Ltd. His research interests include satellite communication and networks. E-mail: dongtaoandy@163.com

LIU Zhihui was born in 1989. She received her M.S. degree in computational mathematics from Xiangtan University, Xiangtan, Hunan, in 2012, and her Ph.D. degree in signal and information processing at the School of Information and Communication Engineering from Beijing University of Posts and Telecommunications, Beijing, China, in 2016. She is currently a senior engineer in State Key Laboratory of Space-Ground Integrated Information Technology, Space Star Technology Co., Ltd. Her research interests include space communication and networks, resource management and control for heterogeneous networks, and network resource sensing. E-mail: liuzhihui1225@163.com

JIN Shichao was born in 1980. He received his B.S. degree in physics from Lanzhou University in 2004, and Ph.D. degree in physics from Tsinghua University in 2009. He is currently a professor with the State Key Laboratory of Space-Ground Integrated Information Technology, Space Star Technology Co., Ltd. His research interests include satellite communication and networks. E-mail: jinshch@spacestar.com.cn
Supported by:
This work was supported by the National Key Research and Development Program (2021YFB2900604).

Abstract

Abstract:

Low Earth orbit (LEO) satellite networks exhibit distinct characteristics, e.g., limited resources of individual satellite nodes and dynamic network topology, which have brought many challenges for routing algorithms. To satisfy quality of service (QoS) requirements of various users, it is critical to research efficient routing strategies to fully utilize satellite resources. This paper proposes a multi-QoS information optimized routing algorithm based on reinforcement learning for LEO satellite networks, which guarantees high level assurance demand services to be prioritized under limited satellite resources while considering the load balancing performance of the satellite networks for low level assurance demand services to ensure the full and effective utilization of satellite resources. An auxiliary path search algorithm is proposed to accelerate the convergence of satellite routing algorithm. Simulation results show that the generated routing strategy can timely process and fully meet the QoS demands of high assurance services while effectively improving the load balancing performance of the link.

Key words: low Earth orbit (LEO) satellite network, reinforcement learning, multi-quality of service (QoS), routing algorithm

Yifan ZHANG, Tao DONG, Zhihui LIU, Shichao JIN. Multi-QoS routing algorithm based on reinforcement learning for LEO satellite networks[J]. Journal of Systems Engineering and Electronics, 2025, 36(1): 37-47.

Figures/Tables 8

Fig 1

Table 1

Fig 2

Fig 3

Fig 4

Fig 5

Fig 6

Fig 7

References 37

1	YAO H P, WANG L Y, LIU Y J The space-terrestrial integrated network: an overview. IEEE Communications Magazine, 2018, 56, 178- 185.
2	DENH X, CHANG L, ZENG S Y, et al Distance-based back-pressure routing for load-balancing LEO satellite networks. IEEE Trans. on Vehicular Technology, 2023, 72 (1): 1240- 1253. doi: 10.1109/TVT.2022.3206616
3	ZHANG L H Development and prospect of Chinese lunar relay communication satellite. Space: Science & Technology, 2021, 2021, 3471608.
4	LIU W, TAO Y, LIU L Load-balancing routing algorithm based on segment routing for traffic return in LEO satellite networks. IEEE Access, 2019, 7, 112044- 112053. doi: 10.1109/ACCESS.2019.2934932
5	DUN C Y, TAN S H, JIN F L, et al. Traffic allocation strategy based on reinforcement learning in two-layer satellite network. Proc of the International Conference on Internet, Education and Information Technology , 2021: 159−163.
6	HUANG L M, HUANG B, ZHANG M T. A load balancing routing method based on real time traffic in LEO satellite constellation space networks. Proc. of the 95th Vehicular Technology Conference, 2022. DOI: 10.1109/VTC2022-Spring54318.2022.9860540.
7	ZHANG X D, DING L H, RAO Y. QoS routing by genetic algorithm for LEO satellite networks. Proc. of the 2nd International Symposium on Computational Intelligence and Design, 2009: 341−344.
8	JIANG W J, ZHANG P. QoS routing algorithm based on traffic classification in LEO satellite networks. Proc. of the Eighth International Conference on Wireless and Optical Communications Networks, 2011. DOI: 10.1109/WOCN.2011.5872957.
9	BI J L, LI Z Y, WANG R. An ant colony optimization-based load balancing routing algorithm for wireless multimedia sensor networks. Proc. of the 12th International Conference on Communication Technology, 2010: 584−587.
10	LIN W L, DONG Z W, WANG K, et al A novel load balancing scheme for satellite IoT networks based on spatial-temporal distribution of users and advanced genetic algorithms. Sensors, 2022, 22 (20): 7930. doi: 10.3390/s22207930
11	WANG H, RAN Y Y, ZHAO L, et al. GRouting: dynamic routing for LEO satellite networks with graph-based deep reinforcement learning. Proc. of the 4th International Conference on Hot Information-Centric Networking, 2021: 123−128.
12	ZUO P L, WANG C, WEI Z Z, et al. Deep reinforcement learning based load balancing routing for LEO satellite network. Proc. of the IEEE 95th Vehicular Technology Conference, 2022. DOI: 10.1109/VTC2022-Spring54318.2022.9860582.
13	YANG Z A, LONG F, SUN F C. A QoS routing scheme based on ground station for LEO satellite networks. Proc. of the 2nd International Workshop on Education Technology and Computer Science, 2010: 55−58.
14	LOGN F, SUN F C, YANG Z A. A multi-QoS objective optimization routing for hierarchical satellite networks. Proc. of the Second International Workshop on Education Technology and Computer Science, 2010: 51−54.
15	PFANDZELTER T, BERMBACH D. QoS-aware resource placement for LEO satellite edge computing. Proc. of the 6th International Conference on Fog and Edge Computing, 2022: 66−72.
16	WANG P, GUO X M, LI G L. Multi-QoS routing for LEO satellite networks. Proc. of the 9th International Conference on Advanced Communication Technology, 2007: 728−731.
17	WU Z G, REN P Y, XU D Y. Flexible resource allocation for differentiated QoS provisioning in beam-hopping satellite communications system. Proc. of the 95th Vehicular Technology Conference, 2022. DOI: 10.1109/VTC2022-Spring54318.2022.9860761.
18	ZHANG J Z, DING R, LIU J, et al. QoSRA: a QoS-aware routing algorithm for software defined satellite networks. Proc. of the 2nd Information Communication Technologies Conference, 2021: 165−171.
19	WANG Y L, QIN X H, TANG Z X, et al. QoS-centric handover for civil aviation aircraft access in ultra-dense LEO satellite networks. Proc. of the International Conference on Communications, 2022: 1085−1089.
20	LIU H T, WANG Y C, WANG Y X, et al. A successive deep Q-learning based distributed handover scheme for large-scale LEO satellite networks. Proc. of the IEEE 95th Vehicular Technology Conference, 2022. DOI: 10.1109/VTC2022-Spring54318.2022.9860376.
21	YIN Y B, HUANG C H, WU D F, et al. Reinforcement learning-based routing algorithm in satellite-terrestrial integrated networks. Wireless Communications and Mobile Computing, 2021, 12: 3759631.
22	ZHANG L, YAN F, ZHANG Y Y, et al. A routing algorithm based on link state information for LEO satellite networks. Proc. of the Globecom Workshops, 2020. DOI: 10.1109/GCWkshps50303.2020.9367496.
23	WANG P, ZHANG X S, ZHANG S, et al Time-expanded graph-based resource allocation over the satellite networks. IEEE Wireless Communications Letters, 2019, 8 (2): 360- 363. doi: 10.1109/LWC.2018.2872996
24	BARSALEAU D A, TUMMALA M Testing of diffServ performance over a U.S. Navy satellite communication network. Proc. of the Military Communications Conference, 2004, 528- 534.
25	AUDAH L, SUN Z, CRUICKSHANK H. QoS evaluation of multiservice applications over integrated satellite-terrestrial networks. Proc. of the 3rd International Congress on Ultra Modern Telecommunications and Control Systems and Workshops, 2011. https://ieeexplore.ieee.org/abstract/document/6078906.
26	GUO Q, GUO R T, DONG T, et al SDN-based end-to-end fragment-aware routing for elastic data flows in LEO satellite-terrestrial network. IEEE Access, 2019, 7, 396- 410. doi: 10.1109/ACCESS.2018.2885473
27	TODOROV D, VALCHANOV H, ALEKSIEVA V. Load balancing model based on machine learning and segment routing in SDN. Proc. of the International Conference Automatics and Informatics, 2020. DOI: 10.1109/ICAI50593.2020.9311385.
28	SHI X J, REN P Y, DU Q H. Heterogeneous satellite network routing algorithm based on reinforcement learning and mobile agent. Proc. of the IEEE Globecom Workshops, 2020. DOI: 10.1109/GCWkshps50303.2020.9367476.
29	ZHANG L Y, RUI L L, YANG Y, et al. A reliable routing algorithm based on reinforcement learning for self-organizing network. Proc. of the 11th International Conference on Computer Engineering and Networks, 2022: 378−386.
30	WANG X T, DAI Z Q, XU Z. LEO satellite network routing algorithm based on reinforcement learning. Proc. of the 4th International Conference on Electronics Technology, 2021: 1105−1109.
31	LITTMAN M L. Markov games as a framework for multi-agent reinforcement learning. Machine Learning Proceedings, 1994. DOI: 10.1016/B978-1-55860-335-6.50027-1.
32	TSAI K C, FAN L, WANG L C, et al. Multi-commodity flow routing for large-scale LEO satellite networks using deep reinforcement learning. Proc. of the Wireless Communications and Networking Conference, 2022: 626−631.
33	LUO Z J, TIAN P, SONG E G, et al. A refined Dijkstra’s algorithm with stable route generation for topology-varying satellite networks. Proc. of the 41st International Conference on Distributed Computing Systems, 2021: 1146−1147.
34	LI X Y, LI G S, ZHANG S T. Routing space internet based on dijkstra's algorithm. Proc. of the 2nd International Conference on Networks Security, Wireless Communications and Trusted Computing, 2010: 118−121.
35	ZHOU W, ZHANG Q, XIN X J, et al. Extended path-finding RWA algorithm based on ACO in optical satellite network. Proc. of the 18th International Conference on Optical Communications and Networks, 2019. DOI: 1109/ICOCN.2019.8934043.
36	WEN G L, ZHANG Q, XIN X J, et al. Cross-layer design based ant colony optimization for routing and wavelength assignment in an optical satellite network. Proc. of the 15th International Conference on Optical Communications and Networks, 2016. DOI: 10.1109/ICOCN.2016.7875848.
37	XIE F, LONG J, QIAN Z M, et al. Multi-objective optimization routing for satellite network based on ant colony algorithm. Proc. of the 13th International Conference on Measuring Technology and Mechatronics Automation, 2021: 353−356.

Symbol	Meaning
$ {\tau _n} $	Time slice
$ {{{\mathrm{User}}}_{{\tau _n}}} $	Service request set in time slice
$ {v_s} $	Satellite node s
$ {u_{s \to d}} $	Service request from node s to node d
$ {\bf{pat}}{{\bf{h}}_{{\tau _n}}}\left( {{u_{s \to d}}} \right) $	Routing path set for service with source node s and sink node d
$ {e_{i,j}} $	ISL between satellite nodes i and j
$ {\mathrm{band}}{\text{(}}{\cdot}) $	Bandwidth calculation function
$ {\mathrm{delay}}{\text{(}}{\cdot}) $	Time delay calculation function
$ {\mathrm{loss}}{\text{(}}{\cdot}) $	Bit error rate calculation function
$ {\mathrm{Band}} $(·)	Maximum available bandwidth value

[1]	Nanxun DUO, Qinzhao WANG, Qiang LYU, Wei WANG. Tactical reward shaping for large-scale combat by multi-agent reinforcement learning [J]. Journal of Systems Engineering and Electronics, 2024, 35(6): 1516-1529.
[2]	Guofei LI, Shituo LI, Bohao LI, Yunjie WU. Deep reinforcement learning guidance with impact time control [J]. Journal of Systems Engineering and Electronics, 2024, 35(6): 1594-1603.
[3]	Tingting ZHANG, Zijian GUO, Bin LI, Yuan FENG, Qi FU, Mingyu HU, Yunbo QU. Delay-optimal multi-satellite collaborative computation offloading supported by OISL in LEO satellite network [J]. Journal of Systems Engineering and Electronics, 2024, 35(4): 805-814.
[4]	Qi WANG, Zhizhong LIAO. Computational intelligence interception guidance law using online off-policy integral reinforcement learning [J]. Journal of Systems Engineering and Electronics, 2024, 35(4): 1042-1052.
[5]	Guang ZHAN, Kun ZHANG, Ke LI, Haiyin PIAO. UAV maneuvering decision-making algorithm based on deep reinforcement learning under the guidance of expert experience [J]. Journal of Systems Engineering and Electronics, 2024, 35(3): 644-665.
[6]	Yaozhong ZHANG, Zhuoran WU, Zhenkai XIONG, Long CHEN. A UAV collaborative defense scheme driven by DDPG algorithm [J]. Journal of Systems Engineering and Electronics, 2023, 34(5): 1211-1224.
[7]	Jiawei XIA, Xufang ZHU, Zhong LIU, Qingtao XIA. LSTM-DPPO based deep reinforcement learning controller for path following optimization of unmanned surface vehicle [J]. Journal of Systems Engineering and Electronics, 2023, 34(5): 1343-1358.
[8]	Yunxiu ZENG, Kai XU. Recognition and interfere deceptive behavior based on inverse reinforcement learning and game theory [J]. Journal of Systems Engineering and Electronics, 2023, 34(2): 270-288.
[9]	Yaozhong ZHANG, Yike LI, Zhuoran WU, Jialin XU. Deep reinforcement learning for UAV swarm rendezvous behavior [J]. Journal of Systems Engineering and Electronics, 2023, 34(2): 360-373.
[10]	Lu DONG, Zichen HE, Chunwei SONG, Changyin SUN. A review of mobile robot motion planning methods: from classical motion planning workflows to reinforcement learning-based architectures [J]. Journal of Systems Engineering and Electronics, 2023, 34(2): 439-459.
[11]	Guangran CHENG, Lu DONG, Xin YUAN, Changyin SUN. Reinforcement learning-based scheduling of multi-battery energy storage system [J]. Journal of Systems Engineering and Electronics, 2023, 34(1): 117-128.
[12]	Peng LIU, Boyuan XIA, Zhiwei YANG, Jichao LI, Yuejin TAN. A deep reinforcement learning method for multi-stage equipment development planning in uncertain environments [J]. Journal of Systems Engineering and Electronics, 2022, 33(6): 1159-1175.
[13]	Bohao LI, Yunjie WU, Guofei LI. Hierarchical reinforcement learning guidance with threat avoidance [J]. Journal of Systems Engineering and Electronics, 2022, 33(5): 1173-1185.
[14]	Xiaofeng LI, Lu DONG, Changyin SUN. Hybrid Q-learning for data-based optimal control of non-linear switching system [J]. Journal of Systems Engineering and Electronics, 2022, 33(5): 1186-1194.
[15]	Ang GAO, Qisheng GUO, Zhiming DONG, Zaijiang TANG, Ziwei ZHANG, Qiqi FENG. Research on virtual entity decision model for LVC tactical confrontation of army units [J]. Journal of Systems Engineering and Electronics, 2022, 33(5): 1249-1267.

Multi-QoS routing algorithm based on reinforcement learning for LEO satellite networks

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

Share this article

Figures/Tables 8

References 37

Related Articles 15

Recommended Articles

Metrics

Comments