隨著5G技術的快速發展，網路流量不斷增加且對於低延遲的需求日益強烈。衛星網路因此成為下一代通訊網路的關鍵組成部分。然而，由於地球上陸地和海洋分佈的不均，導致全球人口分布存在差異。這使得在衛星網路中使用傳統的路由算法可能導致衛星節點在穿越人口密集區域時出現傳輸壅塞的情況。為了確保衛星網路的性能，並應對不斷增長的網路流量，本研究提出了一種基於強化學習的路由方法，旨在實現流量的有效分配。我們的方法採用集中式管理，融合衛星鏈路的動態資訊，包括剩餘頻寬、傳輸延遲、可用時間和接收節點的平均等待時間作為學習參數。我們採用雙重深度強化學習網路（DDQN）模型來最佳化路由決策。通過與其他負載平衡路由方法進行性能比較，我們在不同網路場景和傳輸需求下進行模擬，結果顯示我們提出的路由方法在較小傳輸延遲的情況下，在剩餘頻寬和封包遺失率等方面表現出色。

With the rapid development of 5G technology, there is a continuous increase in network traffic and a growing demand for low latency. Consequently, satellite networks have become a crucial component of the next-generation communication infrastructure. However, due to the uneven distribution of land and oceans on Earth, there exist disparities in the global population distribution. This disparity can lead to transmission congestion for satellite nodes when passing through densely populated areas, if traditional routing algorithms are employed. In order to ensure the performance of satellite networks and address the ever-growing network traffic, this study introduces a routing approach based on reinforcement learning aimed at achieving effective traffic allocation. Our method involves centralized management and integrates dynamic information from satellite links, including remaining bandwidth, transmission delay, available time, and average waiting time of receiving nodes, as learning parameters. We utilize a Double Deep Q-Network (DDQN) model to optimize routing decisions. Through performance comparisons with other load-balancing routing methods, we conduct simulations under various network scenarios and transmission requirements. The results demonstrate that our proposed routing method exhibits superior performance in terms of reduced transmission delay, improved remaining bandwidth utilization, and reduced packet loss rate.

[1] Junfeng Wang , Lei Li , Mingtian Zhou, “Topological dynamics characterization for LEO satellite networks”, ScienceDirect Computer Network, Pages 43-53 , 2007

[2] L. Zhang et al., "A Routing Algorithm Based on Link State Information for LEO Satellite Networks," 2020 IEEE Globecom Workshops (GC Wkshps, Taipei, Taiwan, 2020, pp. 1-6, doi: 10.1109/GCWkshps50303.2020.9367496.

[3] H. Liming, K. Shaoli, S. Shaohui, M. Deshan, H. Bo and Z. Meiting, "A load balancing routing method based on real time traffic in LEO satellite constellation space networks," 2022 IEEE 95th Vehicular Technology Conference: (VTC2022-Spring), Helsinki, Finland, 2022, pp. 1-5, doi: 10.1109/VTC2022-Spring54318.2022.9860540.

[4] C. Dong, X. Xu, A. Liu and X. Liang, "Load balancing routing algorithm based on extended link states in LEO constellation network," in China Communications, vol. 19, no. 2, pp. 247-260, Feb. 2022, doi: 10.23919/JCC.2022.02.020.

[5] J. Liu, R. Luo, T. Huang and C. Meng, "A Load Balancing Routing Strategy for LEO Satellite Network," in IEEE Access, vol. 8, pp. 155136-155144, 2020, doi: 10.1109/ACCESS.2020.3017615.

[6] P. Liu, H. Chen, S. Wei, L. Li, and Z. Zhu, ‘‘Hybrid-traffic-detour based load balancing for onboard routing in LEO satellite networks,’’ China Commun., vol. 15, no. 6, pp. 28–41, Jun. 2018

[7] Yang, Li, and Jing Sun. "Multi-service routing algorithm based on GEO/LEO satellite networks." 2016 International Conference on Network and Information Systems for Computers (ICNISC). IEEE, 2016.

[8] J. Huang, W. Liu, Y. Su, and F. Wang, ‘‘Load balancing strategy and its lookup-table enhancement in deterministic space delay/disruption tolerant networks,’’ Adv. Space Res., vol. 61, no. 3, pp. 811–822, Feb. 2018.

[9] Y. Huang, W. Cao, X. Liu, X. Jiang, J. Yang and F. Yang, "An Adaptive Multipath Routing for LEO Satellite Network," 2021 IEEE 4th Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC), Chongqing, China, 2021, pp. 1536-1541, doi: 10.1109/IMCEC51613.2021.9482379.

[10] Yin, Yabo, et al. "Reinforcement learning-based routing algorithm in satellite-terrestrial integrated networks." Wireless Communications and Mobile Computing 2021 (2021): 1-15.

[11] K. -C. Tsai, T. -J. Yao, P. -H. Huang, C. -S. Huang, Z. Han and L. -C. Wang, "Deep Reinforcement Learning-Based Routing for Space-Terrestrial Networks," 2022 IEEE 96th Vehicular Technology Conference (VTC2022-Fall), London, United Kingdom, 2022, pp. 1-5, doi: 10.1109/VTC2022-Fall57202.2022.10013028.

[12] Y. Zhao, H. Yao, Z. Qin and T. Mai, "Collaborate Q-learning Aided Load Balance in Satellites Communications," 2021 International Wireless Communications and Mobile Computing (IWCMC), Harbin City, China, 2021, pp. 968-973, doi: 10.1109/IWCMC51323.2021.9498837

[13] T. Li, H. Zhou, H. Luo, W. Quan and S. Yu, "Modeling software defined satellite networks using queueing theory," 2017 IEEE International Conference on Communications (ICC), Paris, France, 2017, pp. 1-6, doi: 10.1109/ICC.2017.7997290.

[14] M. Jarschel, S. Oechsner, D. Schlosser, R. Pries, S. Goll and P. Tran-Gia, "Modeling and performance evaluation of an OpenFlow architecture," 2011 23rd International Teletraffic Congress (ITC), San Francisco, CA, USA, 2011, pp. 1-7.

[15] B. Yang, C. Xi, G. Li, P. Liu and R. Zhu, "Deep Reinforcement Learning-Based Satellite-Ground Links Scheduling for Mega Satellite Constellations," 2022 Asia Communications and Photonics Conference (ACP), Shenzhen, China, 2022, pp. 1284-1288, doi: 10.1109/ACP55869.2022.10089120.

[16] Iridium parameter, https://www.iridium.com/network/