隨著行動通訊快速發展，第三代合作夥伴計畫 (3GPP) 於第17 版標準 (Rel-17) 中，於
第五代行動通訊 (5G New Radio，NR) 要容納數以萬計之設備 (User Equipment，UE)，考量
未來行動通訊系統 (Beyond 5G 或 6G) 將同時支援更大量之終端設備，且服務模式由現行下
行主導 (DL dominant) 逐漸改變為上行主導 (UL dominant)，因此需更多頻譜資源傳送資料。
高頻段如FR2 符合頻寬需求。然而，高頻段之特性需使用更大子載波間距 (Sub-Carrier Spacing，
SCS) 抵抗相位噪聲（phase noise）。提高子載波間距也將造成每個時隙 (Slot) 時間縮短，進
一步壓縮資源排程器演算法所需運算時間之上限。若基站無法及時完成上下行資源排程，將
造成資源浪費，違背增加頻寬之初衷。
針對上行傳輸，此研究提出一種新穎之上行排程器—基於深度強化式學習之輕量化排程
器 (DRL-based Lightweight Scheduler，DRL-LS)，依據排隊中封包之資料特徵 (如剩餘延遲預
算 (Remaining Packet Delay Budget，R-PDB) 與服務等級等) 決定上行之優先等級。考慮排程
器演算法運算時間受限，資源排程器僅排程高優先等級封包；針對尚未排程之上行通道資源，
導入基於競爭 (Contention-based) 之通訊協定，使符合條件之上行封包 (亦即較不急迫之封包)
可於此資源內進行隨機競爭，競爭失敗之封包將於後續上行資源進行重傳。當上行封包隨時
間提高優先等級時，將改以傳統排程模式(亦即免競爭 (Contention-free) 模式) 進行排程，以
符合封包延遲預算。此 DRL-LS 透過 DRL 學習如何與通道環境互動於達成最高吞吐量與控
制排程器演算法運算時間之間取得平衡。
最後，為驗證DRL-LS與競爭上行行為之可行性，本研究將上述DRL-LS之代理人 (Agent)
部署於RIC 內，並且使用OpenAirInterface5G (OAI) 與Mosaic5G FlexRIC 開源軟體進行實際
驗證，觀察UEs 進行競爭之行為。

With the rapid development of mobile communications, the 3rd Generation Partnership Project
(3GPP) in Release 17 specification has aimed to accommodate numerous User Equipments (UE) in
5G New Radio (NR) network. Considering the future mobile communication systems (beyond 5G or
6G), an even larger number of devices should be supported. Furthermore, the transmission behavior
is gradually shifting from downlink-dominant (DL dominant) to uplink-dominant (UL dominant).
This shift necessitates more frequency resources, namely bandwidth (BW), for UL data transmission.
Higher frequency bands, such as FR2, can fulfill the bandwidth requirements. However, the
characteristics of higher frequency bands require larger Sub-Carrier Spacing (SCS) to counteract
phase noise (PN). Increasing the SCS also leads to shorter slot duration, which further shortens the
computational time budget of scheduler. If the base station (gNB) fails to accomplish scheduling in
time, it will result in resource waste consequently.
For uplink transmission, this study proposes a novel uplink scheduler named the Deep
Reinforcement Learning-based Lightweight Scheduler (DRL-LS). The DRL-LS determines the
priority of UL data in the queue by its characteristic, such as the Remaining Packet Delay Budget (RPDB)
and the Quality of Service (QoS). Due to the limited time budget for scheduling, the scheduler
only schedules data of high priority in a contention-free manner. For those unscheduled UL data, a
contention-based (CB) UL transmission protocol is proposed, allowing eligible uplink data (i.e., less
urgent data) to access CB resources in a contention manner. Data packets that experience contention
failure will be retransmitted in subsequent UL resources. To meet the packet delay budget, the priority
of queueing UL data increases over time. Additionally, the DRL-LS would learn how to interact with
the environment to achieve a balance between the maximum throughput and media access protocol.
Finally, to validate the performance of the proposed DRL-LS, this study deploys the agent in the
RAN Intelligence Controller (RIC), which is built over the open platform including the
OpenAirInterface5G (OAI) and Mosaic5G FlexRIC open source software.

[1] 3GPP TS 23.501 “5G; NR; System architecture for the 5G System (5GS),” v17.7.0, Jan. 2023.
[2] 3GPP TS 38.101-1 “5G; NR; User Equipment (UE) radio transmission and reception; Part 1:
Range 1 Standalone,” v17.6.0, Aug. 2022.
[3] 3GPP TS 38.101-2 “5G; NR; User Equipment (UE) radio transmission and reception; Part 2:
Range 2 Standalone,” v17.8.0, Jan. 2023.
[4] 3GPP TS 38.211 “5G; NR; Physical channels and modulation,” v17.2.0, Jul. 2022.
[5] 3GPP TS 38.212 “5G; NR; Multiplexing and channel coding,” v17.2.0, Jul. 2022.
[6] 3GPP TS 38.213 “5G; NR; Physical layer procedures for control,” v17.2.0, Jul. 2022.
[7] 3GPP TS 38.214 “5G; NR; Physical layer procedures for data,” v17.2.0, Jul. 2022.
[8] 3GPP TS 38.300 “5G; NR; NR and NG-RAN Overall description; Stage-2,” v17.1.0, Aug. 2022.
[9] 3GPP TS 38.321 “5G; NR; Medium Access Control (MAC) protocol specification,” v17.1.0,
Aug. 2022.
[10] 3GPP TS 38.331 “5G; NR; Radio Resource Control (RRC); Protocol specification,” v17.1.0,
Aug. 2022.
[11] 3GPP TS 38.913 “5G; Study on scenarios and requirements for next generation access
technologies,” v17.0.0, May 2022.
[12] A. Mamane, M. Fattah, M. E. Ghazi, M. E. Bekkali, Y. Balboul and S. Mazer, "Scheduling
Algorithms for 5G Networks and Beyond: Classification and Survey," in IEEE Access, vol. 10,
pp. 51643-51661, 2022.
[13] S. Shakkottai and R. Srikant, ‘‘Scheduling real-time traffic with deadlines over a wireless
channel,’’ in Proc. 2nd ACM Int. Workshop Wireless Mobile Multimedia (WOWMOM). New
York, NY, USA: ACM Press, 1999, pp. 35–42. [Online]. Available:
http://portal.acm.org/citation.cfm?doid=313256.313273.
[14] X. Yuan and Z. Duan, ‘‘Fair Round-Robin: A low complexity packet scheduler with
proportional and worst-case fairness,’’ IEEE Trans. Comput., vol. 58, no. 3, pp. 365–379, Mar.
2009. [Online]. Available: http://ieeexplore.ieee.org/document/4626953/
[15] S. Ramabhadran and J. Pasquale, “The Stratified Round Robin Scheduler: Design, Analysis,
and Implementation,” IEEE/ACM Trans. Networking, vol. 14, no. 6, pp. 1362-1373, Dec. 2006.
[16] A. Silberschatz, P. B. Galvin, and G. Gagne, “Operating System Concepts,” nineth edition.
[17] F. Kelly, “Charging and rate control for elastic traffic,” Eur. Trans. Telecommun., vol. 8, pp.
33–37, Jan./Feb. 1997.
[18] 3GPP TSG-RAN WG2 Meeting #115 electronic, “Report of [AT115-e][042][eIAB] fairness,
latency and congestion (Interdigital),” Aug. 2021.
[19] R.S. Sutton and A.G. Barto, “Reinforcement learning: An introduction.,” Cambridge, MA,
USA: MIT press, 2018.
[20] V. Mnih et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518,
no. 7540, pp. 529-533, Feb. 2015.
[21] I. Goodfellow, Y. Bengio, and A. Courville, “Deep learning,” Cambridge, MA, USA: MIT
press, 2016.
[22] O-RAN TS “O-RAN Architecture Description,” WG1, OAD-R003-v08.00.
[23] O-RAN TR “AI/ML workflow description and requirements,” WG2, AIML-v01.03.
[24] O-RAN TS “O-RAN Operations and Maintenance Interface Specification,” WG1, O1-
Interface.0-v04.00.
[25] O-RAN TS “Near-RT RIC Architecture,” WG3, RICARCH-R003-v04.00.
[26] O-RAN TS “E2 General Aspects and Principles,” WG3, E2GAP-R003-v03.00.
[27] R. Enns, M. Bjorklund, J. Schoenwaelder, and A. Bierman, “Network configuration protocol
(NETCONF),” Internet Requests for Comments, RFC Editor, RFC 6241, June 2011. [Online].
Available: http://www.rfc- editor.org/rfc/rfc6241.txt
[28] M. Polese, L. Bonati, S. D'Oro, S. Basagni, T. Melodia, “Understanding O-RAN: Architecture,
Interfaces, Algorithms, Security, and Research Challenges", arXiv:2202.01032 [cs.NI], Feb.
2022.
[29] K. Boutiba, M. Bagaa and A. Ksentini, “On using Deep Reinforcement Learning to reduce
Uplink Latency for uRLLC services," GLOBECOM 2022 - 2022 IEEE Global
Communications Conference, Rio de Janeiro, Brazil, pp. 407-412, 2022.
[30] Hossain, Q. C. Li, D. Ying, G. Wu and C. Xiong, “THz Channel Model for 6G
Communications,” 2021 IEEE 32nd Annual International Symposium on Personal, Indoor and
Mobile Radio Communications (PIMRC), Helsinki, Finland, pp. 1-7, 2021.
[31] Y. Xing and T. S. Rappaport, “Propagation Measurement System and Approach at 140 GHz-
Moving to 6G and Above 100 GHz,” 2018 IEEE Global Communications Conference
(GLOBECOM), Abu Dhabi, United Arab Emirates, pp. 1-6, 2018.
[32] S. Tarboush, H. Sarieddeen, M. -S. Alouini and T. Y. Al-Naffouri, “Single- Versus Multicarrier
Terahertz-Band Communications: A Comparative Study,” in IEEE Open Journal of the
Communications Society, vol. 3, pp. 1466-1486, 2022.
[33] T. S. Rappaport et al., “Wireless Communications and Applications Above 100 GHz:
Opportunities and Challenges for 6G and Beyond,” in IEEE Access, vol. 7, pp. 78729-78757,
2019.
[34] L. You, Q. Liao, N. Pappas and D. Yuan, “Resource Optimization with Flexible Numerology
and Frame Structure for Heterogeneous Services,” in IEEE Communications Letters, vol. 22,
no. 12, pp. 2579-2582, Dec. 2018.
[35] Y. Yu, T. Wang, and S. C. Liew, “Deep-Reinforcement Learning Multiple Access for
Heterogeneous Wireless Networks,” 2018 IEEE International Conference on Communications
(ICC), Kansas City, MO, USA, 2018.
[36] Y. Yu, S. C. Liew and T. Wang, “Non-Uniform Time-Step Deep Q-Network for Carrier-Sense
Multiple Access in Heterogeneous Wireless Networks,” in IEEE Transactions on Mobile
Computing, vol. 20, no. 9, pp. 2848-2861, 1 Sept. 2021.
[37] Y. Yu, S. C. Liew and T. Wang, “Multi-Agent Deep Reinforcement Learning Multiple Access
for Heterogeneous Wireless Networks with Imperfect Channels,” in IEEE Transactions on
Mobile Computing, vol. 21, no. 10, pp. 3718-3730, 1 Oct. 2022.
[38] M. Elsayem, H. Abou-zeid, A. Afana and S. Givigi, “Reinforcement Learning-based Dynamic
Resource Allocation for Grant-Free Access,” GLOBECOM 2022 - 2022 IEEE Global
Communications Conference, Rio de Janeiro, Brazil, pp. 1091-1096, 2022.
[39] K. Boutiba, M. Bagaa and A. Ksentini, “Radio Resource Management in Multi-numerology 5G
New Radio featuring Network Slicing,” ICC 2022 - IEEE International Conference on
Communications, Seoul, Korea, Republic of, pp. 359-364, 2022.
[40] Q. Zhou, X. Ye and L. Fu, “Deep Reinforcement Learning Based Scheduling Scheme for the
NR-U/WiGig Coexistence in Unlicensed mmWave Bands,” ICC 2022 - IEEE International
Conference on Communications, Seoul, Korea, Republic of, 2022.
[41] C. J. C. H. Watkins and P. Dayan, “Q-learning,” Machine Learning, vol. 8, no. 3, pp. 279–292,
May 1992. [Online]. Available: https://doi.org/10.1007/BF00992698.
[42] OpenAir5GLAB, http://openairinterface.eurecom.fr
[43] OpenAirInterface5G, Gitlab Open Source, https://gitlab.eurecom.fr/oai/openairinterface5g
[44] FlexRIC, Gitlab Open Source, https://gitlab.eurecom.fr/mosaic5g/flexric
[45] S. Parkvall, E. Dahlman, A. Furuskar, and M. Frenne, “NR: The new 5G radio access
technology,” IEEE Communications Standards Magazine, vol. 1, no. 4, pp. 24-30, Dec. 2017.
[46] T. H. Lee and A. Hajimiri, “Oscillator phase noise: a tutorial,” IEEE Journal of Solid-State
Circuits, vol. 35, no. 3 pp. 326-336, March 2000
[47] Robert Schmidt, “Slicing in heterogeneous software-defined radio access networks,”
Networking and Internet Architecture [cs.NI]. Sorbonne Université, 2021. English, NNT:
2021SORUS525, tel- 03783488.
[48] Ettus Research, USRP B210, https://www.ettus.com/all-products/ub210-kit/
[49] Yan Huang, Shaoran, Tom Hou, and Wenjing Lou, “GPU-Based Design to Achieve 100 𝜇𝑠
Scheduling for 5G NR,” Nvidia, Mar. 2019. [Online]. Available:
https://developer.nvidia.com/blog/gpu-based-100-μs-schedulingr-5g-nr/
[50] “White paper on 6G vision and Candidate Technologies,” IMT-2030 6G Promotion Group, Jun.
2021.