結合類比和數位元件的混合式波束成型架構，是為了減少在毫米波 (mmWave) 頻段附有大規模多輸出多輸入 (MIMO) 系統中的硬體成本和功率消耗的一個替代方案。然而，由於通訊系統越來越複雜且伴隨著傳輸數據量大，人工智慧 (AI) 是一個有希望的輔助工具被用來協助我們解決高維和複雜的問題。在這篇論文中，我們著重的是藉由基於AI的子通道分配方法來最大化資料傳輸率同時考慮所有使用者的服務品質 （QoS），而且我們認為射頻鏈 (RF chain) 的數量在實際的混合波束成型的架構中很稀少。這不只讓子通道分配對於大規模MIMO附加正交分頻多址 (MIMO-OFDMA) 和mmWave系統很重要，而且能夠讓系統在一個訊框中服務更多使用者。與傳統基於迭代的子載波配置方法不同，我們使用深度強化學習 (DRL) 演算法去解決即時策略問題。更進一步，我們提出基於競爭深度雙Q網路 (Dueling-DDQN) 去實行動態的子通道分配。
數值結果顯示提出方法的性能在混合式波束成型架構的測試集中隨著訓練漸漸趨近於貪婪法的性能，而且平均總和傳輸率和每位使用者的平均頻譜效益在合理的中斷率變化範圍內也都有提升。

Hybrid beamforming, which combines analog and digital components, is an alternative for less power and hardware cost consumption in millimeter-wave (mmWave) with massive multiple-input multiple-output (MIMO) systems. However, because communication systems become more sophisticated and come with explosive transmission data, artificial intelligence (AI) is a promising auxiliary approach for assisting us to solve high-dimensional and complex problems. In this paper, we emphasize that the maximum sum rate is reached by AI-based subchannel allocation while considering all users’ quality of service (QoS) for data rate, and we assume that the number of radio frequency (RF) chains is rare in practical hybrid beamforming architecture. This practical assumption not only makes the subchannel allocation important for hybrid beamforming in the massive MIMO with Orthogonal frequency division multiple access (MIMO-OFDMA) and mmWave systems but also enables the system to serve more users at a time slot. Different from the conventional subcarrier allocation algorithms, we use a deep reinforcement learning (DRL) algorithm to solve real-time decision-making problems. Further, we propose the dueling double deep Q-network (Dueling-DDQN) to implement the dynamic subchannel allocation.
Simulation results show that the rate performance of the proposed algorithm in the testing set for hybrid beamforming architecture is gradually close to the greedy method with training. Moreover, the average sum rate and the average spectral efficiency of each user are also raised on the reasonable change of outage probability range.

[1]nK. Mouratidis, A. Papagiannakis, “COVID-19, internet, and mobility: The rise of telework, telehealth, e-learning, and e-shopping,’’ in Sustainable Cities and Society,74 (2021) 103182.
[2]nM. Kamal, A. Aljohani, and E. Alanazi, ‘‘IoT meets COVID-19: Status, challenges, and opportunities,’’ 2020, arXiv:2007.12268. [Online]. Available: http://arxiv.org/abs/2007.12268.
[3] J. G. Andrews et al., ‘‘What Will 5G Be?,’’ in IEEE Journal on Selected Areas in Communications, vol. 32, no. 6, pp. 1065-1082, June 2014.
[4] T. S. Rappaport et al., ‘‘Millimeter Wave Mobile Communications for 5G Cellular: It Will Work!” in IEEE Access, vol. 1, pp. 335-349, May 2013.
[5] S. K. Yong, C. C. Chong, “An overview of multigigabit wireless through millimeter wave technology: Potentials and technical challenges,” EURASIP Journal on Wireless Communications and Networking, vol. 2007, no. 1, pp. 50–50, 2006.
[6] S. A. Hoseini, M. Ding, and M. Hassan, “Massive MIMO performance comparison of beamforming and multiplexing in the Terahertz band,” in Proc. IEEE Globecom Workshops, Dec. 2017, pp. 1–6.
[7] A. Swindlehurst, E. Ayanoglu, P. Heydari, and F. Capolino, “Millimeter-wave massive MIMO: The next wireless revolution?” IEEE Communication Magazine, vol. 52, no. 9, pp. 56–62, Sep. 2014.
[8] L. Lu, G. Y. Li, A. L. Swindlehurst, A. Ashikhmin and R. Zhang, ‘‘An Overview of Massive MIMO: Benefits and Challenges,” in IEEE Journal of Selected Topics in Signal Processing, vol. 8, no. 5, pp. 742-758, Oct. 2014.
[9] H. Q. Ngo, E. G. Larsson, and T. L. Marzetta, “Energy and Spectral Efficiency of Very Large Multiuser MIMO Systems,” IEEE Trans. Commun., vol. 61, Apr. 2013, pp. 1436–1449.
[10] O. El Ayach, S. Rajagopal, S. Abu-Surra, Z. Pi, and R. W. Heath, “Spatially sparse precoding in millimeter wave MIMO systems,” IEEE Trans. Wireless Commun., vol. 13, no. 3, pp. 1499–1513, Mar. 2014.
[11] T. E. Bogale, L. B. Le, A. Haghighat, and L. Vandendorpe, “On the number of RF chains and phase shifters, and scheduling design with hybrid analog–digital beamforming,” IEEE Trans. Wireless Commun., vol. 15, no. 5, pp. 3311–3326, May 2016.
[12] J. Zhang, X. Yu and K. B. Letaief, ‘‘Hybrid Beamforming for 5G and Beyond Millimeter-Wave Systems: A Holistic View,” in IEEE Open Journal of the Communications Society, vol. 1, pp. 77-91, 2020.
[13] A. F. Molisch et al., “Hybrid beamforming for massive MIMO: A survey,” IEEE Commun. Mag., vol. 55, no. 9, pp. 134–141, Sep. 2017.
[14] T. F. Maciel and A. Klein, ‘‘A resource allocation strategy for SDMA/OFDMA systems,” IST Mobile and Wireless Communications Summit, 1-5 July 2007.
[15] C. F. Tsai, C. J. Tsang, F.-C. Ren, and C.-M. Yen, “Adaptive radio resource allocation for downlink OFDMA/SDMA systems,” in Proc. IEEE ICC, Glasgow, U.K., 2007, pp. 5683–5688.
[16] Y. F. Chen and J. W. Chen, “A fast subcarrier, bit, and power allocation algorithm for multiuser OFDM-based systems,” in IEEE Trans. Veh. Technol., vol. 57, no. 2, pp. 873–881, Mar. 2008.
[17] Li, Suo Ping, X. F. Tian, and M. Li, “Study on Subcarrier and Power Allocation Algorithm for Downlink Transmission in Multiuser OFDM System,” in International Symposium on Computer Science and Computational Technology, 2008, pp. 295-298.
[18] Y. Yang, “Analysis of the Impact of Artificial Intelligence Development on Employment,” in International Conference on Computer Engineering and Application (ICCEA), 2020, pp. 324-327.
[19] S. Cheng, B. Wang, “An overview of Publications on Artificial Intelligence Research: A Quantitative Analysis of Recent Papers,” in Fifth International Joint Conference on Computational Sciences and Optimization, June 2012, pp. 683-686.
[20] C. Luo, L. Zhou, Q. Wei, “Identification of Research Fronts in Artificial Intelligence,” in 2017 2nd Asia-Pacific Conference on Intelligent Robot Systems (ACIRS), 2017, pp. 104-108.
[21] R. Li et al., “Intelligent 5G: When cellular networks meet artificial intelligence,” IEEE Wireless Commun., vol. 24, no. 5, pp. 175–183, Oct. 2017.
[22] R. Shafin, L. Liu, V. Chandrasekhar, H. Chen, J. Reed, and J. C. Zhang, “Artificial intelligence-enabled cellular networks: A critical path to beyond-5G and 6G,” IEEE Wireless Commun., vol. 27, no. 2, pp. 212–217, Apr. 2020.
[23] C. Zhang, P. Patras, and H. Haddadi, “Deep learning in mobile and wireless networking: A survey,” IEEE Commun. Surveys Tuts., vol. 21, no. 3, pp. 2224–2287, 3rd Quart, 2019.
[24] J. Liu, X. Ma, W. Han, L. Wang, “Resource Allocation in OFDMA Networks with Deep Reinforcement Learning,” IEEE 8th International Conference on Information, Communication and Network (ICICN), pp.111-117, 2020.
[25] R. Dong, C. She, W. Hardjawana, Y. Li, B. Vucetic, “Deep learning for radio resource allocation with diverse quality-of-service requirements in 5G,” IEEE Trans. Wireless Commun., vol. 20, no. 4, pp. 2309–2324, April 2021.
[26] Y. Yuan, Z. Yuan, and L. Tian, “5G Non-Orthogonal Multiple Access Study in 3GPP,” IEEE Commun. Mag., vol. 58, no. 7, July 2020, pp. 90–96.
[27] W. Shin, M. Vaezi, B. Lee, D. J. Love, J. Lee, H. V. Poor, “Non-Orthogonal Multiple Access in Multi-Cell Networks: Theory, Performance, and Practical Challenges,” IEEE Communications Magazine, vol. 55, no. 10, pp. 176-183, Oct. 2017.
[28] Y. Liu, Z. Qin, M. Elkashlan, Z. Ding, A. Nallanathan, and L. Hanzo, “Nonorthogonal multiple access for 5G and beyond,” Proceedings of the IEEE, vol. 105, no. 12, pp. 2347–2381, Dec. 2017.
[29] S. M. Tseng, Y. F. Chen, C. S. Tsai, W. D. Tsai, “Deep-Learning-Aided Cross-Layer Resource Allocation of OFDMA/NOMA Video Communication Systems,” IEEE Access, vol. 07, pp. 157730-157740, Oct. 2019.
[30] Y. Zhang, X. Wang, Y. Xu, “Energy-Efficient Resource Allocation in Uplink NOMA Systems with Deep Reinforcement Learning,” 2019 11th International Conference on Wireless Communications and Signal Processing (WCSP), 2019.
[31] C. He, Y. Hu, Y. Chen, B. Zeng, “Joint Power Allocation and Channel Assignment for NOMA with Deep Reinforcement Learning,” IEEE Journal on Selected Areas in Communications, vol. 37, no. 10, pp. 2200-2210, Oct. 2019.
[32] W. Saetan, S. Thipchaksurat, “Power Allocation for Sum Rate Maximization in 5G NOMA System with Imperfect SIC: A Deep Learning Approach,” 2019 4th International Conference on Information Technology (InCIT), pp. 195-198, 2019.
[33] M. Liu, T. Song, G. Gui, “Deep Cognitive Perspective: Resource Allocation for NOMA-Based Heterogeneous IoT With Imperfect SIC,” IEEE Internet of Things Journal, vol. 6, no. 2, pp. 2885-2894, Apr. 2019.
[34] J. Xu, B. Ai, Y. Sun, Y. Chen, “Power Allocation for Millimeter-Wave Railway Systems with Multi-Agent Deep Reinforcement Learning,” 2020 IEEE Global Communications Conference, 2020.
[35] I. Ahmed, H. Khammari, “Joint machine learning based resource allocation and hybrid beamforming design for massive MIMO systems,” in Proc. IEEE Globecom Workshops, Dec. 2018.
[36] E. G. Larsson, O. Edfors, F. Tufvesson and T. L. Marzetta, ‘‘Massive MIMO for next generation wireless systems,” in IEEE Communications Magazine, vol. 52, no. 2, pp. 186-195, February 2014.
[37] F. Sohrabi and W. Yu, “Hybrid Analog and Digital Beamforming for mmWave OFDM Large-Scale Antenna Arrays,” IEEE J. Sel. Areas Commun., vol. 35, no. 7, pp. 1432–1443, Jul. 2017.
[38] Z. Pi and F. Khan, “An introduction to millimeter-wave mobile broadband systems,” IEEE Commun. Mag., vol. 49, no. 6, pp. 101–107, Jun. 2011.
[39] R. W. Heath, N. González-Prelcic, S. Rangan, W. Roh, and A. M. Sayeed, “An overview of signal processing techniques for millimeter wave MIMO systems,” IEEE J. Sel. Topics Signal Process., vol. 10, no. 3, pp. 436–453, Apr. 2016.
[40] 3GPP Technical Report 38.901 v16.1.0, Study on channel model for frequencies from 0.5 to 100 GHz (Release 16), 2020.
[41] V. Mnih, K. Kavukcuoglu, D. Silver, et al., ‘‘Human-level control through deep reinforcement learning,” Nature, vol. 518, no. 7540, pp. 529-533, Feb.2015.
[42] 3GPP Technical specification 38.211 v15.2.0, Physical channels and modulation (Release 15), 2018.
[43] 3GPP Technical specification 38.214 v15.3.0, Physical layer procedures for data (Release 15), 2018.
[44] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” Nature, vol. 323, no. 6088, pp.533-536,1986.
[45] Z. Wang, N. de Freitas, and M. Lanctot, “Dueling Network Architectures for Deep Reinforcement Learning,” In Proceedings of the 33rd International Conference on Machine Learning (ICML), 2016.
[46] A. F. Agarap, “Deep learning using rectified linear units (relu),” arXiv preprint arXiv:1803.08375, 2018.
[47] H. van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double Q-learning,” in Proc. Association for the Advancement of Artificial Intelligence, pp. 2094–2100, 2016.
[48] T. Hastie, R. Tibshirani, and J. Friedman, “The Elements of Statistical Learning,” ser. Springer Series in Statistics. New York, NY, USA: Springer New York Inc., 2001.
[49] V. N. Ha, D. H. N. Nguyen, and J. Frigon, “Subchannel allocation and hybrid precoding in millimeter-wave OFDMA systems,” IEEE Trans. Wireless Commun., vol.17, no.9, pp.5900-5914, Sep. 2018.
[50] Y. Kwon, J. Chung, and Y. Sung, “Hybrid beamformer design for mmWave wideband multiuser MIMO-OFDM systems: (Invited paper),” in Proc. IEEE 18th Int. Workshop Signal Process. Adv. Wireless Commun. (SPAWC), Sapporo, Japan, Jul. 2017, pp. 1–5.
[51] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[52] 3GPP Technical Specification 38.104 v16.5.0, Base Station (BS) radio transmission and reception (Release 16), 2020.
[53] X. Wan, “Influence of feature scaling on convergence of gradient iterative algorithm,” J. Physics: Conf. Series, vol. 1213, no. 3, pp. 1-5, 2019.
[54] R. Pascanu, T. Mikolov, and Y. Bengio, “On the difficulty of training recurrent neural networks,” In ICML, 2013.