無細胞大規模MIMO系統是一項具有潛力的技術，被提出為5G和6G的關鍵技術之一。不同於傳統的蜂巢式結構，在無細胞大規模MIMO系統中具有一個中央控制器及大量的無線存取點(AP)在覆蓋範圍內，並且每個無線存取點都具備大量的服務天線，能夠同時為覆蓋範圍內的所有用戶進行聯合傳輸。一個關鍵挑戰在於當無線存取點受到流量限制時，要如何選擇服務用戶及功率分配，使所有使用者能夠獲得最佳的資料傳輸率。在本篇論文中，使用深度強化學習(Deep reinforcement learning)技術應用在毫米波無細胞大規模MIMO系統中的用戶選擇及功率分配，透過放入適當的環境資訊和設定回饋方法，並且經過有效的訓練，來達到最適合的多用戶選擇及無線存取點的功率分配。我們的環境資訊包括所有無線存取點對於所有用戶的路徑損耗和通道狀態資訊，獎勵的方法設定為所有用戶的最大頻譜效率，透過隨機分布的無線存取點和用戶來做為訓練的輸入，在訓練結束後，將測試環境放入訓練好的神經網路，就能獲得連續動作，相當於用戶選擇及功率分配。最後根據深度強化學習的結果來計算頻譜效率，能夠證明此方法是具有優勢的。

The cell-free massive MIMO system is a potential technology and has been proposed as one of the key technologies for 5G and 6G. Different from the traditional cellular structure, in a cell-free massive MIMO system there is a central controller and a number of wireless access points within the coverage area, and each access point has a large number of serving antennas. The system is capable of joint transmission for all user equipments within the coverage area at the same time. A key challenge is how to select service user equipments and allocate power so that all user equipments can obtain the better transmission data rate when the wireless access point is limited by traffic load. In this paper, deep reinforcement learning technique is applied to user selection and power allocation in millimeter wave cell-free massive MIMO systems. By putting in the appropriate channel state information and setting the reward method. After effective training, the optimal multi-user selection and power allocation of the access point can be achieved. Our environmental information includes the path loss and the channel state information for all access points for all user equipments. The reward method is set to the maximum spectral efficiency of all UEs. A random distribution of access points and user equipments is used as training data. After training, put the test environment into the trained neural network, and we can get continuous action, which is equivalent to user selection and power allocation. Finally, the spectral efficiency is calculated according to the results of deep reinforcement learning, which can prove the advantage of this method.

[1] Zhouyue Pi and F. Khan, "An introduction to millimeter-wave mobile broadband systems," IEEE Communications Magazine, vol. 49, pp. 101-107, 2011.
[2] S. Rangan, T. S. Rappaport, and E. Erkip, "Millimeter-Wave Cellular Wireless Networks: Potentials and Challenges," Proceedings of the IEEE, vol. 102, no. 3, pp. 366-385, 2014.
[3] L. Lu, G. Y. Li, A. L. Swindlehurst, A. Ashikhmin, and R. Zhang, "An Overview of Massive MIMO: Benefits and Challenges," IEEE Journal of Selected Topics in Signal Processing, vol. 8, no. 5, pp. 742-758, 2014.
[4] J. Zhang, S. Chen, Y. Lin, J. Zheng, B. Ai, and L. Hanzo, "Cell-Free Massive MIMO: A New Next-Generation Paradigm," IEEE Access, vol. 7, pp. 99878-99888, 2019.
[5] S. Elhoushy, M. Ibrahim, and W. Hamouda, "Cell-Free Massive MIMO: A Survey," IEEE Communications Surveys & Tutorials, vol. 24, no. 1, pp. 492-523, 2022.
[6] S. Biswas and P. Vijayakumar, "AP selection in Cell-Free Massive MIMO system using Machine Learning Algorithm," presented at the 2021 Sixth International Conference on Wireless Communications, Signal Processing and Networking (WiSPNET), 2021.
[7] Y. Lin, R. Zhang, L. Yang, and C. L. a. L. Hanzo, "User Centric Clustering for Designing Ultradense Networks.," IEEE Vehicular Technology Magazine, vol. 14, no. 3, pp. 107-114, 2019.
[8] Aditya Akella, Glenn Judd, Srinivasan Seshan, and P. Steenkiste, "Self-Management in Chaotic Wireless Deployments," Proc. ACM MOBICOM, 2005.
[9] Qilin Qi, Andrew Minturn, and Y. L. Yang, "An efficient water filling algorithm for power allocation in OFDM-based cognitive radio systems," ICSAI, 2012.
[10] P.Mangayarkarasi, M.Ramya, and S.Jayashri, "Analysis of various power allocation algorithms for wireless networks," International Conference on Communication and Signal Processing, 2012.
[11] Timothy P. Lillicrap et al., "Continuous control with deep reinforcement learning," ICLR,arXiv preprint arXiv:1509.02971, 2015.
[12] Wang Qiang and Z. Zhongli, "Reinforcement learning model algorithms and its application," In 2011 International Conference on Mechatronic Science, Electric Engineering and Computer, pp. 1143-1146, 2011.
[13] S. Bhatnagar and S. Kumar, "A Simultaneous Perturbation Stochastic Approximation-Based Actor–Critic Algorithm for Markov Decision Processes," IEEE Transactions on Automatic Control, vol. 49, no. 4, pp. 592-598, 2004.
[14] F. Q. Lauzon, "An introduction to deep learning," In 2012 11th International Conference on Information Science, Signal Processing and their Applications (ISSPA) pp. 1438-1439, 2012.
[15] R. Sutton, D. McAllester, S. Singh, and Y. Mansour, "Policy Gradient Methods for Reinforcement Learning with Function Approximation," Proceedings of the 9th Yale Workshop on Adaptive and Learning Systems, vol. 12, pp. 1057-1063, 2000.
[16] S.-F. H. Ronald Y. Chang, and Feng-Tsun Chien, "Reinforcement Learning-Based Joint Cooperation Clustering and Content Caching in Cell-Free Massive MIMO Networks," IEEE 94th Vehicular Technology Conference, 2021.
[17] Y. Zhao, I. G. Niemegeers, and S. M. H. De Groot, "Dynamic Power Allocation for Cell-Free Massive MIMO: Deep Reinforcement Learning Methods," IEEE Access, vol. 9, pp. 102953-102965, 2021.
[18] Yasser Al-Eryani, Mohamed Akrout, and E. Hossain, "Antenna Clustering for Simultaneous Wireless Information and Power Transfer in a MIMO Full-Duplex System: A Deep Reinforcement Learning-Based Design," IEEE Transactions on Communications, vol. 69, no. 4, pp. 2331-2345, 2021.
[19] F. S. Liang Chen, Kai Li, Ruiqing Chen, Yang Yang, Jun Wang, "Deep Reinforcement Learning for Resource Allocation in Massive MIMO," In 2021 29th European Signal Processing Conference (EUSIPCO),IEEE, pp. 1611-1615, 2021.
[20] H. Huang, Y. Yang, H. Wang, Z. Ding, H. Sari, and F. Adachi, "Deep Reinforcement Learning for UAV Navigation Through Massive MIMO Technique," IEEE Transactions on Vehicular Technology, vol. 69, no. 1, pp. 1117-1121, 2020.
[21] Vijay R. Konda and J. N. Tsitsiklis, "Actor-Critic Algorithms," NIPS Proceedings, vol. 42, pp. 1143-1166, 1999.
[22] E. G. Larsson, O. Edfors, F. Tufvesson, and T. L. Marzetta, "Massive MIMO for next generation wireless systems," IEEE Communications Magazine, vol. 52, no. 2, pp. 186-195, 2014.
[23] R. Irmer et al., "Coordinated multipoint Concepts performance and field trial results," IEEE Communications Magazine, vol. 49, no. 2, pp. 102-111, 2011.
[24] Z. Pi and F. Khan, "An introduction to millimeter wave mobile broadband systems," IEEE Communications Magazine, vol. 49, pp. 101-107, 2011.
[25] P. Zhu, H. Mao, J. Li, and X. You, "Energy efficient joint energy cooperation and power allocation in multiuser distributed antenna systems with hybrid energy supply," IET Communications, vol. 13, no. 2, pp. 153-161, 2019.
[26] T. C. Mai, H. Q. Ngo, and T. Q. Duong, "Downlink Spectral Efficiency of Cell-Free Massive MIMO Systems With Multi-Antenna Users," IEEE Transactions on Communications, vol. 68, no. 8, pp. 4803-4815, 2020.
[27] S. Buzzi and C. D'Andrea, "Cell-Free Massive MIMO: User-Centric Approach," IEEE Wireless Communications Letters, vol. 6, no. 6, pp. 706-709, 2017.
[28] H. T. Dao and S. Kim, "Power Allocation and User-AP Connection in Distributed Massive MIMO Systems," IEEE Communications Letters, vol. 25, no. 2, pp. 565-569, 2021.
[29] J. Wang, B. Wang, J. Fang, and H. Li, "Millimeter Wave Cell-Free Massive MIMO Systems: Joint Beamforming and AP-User Association," IEEE Wireless Communications Letters, vol. 11, no. 2, pp. 298-302, 2022.
[30] O. E. Ayach, S. Rajagopal, S. Abu-Surra, Z. Pi, and R. W. Heath, "Spatially Sparse Precoding in Millimeter Wave MIMO Systems," IEEE Transactions on Wireless Communications, vol. 13, no. 3, pp. 1499-1513, 2014.
[31] Mario Alonzo and S. Buzzi, "Cell-free and user-centric massive MIMO at millimeter wave frequencies," In 2017 IEEE 28th Annual International Symposium on Personal, Indoor, and Mobile Radio Communications (PIMRC),IEEE, pp. 1-5, 2017.
[32] Stefano Buzzi, Carmen D’Andrea, and C. D’Elia, "User-Centric Cell-Free Massive MIMO with Interference Cancellation and Local ZF Downlink Precoding," In 2018 15th International Symposium on Wireless Communication Systems (ISWCS),IEEE, pp. 1-5, 2018.
[33] Lucas Claudino and T. Abrao, "Efficient ZF-WF Strategy for Sum-Rate Maximization of MU-MISO Cognitive Radio Networks," AEU-International Journal of Electronics and Communications, vol. 84, pp. 366-374, 2018.
[34] P. V. D. Tse, "Fundamentals Wireless Communications," Cambridge University Press, 2005. Cambridge University Press.
[35] V. Mnih, Kavukcuoglu, Koray, Silver, David,, A. Graves, Antonoglou,Ioannis,stra, Daan, Wierstra, and M. Riedmiller, "Playing Atari with Deep Reinforcement Learning," arXiv preprint arXiv:1312.5602, 2013.
[36] G. L. David Silver, Nicolas Heess, Thomas Degris, Daan Wierstra, Martin Riedmiller, "Deterministic Policy Gradient Algorithms," In International conference on machine learning, pp. 387-395, 2014.
[37] V. Mnih, Kavukcuoglu, Koray, Silver, David, Rusu, Andrei A, Veness, Joel, Bellemare, and G. Marc G, Alex, Riedmiller, Martin, Fidjeland, Andreas K, Ostrovski, Georg, et al., "Human-level control through deep reinforcement learning," Nature, pp. 529-533, 2015. Natrue.
[38] "3rd Generation Partnership Project; Technical Specification Group Radio Access Network; Study on channel model for frequencies from 0.5 to 100 GHz (Release 16)."