本論文為探討如何使用強化學習解決在感知無線電網路中多通道交會問題。感知無線電網路中多通道交會問題是指兩個二級用戶如何在有限的時間內選擇跳到同個通道並成功交換彼此的訊息。本篇論文是在一個對稱性、同步、同質性以及全局共同標號的假設下進行研究，同時為了更貼近實際情況，我們假設每個通道有著不同的通道狀態，並且無法讓用戶得知該通道的狀態情況，某些狀態可能會導致雖然兩個用戶在同個通道卻無法成功通訊。此種無法得知通道狀態的情況下的交會問題稱作盲交會。我們在上述假設下提出了一個快速強化學習演算法，讓兩個用戶學習在不同的狀態下成功交會的交會策略。我們提出的快速強化學習演算法能大幅增加通道選擇策略學習的效率，並且在收斂後學到一組能與已有最佳解的特定通道狀態相比擬的平均交會時間（ETTR），與在沒有最佳解的狀態下有著最佳的效能（最低的ETTR）。

In this thesis, we consider the multichannel rendezvous problem in cognitive radio networks (CRNs) where the probability that two users hopping on the same channel have a successful rendezvous is a function of channel states. The channel states are modelled by stochastic processes with joint distributions known to users. However, the exact state of a channel at any time is not observable. We derived that the lower bound of the ETTR of the general channel model is the ETTR of the fast time-varying channel model and the upper bound is the ETTR of the slow time-varying channel model. By formulating such a multichannel rendezvous problem as an adversarial bandit problem, we propose using a reinforcement learning approach to learn the channel selection probabilities pi(t), i = 1; 2; : : : ;N. Our experimental results show that the reinforcement learning approach is very effective and yields comparable ETTRs when comparing to various approximation policies in the literature.

［1］Kolodzy, Paul, and Interference Avoidance. "Spectrum policy task force." Federal Commun. Comm., Washington, DC, Rep. ET Docket 40.4 (2002): 147-158.
［2］Mitola, Joseph, and Gerald Q. Maguire. "Cognitive radio: making software radios more personal." IEEE personal communications 6.4 (1999): 13-18.
［3］Chen, Tao, et al. "CogMesh: A cluster-based cognitive radio network." 2007 2nd IEEE international symposium on new frontiers in dynamic spectrum access networks. IEEE, 2007.
［4］Le, Long, and Ekram Hossain. "A MAC protocol for opportunistic spectrum access in cognitive radio networks." 2008 IEEE Wireless Communications and Networking Conference. IEEE, 2008.
［5］Zhang, Xi, and Hang Su. "CREAM-MAC: Cognitive radio-enabled multi-channel MAC protocol over dynamic spectrum access networks." IEEE Journal of Selected Topics in Signal Processing 5.1 (2010): 110-123.
［6］Zhao, Jun, Haitao Zheng, and Guang-Hua Yang. "Distributed coordination in dynamic spectrum allocation networks." First IEEE International Symposium on New Frontiers in Dynamic Spectrum Access Networks, 2005. DySPAN 2005.. IEEE, 2005.
［7］Chang, Guey-Yun, and Jen-Feng Huang. "A fast rendezvous channel-hopping algorithm for cognitive radio networks." IEEE Communications Letters 17.7 (2013): 1475-1478.
［8］Shih, C-F., Tsung Ying Wu, and Wanjiun Liao. "DH-MAC: A dynamic channel hopping MAC protocol for cognitive radio networks." 2010 IEEE International Conference on Communications. IEEE, 2010.
［9］Bahl, Paramvir, Ranveer Chandra, and John Dunagan. "SSCH: slotted seeded channel hopping for capacity improvement in IEEE 802.11 ad-hoc wireless networks." Proceedings of the 10th annual international conference on Mobile computing and networking. 2004.
［10］Bian, Kaigui. "Maximizing rendezvous diversity in rendezvous protocols for decentralized cognitive radio networks." IEEE transactions on Mobile Computing 12.7 (2012): 1294-1307.
［11］Bian, Kaigui, Jung-Min Park, and Ruiliang Chen. "A quorum-based framework for establishing control channels in dynamic spectrum access networks." Proceedings of the 15th annual international conference on Mobile computing and networking. 2009.
［12］Chang, Guey-Yun, et al. "Novel channel-hopping schemes for cognitive radio networks." IEEE Transactions on Mobile Computing 13.2 (2012): 407-421.
［13］DaSilva, Luiz A., and Igor Guerreiro. "Sequence-based rendezvous for dynamic spectrum access." 2008 3rd IEEE Symposium on New Frontiers in Dynamic Spectrum Access Networks. IEEE, 2008.
［14］Hou, Fen, et al. "Asynchronous multichannel MAC design with difference-set-based hopping sequences." IEEE Transactions on Vehicular Technology 60.4 (2011): 1728-1739.
［15］Kondareddy, Yogesh Reddy, and Prathima Agrawal. "Synchronized MAC protocol for multi-hop cognitive radio networks." 2008 IEEE International Conference on Communications. IEEE, 2008.
［16］Lin, Zhiyong, et al. "Jump-stay based channel-hopping algorithm with guaranteed rendezvous for cognitive radio networks." 2011 Proceedings IEEE INFOCOM. IEEE, 2011.
［17］Mo, Jeonghoon, Hoi-Sheung Wilson So, and Jean Walrand. "Comparison of multichannel MAC protocols." IEEE Transactions on mobile computing 7.1 (2007): 50-65.
［18］Shin, Jongmin, Dongmin Yang, and Cheeha Kim. "A channel rendezvous scheme for cognitive radio networks." IEEE Communications Letters 14.10 (2010): 954-956.
［19］Wang, Jen-Hung, et al. "A Reinforcement Learning Approach for the Multichannel Rendezvous Problem." 2019 IEEE Globecom Workshops (GC Wkshps). IEEE, 2019.
［20］Yang, D., J. Shin, and C. Kim. "Deterministic rendezvous scheme in multichannel access networks." Electronics Letters 46.20 (2010): 1402-1404.
［21］Zhang, Yifan, et al. "ETCH: Efficient channel hopping for communication rendezvous in dynamic spectrum access networks." 2011 Proceedings IEEE INFOCOM. IEEE, 2011.
［22］Bian, Kaigui, Jung-Min Park, and Ruiliang Chen. "A quorum-based framework for establishing control channels in dynamic spectrum access networks." Proceedings of the 15th annual international conference on Mobile computing and networking. 2009.
［23］Yang, D., J. Shin, and C. Kim. "Deterministic rendezvous scheme in multichannel access networks." Electronics Letters 46.20 (2010): 1402-1404.
［24］Theis, Nick C., Ryan W. Thomas, and Luiz A. DaSilva. "Rendezvous for cognitive radios." IEEE transactions on mobile computing 10.2 (2010): 216-227.
［25］Lin, Zhiyong, et al. "Jump-stay based channel-hopping algorithm with guaranteed rendezvous for cognitive radio networks." 2011 Proceedings IEEE INFOCOM. IEEE, 2011.
［26］Gu, Zhaoquan, et al. "Nearly optimal asynchronous blind rendezvous algorithm for cognitive radio networks." 2013 IEEE international conference on sensing, communications and networking (SECON). IEEE, 2013.
［27］Chang, Guey-Yun, and Jen-Feng Huang. "A fast rendezvous channel-hopping algorithm for cognitive radio networks." IEEE Communications Letters 17.7 (2013): 1475-1478.
［28］Chang, Guey-Yun, et al. "Novel channel-hopping schemes for cognitive radio networks." IEEE Transactions on Mobile Computing 13.2 (2012): 407-421.
［29］Gu, Zhaoquan, Qiang-Sheng Hua, and Weiguo Dai. "Fully distributed algorithms for blind rendezvous in cognitive radio networks." Proceedings of the 15th ACM international symposium on Mobile ad hoc networking and computing. 2014.
［30］Chang, Cheng-Shang, et al. "Efficient encoding of user IDs for nearly optimal expected time-to-rendezvous in heterogeneous cognitive radio networks." IEEE/ACM Transactions on Networking 25.6 (2017): 3323-3337.
［31］Li, Guyue, et al. "Deterministic distributed rendezvous algorithms for multi-radio cognitive radio networks." Proceedings of the 17th ACM international conference on Modeling, analysis and simulation of wireless and mobile systems. 2014.
［32］Yu, Lu, et al. "Multiple radios for fast rendezvous in cognitive radio networks." IEEE Transactions on Mobile Computing 14.9 (2014): 1917-1931.
［33］ Yu, Lu, et al. "Adjustable rendezvous in multi-radio cognitive radio networks." 2015 IEEE global communications conference (GLOBECOM). IEEE, 2015.
［34］Chang, Yeh-Cheng, Cheng-Shang Chang, and Jang-Ping Sheu. "An enhanced fast multi-radio rendezvous algorithm in heterogeneous cognitive radio networks." IEEE Transactions on Cognitive Communications and Networking 4.4 (2018): 847-859.
［35］Chang, Cheng-Shang, et al. "ETTR Bounds and Approximation Solutions of Blind Rendezvous Policies in Cognitive Radio Networks with Random Channel States." arXiv preprint arXiv:1906.10424 (2019).
［36］Wang, Jen-Hung, et al. "A Reinforcement Learning Approach for the Multichannel Rendezvous Problem." 2019 IEEE Globecom Workshops (GC Wkshps). IEEE, 2019.
［37］Ross, Sheldon M. "Stochastic Processes. John Wiley & Sons." New York (1996).
［38］Chang, Cheng-Shang, XiuLi Chao, and Michael Pinedo. "Integration of discrete-time correlated Markov processes in a TDM system." Probability in the Engineering and Informational Sciences 4.1 (1990): 29-56.
［39］Marshall, Albert W., Ingram Olkin, and Barry C. Arnold. Inequalities: theory of majorization and its applications. Vol. 143. New York: Academic press, 1979.
［40］Tchen, André H. "Inequalities for distributions with given marginals." The Annals of Probability 8.4 (1980): 814-827.
［41］Rolski, Tomasz. "Upper bounds for single server queues with doubly stochastic Poisson arrivals." Mathematics of Operations Research 11.3 (1986): 442-450.
［42］Auer, Peter, et al. "The nonstochastic multiarmed bandit problem." SIAM journal on computing 32.1 (2002): 48-77.