為了滿足第六代（6G）行動通訊系統的高流量需求，已經為未
來的 6G 無線網路提出了各種新興技術。本論文提出了一種全雙工
（FD）通訊架構，透過部署無人機（UAV）並輔以多功能可重構智慧
表面（MF-RIS）和流體天線（FA）來支援上行（UL）和下行（DL）
用戶，以最大限度地提高系統能源效率（EE）。MF-RIS 具有同時反
射、折射、放大和收集能量的能力。系統模型包括無人機到基地台、
無人機到使用者的雙向通道，以及 MF-RIS 放大係數、相位矩陣等控
制變數。為了實現 EE 優化，我們設計了一種具有自註意力機制的混
合深度強化學習 (DRL) 和超參數優化演算法 (HIPPO)，該演算法將
深度 Q 網路 (DQN) 與近端策略優化 (PPO) 相結合，以優化無人機的
位置、發射波束成形、FA 位置和 MF-RIS 參數。模擬結果表明，與
傳統 RIS、固定天線陣列、半雙工和 FD 或其他現有的 DRL 方法相
比，所提出的 HIPPO 方案在不同場景下均實現了最高的 EE 性能。

Tomeetthehightrafficdemandinsixthgeneration(6G)mobilecommunica-
tion systems,variousemergingtechnologieshavebeenproposedforfuture6G
wireless networks.Thisthesisproposesafull-duplex(FD)communication
architectureinsupportofuplink(UL)anddownlink(DL)usersbydeploying
unmanned aerialvehicles(UAVs)complementedbyamulti-functionalrecon-
figurable smartsurface(MF-RIS)andfluidantennas(FA)tomaximizethe
system energyefficiency(EE).MF-RIShastheabilitytoreflect,refract,am-
plify andharvestenergysimultaneously.Thesystemmodelincludesbidirec-
tional channelsfromdronetobasestationandfromdronetouser,aswellas
MF-RIS amplificationfactor,phasematrix,andothercontrolvariables.To
achievetheEEoptimization,wedesignahybriddeepreinforcementlearning
(DRL) withself-attentionmechanismandhyperparameteroptimizationalgo-
rithm (HIPPO)thatcombinesdeepQ-network(DQN)withproximalpolicy
optimization (PPO)tooptimizeUAVs’positions,transmitbeamforming,FA
positionsandMF-RISparameters.Simulationresultsshowthatcompared
to traditionalRIS,fixedantennaarrays,half-duplexandFDorotherex-
isting DRLmethods,theproposedHIPPOschemeaccomplishesthehighest
EE performanceunderdifferentscenarios.

Bibliography
[1] C.-J. Ku,L.-H.Shen,andK.-T.Feng,“Reconfigurableintelligentsurfaceassisted
interferencemitigationfor6Gfull-duplexMIMOcommunicationsystems,”in Proc.
IEEE AnnualInternationalSymposiumonPersonal,IndoorandMobileRadioCom-
munications(PIMRC), September2022,pp.327–332.
[2] Y. Zeng,R.Zhang,andT.J.Lim,“Wirelesscommunicationswithunmannedaerial
vehicles:Opportunitiesandchallenges,”vol.54,no.5,pp.36–42,May2016.
[3] R. I.Bor-Yaliniz,A.El-Keyi,andH.Yanikomeroglu,“Efficient3-Dplacementofan
aerial basestationinnextgenerationcellularnetworks,”in Proc.IEEEInternational
ConferenceonCommunications(ICC), May2016.
[4] M. Mozaffari,W.Saad,M.Bennis et al., “Efficientdeploymentofmultipleunmanned
aerial vehiclesforoptimalwirelesscoverage,”vol.20,no.8,pp.1647–1650,August
2016.
[5] A. Al-Hourani,S.Kandeepan,andS.Lardner,“OptimalLAPaltitudeformaximum
coverage,”vol.3,no.6,pp.569–572,July2014.
[6] A. MerwadayandI.Guvenc,“UAVassistedheterogeneousnetworksforpublicsafety
communications,”in Proc.IEEEWirelessCommunicationsandNetworkingConfer-
enceWorkshops(WCNCW), March2015,pp.329–334.
[7] A. Osseiran,F.Boccardi,V.Braun et al., “Scenariosfor5Gmobileandwireless
communications:thevisionofthemetisproject,”vol.52,no.5,pp.26–35,May
2014.
[8] Y. Liu,S.Wang,L.Ma et al., “Energy-efficientRIS-UAVrelaywithtrajectoryopti-
mization andfaircommunication,”in Proc.IEEEInternationalConferenceonCom-
puter andCommunications(ICCC), December2022,pp.339–343.
55
[9] Y. Yao,K.Lv,S.Huang et al., “3Ddeploymentandenergyefficiencyoptimiza-
tion basedonDRLforRIS-assistedair-to-groundcommunicationsnetworks,” IEEE
TransactionsonVehicularTechnology, vol.73,no.10,pp.14988–15003,October
2024.
[10] Q. WuandR.Zhang,“Towardssmartandreconfigurableenvironment:Intelligent
reflecting surfaceaidedwirelessnetwork,”vol.58,no.1,pp.106–112,January2020.
[11] C. Huang,A.Zappone,G.C.Alexandropoulos et al., “Reconfigurableintelligent
surfaces forenergyefficiencyinwirelesscommunication,”vol.18,no.8,pp.4157–
4170, August2019.
[12] W. Wang,W.Ni,H.Tian et al., “Multi-functionalreconfigurableintelligentsurface:
System modelingandperformanceoptimization,”vol.23,no.4,pp.3025–3041,April
2024.
[13] Y.-T. Li,L.-H.Shen,K.-T.Feng et al., “Geneticmulti-agentreinforcementlearning
for multipledouble-sidedSTAR-RISsinfull-duplexmimonetworks,”in Proc.IEEE
International ConferenceonCommunications(ICC), June2024,pp.5003–5008.
[14] D. Zhang,S.Ye,M.Xiao et al., “Fluidantennaarrayenhancedover-the-aircomputa-
tion,” IEEE WirelessCommunicationsLetters, vol.13,no.6,pp.1541–1545,March
2024.
[15] K.-K. Wong,W.K.New,X.Hao et al., “Fluidantennasystem—part i:Preliminar-
ies,” IEEE CommunicationsLetters, vol.27,no.8,pp.1919–1923,August2023.
[16] L. Zhu,W.Ma,andR.Zhang,“Movableantennasforwirelesscommunication:Op-
portunitiesandchallenges,” IEEE CommunicationsMagazine, vol.62,no.6,pp.
114–120, June2024.
[17] K.-K. Wong,A.Shojaeifard,K.-F.Tong et al., “Fluidantennasystems,” IEEE Trans-
actions onWirelessCommunications, vol.20,no.3,pp.1950–1962,March2021.
[18] X. Liu,Y.Liu,andY.Chen,“Machinelearningempoweredtrajectoryandpassive
beamformingdesigninUAV-RISwirelessnetworks,” IEEE JournalonSelectedAreas
in Communications, vol.39,no.7,pp.2042–2055,July2021.
[19] W. Xue,P.Kolaric,J.Fan et al., “Inversereinforcementlearningintrackingcon-
trol basedoninverseoptimalcontrol,” IEEE TransactionsonNeuralNetworksand
LearningSystems, vol.52,no.10,pp.10570–10581,October2022.
56
[20] K. Doya,“Reinforcementlearningincontinuoustimeandspace,” NeuralComputa-
tion, vol.12,no.1,pp.219–245,January2000.
[21] H. ModaresandF.L.Lewis,“Optimaltrackingcontrolofnonlinearpartially-
unknownconstrained-inputsystemsusingintegralreinforcementlearning,” Automat-
ica, vol.50,no.7,pp.1780–1792,July2014.
[22] J. Zhang,Z.Wang,andH.Zhang,“Data-basedoptimalcontrolofmultiagentsys-
tems: Areinforcementlearningdesignapproach,” IEEE TransactionsonCybernetics,
vol.49,no.12,pp.4441–4449,December2019.
[23] B. Kiumarsi,F.L.Lewis,H.Modares et al., “ReinforcementQ-learningforoptimal
trackingcontroloflineardiscrete-timesystemswithunknowndynamics,” Automatica,
vol.50,no.4,pp.1167–1175,April2014.
[24] Y. Jiang,J.Fan,T.Chai et al., “Trackingcontrolforlineardiscrete-timenetworked
controlsystemswithunknowndynamicsanddropout,” IEEE TransactionsonNeural
Networks andLearningSystems, vol.29,no.10,pp.4607–4620,October2018.
[25] H. Mei,K.Yang,Q.Liu et al., “3D-trajectoryandphaseshiftdesignforRIS-assisted
UAVsystemsusingdeepreinforcementlearning,” IEEE TransactionsonVehicular
Technology, vol.71,no.3,pp.3020–3029,March2022.
[26] P.-C.Wu,L.-H.Shen,K.-T.Feng et al., “Federatedreinforcementlearningformulti-
dual-STAR-RISassistedDFRC-enabledmulti-BSinISACsystems,”in Proc.IEEE
International ConferenceonCommunications(ICC), June2024,pp.2986–2991.
[27] X. Li,L.Dong,L.Xue,andC.Sun,“Hybridreinforcementlearningforoptimal
controlofnon-linearswitchingsystem,” IEEE TransactionsonNeuralNetworksand
LearningSystems, vol.34,no.11,pp.9161–9170,November2023.
[28] R. Zhong,Y.Liu,X.Mu et al., “HybridreinforcementlearningforSTAR-RISs:a
coupled phase-shiftmodelbasedbeamformer,” IEEE JournalonSelectedAreasin
Communications, vol.40,no.9,pp.2556–2569,September2022.
[29] L.-H. ShenandY.-H.Chiu,“RIS-aidedfluidantennaarray-mountedUAVnetworks,”
IEEE WirelessCommunicationsLetters, vol.14,no.4,pp.1049–1053,January2025.
[30] A. Faisal,I.Al-Nahhal,O.A.Dobre et al., “Deepreinforcementlearningforoptimiz-
ing RIS-assistedHD-FDwirelesssystems,” IEEE CommunicationsLetters, vol.25,
no. 12,pp.3893–3897,December2021.
57
[31] Y. Cui,T.Lv,andY.Cao,“DRL-basedresourcemanagementinRIS-assisteduplink
cell-free network,”in Proc.IEEEGlobecomWorkshops(GCWkshps), December2022,
pp. 1495–1500.
[32] J. Chen,Y.Xu,D.Yang et al., “UAV-assistedISCCnetworks:jointresourceand
trajectoryoptimization,” IEEE WirelessCommunicationsLetters, vol.13,no.9,pp.
2372–2376, September2024.
[33] Y. Wan,Z.Zhao,J.Tang et al., “Multi-UAVformationobstaclesavoidancepath
planning basedonPPOalgorithm,”in Proc.InternationalConferenceonBigData
and InformationAnalytics(BigDIA), December2023,pp.55–62.
[34] K. Chikhaoui,H.Ghazzai,andY.Massoud,“PPO-basedreinforcementlearningfor
UAVnavigationinurbanenvironments,”in Proc.IEEEInternationalMidwestSym-
posiumonCircuitsandSystems(MWSCAS), August2022.
[35] A. Al-Hourani,S.Kandeepan,andS.Lardner,“Optimallapaltitudeformaximum
coverage,” IEEE WirelessCommunicationsLetters, vol.3,no.6,pp.569–572,De-
cember2014.
[36] Z. Li,J.Zhang,J.Zhu et al., “Risenergyefficiencyoptimizationwithpracticalpower
models,”in Proc.InternationalWirelessCommunicationsandMobileComputing
(IWCMC), June2023,pp.1172–1177.
[37] K. Ntontin,A.A.Boulogeorgos,E.Björnson et al., “Wirelessenergyharvesting
for autonomousreconfigurableintelligentsurfaces,” IEEE TransactionsonGreen
CommunicationsandNetworking, vol.7,no.1,pp.2473–2400,March2023.
[38] K. Ntontin,A.-A.A.Boulogeorgos,E.Björnson et al., “Millimeterwavevs.THz
energy harvestingforautonomousreconfigurableintelligentsurfaces,”in Proc.IEEE
International ConferenceonCommunicationsWorkshops(ICCWorkshops), May
2022, pp.1213–1218.
[39] H. T.Le,T.V.Nguyen,H.T.T.Pham et al., “Harvestedenergyevaluationoffree-
space opticsris-assistedground-hap-uavsystemovercompositechannels,”in Proc.
IEEE VehicularTechnologyConference(VTC2024-Spring), June2024.
[40] Q. WuandR.Zhang,“Intelligentreflectingsurfaceenhancedwirelessnetworkvia
jointactiveandpassivebeamforming,” IEEE TransactionsonWirelessCommunica-
tions, vol.18,no.11,pp.5394–5409,November2019.
58
[41] A. Vaswani,N.Shazeer,N.Parmar et al., “Attentionisallyouneed,”in Proc.Ad-
vancesinNeuralInformationProcessingSystems, I.Guyon,U.V.Luxburg,S.Bengio
et al., Eds.,vol.30,2017,pp.1–11.
[42] C. Meng,K.Xiong,W.Chena et al., “Sum-ratemaximizationinSTAR-RIS-assisted
RSMA networks:APPO-basedalgorithm,” IEEE InternetofThingsJourna, vol.11,
no. 4,pp.5667–5680,February2024.
[43] T. M.Hoang,L.T.Dung,,B.C.Nguyen et al., “Outageprobabilityandthroughput
of mobilemultiantennaUAV-assistedFD-NOMArelaysystemwithimperfectCSI,”
IEEE SystemsJournal, vol.17,no.1,pp.1477–1488,June2023.
[44] P.Raut,K.Singh,C.-P.Li et al., “NonlinearEH-basedUAV-assistedFDIoTnet-
works:Infiniteandfiniteblocklengthanalysis,” IEEE InternetofThingsJournal,
vol.8,no.24,pp.17655–17668,December2021.
[45] 3GPP,“Studyonchannelmodelforfrequenciesfrom0.5to100GHz(v16.1.0,release
16),” 3GPP, SophiaAntipolis,France,Tech.Rep.TR38.901, December2019.
[46] H. Yan,Y.Chen,andS.-H.Yang,“UAV-enabledwirelesspowertransferwithbase
station chargingandUAVpowerconsumption,” IEEE TransactionsonVehicular
Technology, vol.69,no.11,pp.12883–12896,November2020.
[47] L. Jiao,K.Yu,J.Chen et al., “Performanceanalysisofuplink/downlinkdecoupled
access incellular-V2Xnetworks,” IEEE TransactionsonMobileComputing, vol.23,
no. 5,pp.5516–5630,May2024.
[48] M. Li,H.Li,P.Ma,andH.Wang,“EnergymaximizationforgroundnodesinUAV-
enabled wirelesspowertransfersystems,” IEEE InternetofThingsJournal, vol.10,
no. 19,pp.17096–17109,October2023.