本研究藉由不同感官通道刺激交錯形成的多重感官序列，探討個體是否能 進行跨感官通道與相同感覺通道內刺激的整合連結。在本研究中，我們進行兩 個行為實驗與一個相關事件腦電位研究(event-related potentials, ERPs)釐清並 更進一步討論序列中的刺激如何進行跨感官通道與感官通道內整合的運作機制。 實驗一中沿用 Kemény 與 Lukács (2019) 的實驗程序，並且增設一個排除動作反 應為指標的內隱測驗，以作為檢測參與者是否於不同感覺管道刺激交錯呈現的 序列中有非動作學習的表徵被形成。我們與先前研究結果同樣觀察到參與者展 現多重感官通道序列學習的效果，且刺激會受到跨感官通道刺激的隨機序列影 響反應時間。此外，與先前研究不同的是在單一通道刺激單獨呈現時，視、聽 覺單一通道序列均會展現學習效果，但依然聽覺序列學習效果大於視覺序列。 在增設的內隱測驗結果發現在沒有指示參與者進行針對刺激做序列按鍵動作下， 參與者對於接收到的序列片段無法回應高於猜測機率的正確判斷率，因此推論 序列學習主要以動作學習為主。
實驗二透過詞彙判斷作業檢驗視、聽覺刺激在交錯出現的情況下的促發效 果持續時間，並且與序列學習效果的結果進行相關分析。參與者在視、聽覺字 詞中都可以觀察到促發效果持續至間隔三個刺激出現後，並且此結果與參與者 先前在單一通道階段中，展現視、聽覺序列都能在獨立呈現時展現序列學習效 果的結果一致。實驗三透過 ERP 討論在內隱測驗中，出現序列片段與先前學習 的序列內容不一致時，是否有 N400 效果出現以檢驗參與者學習到的感覺序列是 否有進行跨感官通道的整合。結果發現只有感官通道與刺激本身均不一致時， 才能發現較大的 N400 效果，並且只要有感官通道的變化(無論刺激本身一致與 否)，結果均能發現在刺激出現後期有 400~800 毫秒的負向腦電位差異。由上述 結果得知，個體在感官訊息交錯出現時，能夠將刺激進行跨感官通道的整合， 且同時也會有感官通道內非連續刺激表徵的連結。

This study used multimodal sequences to explore whether individuals can integrate and connect stimuli across and within modality. In the current study, we used two behavioral experiments and one event-related potentials (ERPs) study to discuss and further explore how the stimuli were integrated across modalities and within modality. In Experiment 1, the procedure of Kemény and Lukács (2019) was followed, and an implicit test that excluded serial responses was added to examine whether the participants have a non-motor learning representation in the multimodal sequence. Same as the previous study, we observed the multimodal sequential learning effect and the reaction times for stimuli were affected by the cross-modal random sequence. In addition, we found different results from previous study that was both auditory and visual unimodal sequence revealed sequential learning effect, but the unimodal learning effect in auditory stimuli still larger than in visual stimuli. The results in the additional implicit test which was without the instructions of serial key pressings to the stimuli found that participants cannot perform a higher correct response rate than the pure- guessing probability. Therefore, we inferred that the sequential learning is mainly based on motor learning.
Experiment 2 used lexical decision task to examine the duration of the priming effect for visual and auditory words, and conduct correlation analysis with the results of sequential learning effects. The results indicated that the priming effect for the visual and auditory words lasted until the interval of three stimuli, and this result was consistent to the unimodal sequential learning effect in both auditory and visual modalities. Experiment 3 contained the ERP to discuss whether there is an effect of N400 when the sequence fragments in the implicit test was incongruent with the previously learned multimodal sequence structure, and examining whether the stimuli
ii
were integrated across modalities for participants. It was found that only when the modality and the identity of stimulus were incongruent, the greater N400 effect can be found. Further, the ERP results indicated that a negative going waveform was found during the late time window of 400 to 800 milliseconds, when the modality was incongruent (regardless of whether the identity of stimulus is congruent or not). Given the results above, individuals might integrate the stimulus across modalities when the sensory information is intermixed, and at the same time, there will also be a connection between the non-adjacent stimuli within modality.

劉怡君(2008)。以語意促發作業探討項目指示遺忘中線索對於記憶登錄歷程影 響之行為及事件相關腦電位研究 [未出版之碩士論文]。國立中央大學認知 與神經科學研究所，桃園縣。 取自https://hdl.handle.net/11296/427482
Abla, D., Katahira, K., & Okanoya, K. (2008). On-line assessment of statistical learning by event-related potentials. Journal of Cognitive Neuroscience, 20(6), 952–964. https://doi.org/10.1162/jocn.2008.20058
Abla, D., & Okanoya, K. (2009). Visual statistical learning of shape sequences: An ERP study. Neuroscience Research, 64(2), 185–190. https://doi.org/10.1016/j.neures.2009.02.013
Baddeley, A., & Hitch, G. (1974). The social design of virtual worlds: constructing the user and community through code. Medical Research Council, 47–88.
Ball, F., Fuehrmann, F., Stratil, F., & Noesselt, T. (2018). Phasic and sustained interactions of multisensory interplay and temporal expectation. Scientific Reports, 8(1), 1–12. https://doi.org/10.1038/s41598-018-28495-7
Coderre, E. L., O’Donnell, E., O’Rourke, E., & Cohn, N. (2020). Predictability modulates neurocognitive semantic processing of non-verbal narratives. Scientific Reports, 10(1), 1–11. https://doi.org/10.1038/s41598-020-66814-z
Cohen, A., Ivry, R. I., & Keele, S. W. (1990). Attention and Structure in Sequence Learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 16(1), 17–30. https://doi.org/10.1037/0278-7393.16.1.17
Conway, C. M., & Christiansen, M. H. (2005). Modality-constrained statistical
learning of tactile, visual, and auditory sequences. Journal of Experimental
Psychology: Learning Memory and Cognition, 31(1), 24–39. 87

https://doi.org/10.1037/0278-7393.31.1.24
Deroost, N., & Soetens, E. (2006). Perceptual or motor learning in SRT tasks with complex sequence structures. Psychological Research, 70(2), 88–102. https://doi.org/10.1007/s00426-004-0196-3
Fiser, J., & Aslin, R. N. (2002). Statistical Learning of Higher-Order Temporal Structure from Visual Shape Sequences. Journal of Experimental Psychology: Learning Memory and Cognition, 28(3), 458–467. https://doi.org/10.1037/0278- 7393.28.3.458
Fletcher, P. C., Zafiris, O., Frith, C. D., Honey, R. A. E., Corlett, P. R., Zilles, K., & Fink, G. R. (2005). On the benefits of not trying: Brain activity and connectivity reflecting the interactions of explicit and implicit sequence learning. Cerebral Cortex, 15(7), 1002–1015. https://doi.org/10.1093/cercor/bhh201
Frost, R., Armstrong, B. C., Siegelman, N., & Christiansen, M. H. (2015). Domain generality versus modality specificity: The paradox of statistical learning. Trends in Cognitive Sciences, 19(3), 117–125. https://doi.org/10.1016/j.tics.2014.12.010
Giard, M. H., & Peronnet, F. (1999). Auditory-visual integration during multimodal object recognition in humans: A behavioral and electrophysiological study. Journal of Cognitive Neuroscience, 11(5), 473–490. https://doi.org/10.1162/089892999563544
Giorgio, J., Karlaftis, V. M., Wang, R., Shen, Y., Tino, P., Welchman, A., & Kourtzi, Z. (2018). Functional brain networks for learning predictive statistics. Cortex, 107, 204–219. https://doi.org/10.1016/j.cortex.2017.08.014
Gladwin, T. E., ’t Hart, B. M., & de Jong, R. (2008). Dissociations between motor- related EEG measures in a cued movement sequence task. Cortex, 44(5), 521–
88

536. https://doi.org/10.1016/j.cortex.2007.10.005
Golby, A., Silverberg, G., Race, E., Gabrieli, S., O’Shea, J., Knierim, K., Stebbins, G., & Gabrieli, J. (2005). Memory encoding in Alzheimer’s disease: An fMRI study of explicit and implicit memory. Brain, 128(4), 773–787. https://doi.org/10.1093/brain/awh400
Holcomb, P. J. (1993). Semantic priming and stimulus degradation: Implications for the role of the N400 in language processing. Psychophysiology, 30(1), 47–61. https://doi.org/10.1111/j.1469-8986.1993.tb03204.x
Holcomb, P. J., & Neville, H. J. (1990). Auditory and Visual Semantic Priming in Lexical Decision: A Comparison Using Event-related Brain Potentials. Language and Cognitive Processes, 5(4), 281–312. https://doi.org/10.1080/01690969008407065
Hu, W., Lu, Y., Wang, L., & Zhang, J. X. (2013). Sequence representation during response preparation in the serial reaction time task. NeuroReport, 24(10), 544– 549. https://doi.org/10.1097/WNR.0b013e3283621329
Keele, S. W., Mayr, U., Ivry, R., Hazeltine, E., & Heuer, H. (2003). The Cognitive and Neural Architecture of Sequence Representation. Psychological Review, 110(2), 316–339. https://doi.org/10.1037/0033-295X.110.2.316
Kemény, F., & Lukács, Á. (2019). Sequence in a sequence: Learning of auditory but not visual patterns within a multimodal sequence. Acta Psychologica, 199(June), 102905. https://doi.org/10.1016/j.actpsy.2019.102905
Kemény, F., & Meier, B. (2016). Multimodal sequence learning. Acta Psychologica, 164, 27–33. https://doi.org/10.1016/j.actpsy.2015.10.009
Kemény, F., & Németh, K. (2017). Stimulus dependence and cross-modal
89

interference in sequence learning. Quarterly Journal of Experimental Psychology, 70(12), 2535–2547. https://doi.org/10.1080/17470218.2016.1246579
Kóbor, A., Takács, Á., Kardos, Z., Janacsek, K., Horváth, K., Csépe, V., & Nemeth, D. (2018). ERPs differentiate the sensitivity to statistical probabilities and the learning of sequential structures during procedural learning. Biological Psychology, 135, 180–193. https://doi.org/10.1016/j.biopsycho.2018.04.001
Kutas, M., & Federmeier, K. D. (2011). Thirty years and counting: Finding meaning in the N400 component of the event-related brain potential (ERP). Annual Review of Psychology, 62, 621–647. https://doi.org/10.1146/annurev.psych.093008.131123
Li, S., Chen, S., Zhang, H., Zhao, Q., Zhou, Z., Huang, F., Sui, D., Wang, F., & Hong, J. (2020). Dynamic cognitive processes of text-picture integration revealed by event-related potentials. Brain Research, 1726(October), 146513. https://doi.org/10.1016/j.brainres.2019.146513
Li, X., Zhao, X., Shi, W., Lu, Y., & Conway, C. M. (2018). Lack of cross-modal effects in dual-modality implicit statistical learning. Frontiers in Psychology, 9(FEB), 1–10. https://doi.org/10.3389/fpsyg.2018.00146
Martens, S., Kandula, M., & Duncan, J. (2010). Restricted attentional capacity within but not between sensory modalities: An individual differences approach. PLoS ONE, 5(12), 1–6. https://doi.org/10.1371/journal.pone.0015280
Martini, M., Sachse, P., Furtner, M. R., & Gaschler, R. (2015). Why should working memory be related to incidentally learned sequence structures? Cortex, 64(0), 407–410. https://doi.org/10.1016/j.cortex.2014.05.016
90

Mayr, U. (1996). Spatial attention and implicit sequence learning: Evidence for independent learning of spatial and nonspatial sequences. Journal of Experimental Psychology: Learning Memory and Cognition, 22(2), 350–364. https://doi.org/10.1037/0278-7393.22.2.350
McKone, E. (1995). Short-Term Implicit Memory for Words and Nonwords. Journal of Experimental Psychology: Learning, Memory, and Cognition, 21(5), 1108– 1126. https://doi.org/10.1037/0278-7393.21.5.1108
Mckone, E., & Dennis, C. (2000). Short-term implicit memory : Visual , auditory , and cross-modality priming. 7(2), 341–346.
McPherson, W. B., & Holcomb, P. J. (1999). An electrophysiological investigation of semantic priming with pictures of real objects. Psychophysiology, 36(1), 53–65. https://doi.org/10.1017/S0048577299971196
Mitchel, A. D., & Weiss, D. J. (2011). Learning Across Senses: Cross-Modal Effects in Multisensory Statistical Learning. Journal of Experimental Psychology: Learning Memory and Cognition, 37(5), 1081–1091. https://doi.org/10.1037/a0023700
Molinaro, N., Barber, H. A., Caffarra, S., & Carreiras, M. (2015). On the left anterior negativity (LAN): The case of morphosyntactic agreement: A Reply to Tanner etal. Cortex, 66, 156–159. https://doi.org/10.1016/j.cortex.2014.06.009
Nissen, M. J., & Bullemer, P. (1987). Attentional requirements of learning: Evidence from performance measures. Cognitive Psychology, 19(1), 1–32. https://doi.org/10.1016/0010-0285(87)90002-8
Palolahti, M., Leino, S., Jokela, M., Kopra, K., & Paavilainen, P. (2005). Event- related potentials suggest early interaction between syntax and semantics during
91

on-line sentence comprehension. Neuroscience Letters, 384(3), 222–227. https://doi.org/10.1016/j.neulet.2005.04.076
Peterson, D. S., King, L. A., Cohen, R. G., & Horak, F. B. (2016). Cognitive contributions to freezing of gait in parkinson disease: Implications for physical rehabilitation. Physical Therapy, 96(5), 659–670. https://doi.org/10.2522/ptj.20140603
Poncelet, P. E., & Giersch, A. (2015). Tracking visual events in time in the absence of time perception: Implicit processing at the ms level. PLoS ONE, 10(6), 1–24. https://doi.org/10.1371/journal.pone.0127106
Richard, M. V., Clegg, B. A., & Seger, C. A. (2009). Implicit motor sequence learning is not represented purely in response locations. Quarterly Journal of Experimental Psychology, 62(8), 1516–1522. https://doi.org/10.1080/17470210902732130
Rossignol, S., & Jones, G. M. (1976). Audio-spinal influence in man studied by the H-reflex and its possible role on rhythmic movements synchronized to sound. Electroencephalography and Clinical Neurophysiology, 41(1), 83–92. https://doi.org/10.1016/0013-4694(76)90217-0
Silva, S., Folia, V., Inácio, F., Castro, S. L., & Petersson, K. M. (2018). Modality effects in implicit artificial grammar learning: An EEG study. Brain Research, 1687, 50–59. https://doi.org/10.1016/j.brainres.2018.02.020
Tabullo, Á., Sevilla, Y., Segura, E., Zanutto, S., & Wainselboim, A. (2013). An ERP study of structural anomalies in native and semantic free artificial grammar: Evidence for shared processing mechanisms. Brain Research, 1527, 149–160. https://doi.org/10.1016/j.brainres.2013.05.022
92

Teder-Sälejärvi, W. A., Di Russo, F., McDonald, J. J., & Hillyard, S. A. (2005). Effects of spatial congruity on audio-visual multimodal integration. Journal of Cognitive Neuroscience, 17(9), 1396–1409. https://doi.org/10.1162/0898929054985383
Thaut, M. H., Kenyon, G. P., Schauer, M. L., & McIntosh, G. C. (1999). The connection between rhythmicity and brain function. IEEE Engineering in Medicine and Biology Magazine, 18(2), 101–108. https://doi.org/10.1109/51.752991
Urry, K., Burns, N. R., & Baetu, I. (2015). Accuracy-based measures provide a better measure of sequence learning than reaction time-based measures. Frontiers in Psychology, 6(August), 1–14. https://doi.org/10.3389/fpsyg.2015.01158
Vallet, G., Brunel, L., & Versace, R. (2010). The perceptual nature of the cross-modal priming effect: Arguments in favor of a sensory-based conception of memory. Experimental Psychology, 57(5), 376–382. https://doi.org/10.1027/1618- 3169/a000045
Verwey, W. B., & Clegg, B. A. (2005). Effector dependent sequence learning in the serial RT task. Psychological Research, 69(4), 242–251. https://doi.org/10.1007/s00426-004-0181-x
West, W. C., & Holcomb, P. J. (2002). Event-related potentials during discourse-level semantic integration of complex pictures. Cognitive Brain Research, 13(3), 363– 375. https://doi.org/10.1016/S0926-6410(01)00129-X
Willingham, D. B. (1998). A neuropsychological theory of motor skill learning. Psychological Review, 105(3), 558–584. https://doi.org/10.1037//0033- 295x.105.3.558
93

Willingham, D. B. (1999). Implicit motor sequence learning is not purely perceptual. Memory and Cognition, 27(3), 561–572. https://doi.org/10.3758/BF03211549
Willingham, D. B., Nissen, M. J., & Bullemer, P. (1989). On the Development of Procedural Knowledge. Journal of Experimental Psychology: Learning, Memory, and Cognition, 15(6), 1047–1060. https://doi.org/10.1037/0278- 7393.15.6.1047
Ziessler, M. (1998). Response-effect learning as a major component of implicit serial learning. Journal of Experimental Psychology: Learning Memory and Cognition, 24(4), 962–978. https://doi.org/10.1037/0278-7393.24.4.962
Zießler, M. (1994). The impact of motor responses on serial-pattern learning. Psychological Research, 57(1), 30–41. https://doi.org/10.1007/BF00452993
Ziessler, M., & Nattkemper, D. (2001). Learning of Event Sequences is Based on Response-Effect Learning: Further Evidence from a Serial Reaction Task. Journal of Experimental Psychology: Learning Memory and Cognition, 27(3), 595–613. https://doi.org/10.1037/0278-7393.27.3.595