| 研究生: |
蔡昀翰 Yun-Hang Tsai |
|---|---|
| 論文名稱: |
基於Hadoop平台之分散式權重式字尾樹暨天文時序性資料分析系統 Distributed Astronomy Sequential Pattern Analysis System Using Hadoop Platform with Weighted Suffix Tree |
| 指導教授: |
蔡孟峰
Meng-Feng Tsai |
| 口試委員: | |
| 學位類別: |
碩士 Master |
| 系所名稱: |
資訊電機學院 - 資訊工程學系 Department of Computer Science & Information Engineering |
| 論文出版年: | 2015 |
| 畢業學年度: | 103 |
| 語文別: | 中文 |
| 論文頁數: | 56 |
| 中文關鍵詞: | 泛星計畫 、分散式系統 、資料探勘 、權重字尾樹 |
| 外文關鍵詞: | Pan-Starrs, Distributed System, Data Mining, Weighted Suffix Tree |
| 相關次數: | 點閱:13 下載:0 |
| 分享至: |
| 查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報 |
隨著科技的發展,泛星計畫(Panoramic Survey Telescope And Rapid Response System,Pan-STARRS)中所觀測到的資料量也隨之增長,而儲存設備成本降低,也讓天文學家們得以將大量且詳細的觀測資料儲存起來。
由於收集到的天文資料其各個元素間是具有時間順序性的,而傳統的方法卻難以處理此類資料,所以我們選用字尾樹作為其結構的原型,提供天文學家快速而有效率的星體資料查詢功能,並且能夠在分析後提供與查詢相似的星體資訊給天文學家們。
因為字尾樹的資料結構其記憶體使用量驚人,而天文資料的數量又十分龐大,在兩項因素交互影響之下,導致單一機器無法負荷,所以我們選用在開源的OpenStack系統上,建構Hadoop平台的雲端系統來構成分散式環境,將資料分散處理,以提升系統的整體效能。
透過分散式系統處理大量的天文資料,減少了在資料處理上所耗費的人力,在效率上也得到了明顯的提升,提供了研究人員在未來面對大量觀測資料時一個有效的解決方法。在未來我們也期望能利用此系統架構來為所有具有時序性的資料作分析。
Because of the ongoing construction of observatories from Pan-Starrs projects with technological advancements, the size of observation data has exploded. And the storage device cost reduction. Astronomical researchers were able to make a large and detailed observation data stored.
The various elements of collected astronomical data have time sequential features. And the traditional method is difficult to handle such data. So we use the suffix tree as a prototype of system structure to provide astronomical researchers a fast and efficient data query system. And we can provide approximate patterns to astronomical researchers after finish the analysis.
Because the interaction of the amazing memory consumed of suffix tree data structure and the very large number of astronomical data lead to a single machine overload, we use the open source OpenStack system to construct Hadoop platform cloud system to complete a distributed environment. So that we can process astronomical data distributed, and enhance the effectiveness of the system.
To Process large amounts of astronomical data through distributed systems can reduce the cost of manually data processing and the efficiency has been significantly improved. We provided a valid solution when astronomical researchers face a lot of observation data in the future. We hope to use this system architecture to analyze all the time sequential data in the future.
〔1〕陳文屏, 「天文觀測的新挑戰─談泛星計畫」, 科儀新知, 第30卷第3期, 2008年.
〔2〕“General Catalog of Variable Stars,”Institute of Astronomy of Russian Academy of Sciences and Sternberg State Astronomical Institute of the Moscow State University, [Online]. Available: http://www.sai.msu.su/gcvs/gcvs/iii/html.
〔3〕“Pan-STARTS,” Institute for Astronomy, University of Hawaii, 2005. [Online]. Available: http://pan-starrs.ifa.hawaii.edu/public/home.html.
〔4〕Wikipedia,“Variable star,” http://en.wikipedia.org/wiki/Variable_star , 2015.
〔5〕OpenStack, http://www.openstack.org/.
〔6〕Apache Hadoop, http://hadoop.apache.org/.
〔7〕R Project, http://www.r-project.org/.
〔8〕中華R軟體協會, http://www.r-software.org/.
〔9〕P. Weiner, “Linear Pattern Matching Algorithm,” 14th Annual IEEE Symposium on Switching and Automata Theory, 1973.
〔10〕吳彥慶, “Exploiting Frequent Episodes in Weighted Suffix Tree to Improve Intrusion Detection System,” 國立中央大學, 碩士論文, 2007.
〔11〕沈敬軒, “Mining Similar Astronomical Sequence Pattern with Hierarchical Weighted Suffix Tree,” 國立中央大學, 碩士論文, 2011.
〔12〕張哲嘉,“Distributed Suffix Tree Based Sequential Pattern Management System for Astronomical Analysis,” 國立中央大學, 碩士論文, 2013.
〔13〕劉書宏, Distributed Astronomical Sequence Data Indexing System with Suffix Tree, 國立中央大學, 碩士論文, 2014.
〔14〕Tom White, “Hadoop The Definitive Guide 3rd Edition,” O'Reilly, May, 2012.
〔15〕Jiawei Han and Micheline Kamber, “Data Mining Concepts and Techniques Second Edition,” Elsevier Inc., San Francisco, 2006..
〔16〕Yishan Li and Sathiamoorthy Manoharan, “A performance comparison of SQL and NoSQL databases,” University of Auckland, New Zealand, 2013.
〔17〕Lei GU and Huan Li, “Memory or Time: Performance Evaluation for Iterative Operation on Hadoop and Spark,” Beihang University, 2013.
〔18〕Essam Mansour, Ahmed El-Roby, Panos Kalnis, Aron Ahmadia and Ashraf Aboulnaga, ” RACE: A Scalable and Elastic Parallel System for Discovering Repeats in Very Long Sequences,” The 39th International Conference on Very Large Data Bases, 2013.
〔19〕Melita HADZAGIC, Marie-Odette ST-HILAIRE, Sean WEBB, Elisa SHAHBAZIAN, “Maritime Traffic Data Mining Using R,” 16th International Conference on Information Fusion Istanbul, 2013.
〔20〕Rohith Menon, Goutham Bhat and Michael Schatz, “Rapid Parallel Genome Indexing with MapReduce,” State University of New York at Stony Brook, 2011.
〔21〕Prabhat Kumar, Berkin Ozisikyilmaz, Wei-Keng Liao, Gokhan Memik, Alok Choudhary, “High Performance Data Mining Using R on Heterogeneous Platforms,” IEEE International Parallel & Distributed Processing Symposium, 2011.
〔22〕Drew Schmidt, George Ostrouchovy, Wei-Chen Cheny, and Pragneshkumar Patel, “Tight Coupling of R and Distributed Linear Algebra for High-Level Programming with Big Data,” SC Companion: High Performance Computing, Networking Storage and Analysis, 2012
〔23〕Kai Hwang, Geoffrey C. Fox, Jack J. Dongarra, “Distributed and Cloud Computing From Parallel Processing to the Internet of Things,” 2012