| 研究生: |
楊貴安 Kuei-an Yang |
|---|---|
| 論文名稱: |
Hadoop雲端平台在工程應用之探討研究 Study on the Hadoop Cloud Computing Platform for Engineering Applications |
| 指導教授: |
周建成
Chien-Cheng Chou |
| 口試委員: | |
| 學位類別: |
碩士 Master |
| 系所名稱: |
工學院 - 土木工程學系 Department of Civil Engineering |
| 畢業學年度: | 100 |
| 語文別: | 中文 |
| 論文頁數: | 185 |
| 中文關鍵詞: | 雲端運算 、Hadoop 、分散式系統 、虛擬叢集 |
| 外文關鍵詞: | Distributed Systems, Hadoop, Cloud Computing |
| 相關次數: | 點閱:18 下載:0 |
| 分享至: |
| 查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報 |
雲端運算是一種新的網路概念,是藉由網路串聯不同電腦之間的相互合作,藉由網路的概念來產生相應的技術,本質來自於分散式運算與網格運算。分散式運算是將大型工作分成很多小型工作,再分別分配給眾多的電腦運算,最後再匯集所有的結果來完成單機無法完成的工作。網格運算則是分散式運算的一種延伸,主要特色是將不同平台、不同等級、不同架構的電腦藉由分散式運算來做整合,所以雲端運算與網格運算都是一種分散式運算的延伸。網格運算是強調整合眾多不同的平台,而雲端運算是強調在本機有限的資源利用網路來取得運算資源。因此,建置分散式運算的雲端平台研究有其必要性。本研究建置雲端分散式檔案系統Hadoop,Hadoop Distributed File System(HDFS),使用四台實體電腦來架設四台虛擬叢集環境與八台虛擬叢集環境。四台虛擬叢集電腦架設方式是在每台實體電腦各虛擬一台電腦出來,共四台虛擬叢集環境,而八台虛擬叢集電腦架設方式是在每台實體電腦各虛擬兩台電腦出來,共八台虛擬叢集環境。經本研究結果實現,一台實體電腦可以虛擬兩台以上電腦,符合雲端虛擬化上百台或上千台的叢集環境。其次,雲端分散式系統是來處理大量的運算,本研究藉由矩陣大量的運算來測試Hadoop分散式檔案系統。而矩陣運算在工程應用是常見且重要的,不過目前矩陣運算都是以MPI(Message Passing Interface)來實現,並無在雲端平台上來實現,因此本研究藉由雲端平台來實施矩陣運算。
Cloud computing is a new concept of networking. Cloud computing is a co-operation of the network which allows several computers to work together. The nature of cloud computing are from distributed computing and grid computing. Distributed computing is a large work divided into several small parts. Then it will be handed over to several computers to do the computing process. Finally, to bring all the results together to complete the stand-alone computing could not be done. Grid computing is an extension of the distributed computing. The main features of different platforms, different levels of the different computer structure integrated by distributed computing. Cloud computing and grid computing are extensions of a distributed computing. Grid computing is the emphasis on the integration of many different platforms. While cloud computing is the emphasis on limited resources in the machine, which use internet to obtain the computing resources. Therefore, build distributed computing cloud platform is necessary. The objective of this study is to build a cloud distributed systems Hadoop, Hadoop Distributed File System (HDFS).We use four physical computers to host four virtual cluster environments with eight virtual cluster environments. The four cluster computers set up in each physical computer, with one virtual computer exist for each physical computer, so there are four virtual cluster environments. Then each of the four physical computers is built two virtual computers inside, so there are eight virtual cluster environments. It proves that a physical computer can has two or more virtual computers. Comply with the cloud virtualization of hundreds or thousands of cluster environments. Moreover, cloud distributed systems can deal with a lot of computing that we used in computing matrix. Matrix computing is important of engineering application. But usually implement MPI(Message Passing Interface). So this paper implements matrix computing of cloud platform.
Agarwal, C., R., Bale, M., S., Gustavson, G., F., Joshi, M., and Palkar, P. (1995) “A threedimensional approach to parallel matrix multiplication.” IBM Journal of Research and Development, 39(5), 575-582.
Armbrust, M., Fox, A., Griffith, R., Joseph, A. D., Katz, R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A., Stoica, I., and Zaharia, M. (2010) “A View of Cloud Computing.” Communications of the ACM, 53(4), 50-58.
Apache (2012). HDFS Architecture Guide. Retrieved February 17, 2012, from http://hadoop.apache.org/common/docs/current/hdfs_design.html
Chang, F., Dean, J., Ghemawat, S., Hsieh, C., W., Wallach, A., D., Burrows, M., Chandra, T., Fikes, A., and Gruber, E., R. (2008) “Bigtable: A Distributed Storage System for Structured Data.” ACM Transactions on Computer Systems, 26(2).
Cusumano, M. (2010) “Cloud Computing and SaaS as New Computing Platforms.” Communications of the ACM, 53(4), 27-29.
Dean, J., and Ghemawat, S. (2008) “MapReduce: Simplied Data Processing on Large Clusters.” Communications of the ACM, 51(1), 107-113.
Ghemawat, S., Gobioff, H., and Leung, T., S. (2003) “The Google File System.” Proceedings of the nineteenth ACM symposium on Operating systems principles, 29-43.
Krumins, P. (2009). MIT''s Introduction to Algorithms, Lectures 20 and 21: Parallel, Retrieved February 18, 2012, April 27, 2012 from http://www.catonmat.net/blog/mit-introduction-to-algorithms-part-thirteen/
Lars, G. (2011) “HBase: The Definitive Guide.” Oreilly and Associates Inc.
Pease, C., M. (1967) “Matrix Inversion Using Parallel Processing.” Journal of the Association for Computing Machinery, 14(4), 757-764.
Ricky, H. (2008). Hadoop Map/Reduce Implementation. Retrieved February 2, 2012, from http://horicky.blogspot.com/2008/11/ hadoop-mapreduce-implementation.html
Shvachko, K., Kuang, H., Radia, S., and Chansler, R. (2010) “The Hadoop Distributed File System.” Mass Storage Systems and Technologies, 1-10.
Seo, S., Yoon, J. E., Kim, J., Jin, S., Kim, S. J., Maeng, S. (2010) “HAMA: An Efficient Matrix Computation with the MapReduce Framework. ” International Conference on Cloud Computing Technology and Science, 721-726.
Tom, W. and Doug, C. (2010) “Hadoop: The Definitive Guide.” Oreilly and Associates Inc.
Wikipedia (2011). Apache Hadoop. Retrieved February 2, 2012, from http://en.wikipedia.org/wiki/Apache_Hadoop
Wikipedia (2010). Windows Azure. Retrieved February 2, 2012, from http://zh.wikipedia.org/wiki/Windows_Azure
Wikipedia (2012). MapReduce. Retrieved February 18, 2012, from http://en.wikipedia.org/wiki/MapReduce
Zhang, C., and Sterck, D. H. (2010) “Supporting Multi-row Distributed Transactions with Global Snapshot Isolation Using Bare-bones HBase” In Proceedings of the 11th ACM/IEEE International Conference on Grid Computing (Grid), 177-184.
王鵬(2010)。雲端運算的關鍵技術與應用實例。佳魁資訊。
國際商業機器股份有限公司(2010)。精選雲端解決方案。西元2012年2月3日,取自:http://www-03.ibm.com/systems/tw/promotion/2010_q1_hot/cloud.html
黃淑玲(2010)。雲端辦公室Google Office簡介。國立台灣大學計算機及資訊網路中心電子報第15期。民國101年2月2日,取自:http://www.cc.ntu.edu.tw/chinese/epaper/0015/20101220_1506.htm