人機互動漸漸成為電腦科學中的一個重要議題，它讓我們能更輕易地與電腦或各種設備進行交流。本篇提出了一個基於區域搜索的手部追蹤方法，我們採用了三維點雲 (Point Cloud) 資訊的方法來改善手部追蹤的穩定度，並且介紹了一個基於分群的手指指尖偵測方法，可以在手指追蹤失敗時用來還原手指座標。在手部追蹤完成後，我們利用手指指尖座標與我們的吉他和弦手勢資料庫比對，並且將結果整合到虛擬吉他系統中。在最後，我們設計了一個實驗讓一般的使用者在我們的虛擬吉他系統上演奏一首。而我們也分別在Kinect V1和Kinect V2彈上三首歌來討論兩種感測器的差異。結果說明了我們的方法在虛擬吉他系統上表現良好，而一般使用者也能簡單地學習我們的系統。

Human Computer Interaction (HCI) is becoming a hot issue in computer science which makes people communicate with computers easier. A novel hand tracking method based on region search is presented in this paper. We adopt the concept of 3D point clouds to further improve the stability of hand tracking. Also, the cluster-based finger detection is introduced in this paper, which can be used for restoring finger position if finger tracking failed. After hand tracking finished, we use the position of fingertips to match our hand gesture with our guitar hand gesture database, and integrate proposed method into our virtual guitar system. At last, we setup an experiment which allows general users to play a song with our virtual guitar system. And we also test our system by playing three songs on both Kinect V1 and Kinect V2 sensor and discuss their difference. The result shows that our method performs well on virtual guitar system and general users can learn our system easily.

[1] Wu. A, Shah. M, and Da Vitoria Lobo. N, “A virtual 3D blackboard: 3D finger tracking using a single camera,” Proceedings of fourth IEEE International Conference on Automatic Face and Gesture Recognition, pp. 536-543, Mar 2000.

[2] Hui Liang, Junsong Yuan, and Daniel Thalmann, “3D fingertips and palm tracking in depth image sequences”, Proceedings of the 20th ACM international conference on multimedia, 2012, pp.785-788.

[3] Arun Kulshreshth, Chris Zorn, Joseph J., LaViola Jr., “Real-time Markerless Kinect based Finger Tracking and Hand Gesture Recognition for HCI”, 2013 IEEE Symposium on 3D User Interfaces (3DUI), 2013, pp. 187-188

[4] Yan Wen, Chuanyan Hu, Guanghui Yu, Changbo Wang, “A robust method of detecting hand gestures using depth sensors”, 2012 IEEE International Workshop on Haptic Audio Visual Environments and Games (HAVE), 2012, pp. 72-77

[5] Iason Oikonomidis, Nikolaos Kyriazis, Antonis Argyros, “Efficient Model-based 3D Tracking of Hand Articulations using Kinect”, Proceedings of the British Machine Vision Conference, 2011, pp. 101.1-101.11

[6] Zhou Ren, Junsong Yuan, Zhengyou Zhang, “Robust Hand Gesture Recognition Based on FingerEarth Mover’s Distance with a Commodity Depth Camera”, Proceedings of the 19th ACM international conference on Multimedia, 2011, pp. 1093-1096

[7] Jagdish L. Raheja, Ankit Chaudhary, Kunal Singal, “Tracking of Fingertips and Centers of Palm Using KINECT”, 2011 Third International Conference on Computational Intelligence Modelling and Simulation (CIMSiM), 2011, pp. 248-252

[8] Stergios Poularakis, Ioannis Katsavounidis, “Finger detection and hand posture recognition based on depth information”, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014, pp. 4329-4333

[9] G. Odowichuk, S. Trail, P. Driessen, W. Nie, W. Page, “Sensor fusion: Towards a fully expressive 3D music control interface”, 2011 IEEE Pacific Rim Conference on Communications, Computers and Signal Processing (PacRim), 2011, pp. 836-841

[10] Manuela Renzi, Stavros Vassos, Tiziana Catarci, Stephen Kimani, “Touching Notes: A Gesture-Based Game for Teaching Music to Children”, TEI '15 Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction, 2015, pp. 603-606.

[11] Hamed Sarbolandi, Damien Lefloch, Andreas Kolb, “Kinect Range Sensing: Structured-Light versus Time-of-Flight Kinect”, Computer Vision and Image Understanding Volume 139, 2015, pp. 1-20.

[12] http://www.similarityapp.com/

[13] P. Cano; E. Batle; T. Kalker; J. Haitsma, “A review of algorithms for audio fingerprinting”, 2002 IEEE Workshop on Multimedia Signal Processing, 2002, pp. 169-173.
[14] Jonathan T. Foote, “Content-based retrieval of music and audio”, Multimedia Storage and Archiving Systems II Vol. 3229, 1997, pp. 138-147.

[15] http://www.midomi.com/
[16] https://www.cakewalk.com/Products/SONAR
[17] http://www.nerds.de/en/loopbe1.html
[18] https://www.guitar-pro.com/en/index.php
[19] http://www.asio4all.com/
[20] https://goo.gl/or5cu1
[21] https://midi-protocol.wikispaces.com/03-MIDI+Message
[22] https://www.music.mcgill.ca/~gary/rtmidi/

[23] Mu Hsen Hsu, “Real-Time Finger Tracking for Virtual Instruments”, 2014 7th International Conference on Ubi-Media Computing and Workshops (UMEDIA), 2014, pp. 133-138.

[24] https://duo3d.com/product/duo-minilx-lv1#tab=specs