本研究針對第一人稱視角 RGB 影像，進行手指指尖即時偵測，並依此於智慧型眼鏡中實作空中手寫輸入的應用。首先，我們以 Unity3D 建立訓練資料集，即利用 3D 手部模型合成於自然場景中以快速地產生大量且高品質的訓練影像與標記資料，同時避免人工標記所可能產生的誤差。我們討論如何讓人工合成影像更貼近實際影像，並利用包含調整背景複雜度、光線明亮度、色彩對比等方式產生多樣化的影像以增加模型的可靠度。接著，我們改良 Mask R-CNN 模型，藉由簡化特徵提取網路，以及改善網路模型對於偵測小物件的適應性，讓所提出的模型在精準度或速度上都為該領域最佳，在 640×480 的 RGB 影像上進行手指偵測，平均像素誤差僅 8.31 像素點，處理畫幀速度達到每秒 38.8 張。最後我們整合手指偵測網路模型於智慧型眼鏡中，以手指指尖移動軌跡作為手寫輸入，再利用 Google Input API 辨識文字以回傳候選字給智慧型眼鏡使用者選擇，建立適用於智慧型眼鏡的新互動輸入法。

This research investigates real-time fingertip detection in RGB images/frames captured from such wearable devices as smart glasses. First, we established a synthetic dataset by using Unity3D and focused on the pointing gesture for egocentric view. The advantage of synthetic data is to avoid manual labeling errors and provide a large benchmark dataset with high quality. We discuss the dataset generation and how to produce the images in a natural way. Second, a modified Mask Regional Convolution Neural Network (Mask R-CNN) is proposed with one region-based CNN for hand detection and another three-layer CNN for locating the fingertip. We employ MobileNetV2 as the backbone network and simplify the number of bottleneck layers to avoid redundant features. Moreover, we improve the accuracy of detecting small objects by employing FPN and RoIAlign. We achieve fingertip detection with 25 milliseconds per frame for the 640×480 resolution by GPU and average 8.31 pixel errors. The processing speed is high enough to facilitate several interesting applications. One application is to trace the location of a user’s fingertip from first-person perspective to form writing trajectories. A text input mechanism for smart glasses can thus be implemented to enable a user to write letters/characters in air as the input and even interact with the system using simple gestures. Experimental results demonstrate the feasibility of this new text input methodology.

[1] K. He, G. Gkioxari, P. Dollár and R. Girshick, "Mask R-CNN," 2017 IEEE International Conference on Computer Vision (ICCV), Venice, 2017, pp. 2980-2988.
[2] T. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan and S. Belongie, "Feature Pyramid Networks for Object Detection," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp. 936-944.
[3] S. Ren, K. He, R. Girshick and J. Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1137-1149, 1 June 2017.
[4] M. de La Gorce, D. J. Fleet and N. Paragios, "Model-Based 3D Hand Pose Estimation from Monocular Video," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 9, pp. 1793-1805, Sept. 2011.
[5] P. Krejov and R. Bowden, "Multi-touchless: Real-time fingertip detection and tracking using geodesic maxima," 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Shanghai, 2013, pp. 1-7.
[6] Liang, Hui & Yuan, Junsong & Thalmann, Daniel, “3D Fingertip and Palm Tracking in Depth Image Sequences,” MM 2012 - Proceedings of the 20th ACM International Conference on Multimedia, pp. 785-788, 2012
[7] Y. Cao, X. Niu and Y. Dou, "Region-based convolutional neural networks for object detection in very high resolution remote sensing images," 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), Changsha, 2016, pp. 548-554.
[8] J. Redmon, S. Divvala, R. Girshick and A. Farhadi, "You Only Look Once: Unified, Real-Time Object Detection," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, 2016, pp. 779-788.
[9] J. Redmon and A. Farhadi, "YOLO9000: Better, Faster, Stronger," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp. 6517-6525.
[10] J. Redmon and A. Farhadi, “YOLOv3: An Incremental Improvement,” arXiv:1804.02767, April, 2018.
[11] Liu, Xiaorui; Huang, Yichao; Zhang, Xin; Jin, Lianwen, “Fingertip in the Eye: A cascaded CNN pipeline for the real-time fingertip detection in egocentric videos,” arXiv:1511.02282, November, 2015.
[12] Y. Huang, X. Liu, X. Zhang and L. Jin, "A Pointing Gesture Based Egocentric Interaction System: Dataset, Approach and Application," 2016 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Las Vegas, NV, 2016, pp. 370-377.
[13] Mukherjee, Sohom; Ahmed, Arif; Prosad Dogra, Debi; Kar, Samarjit; Pratim Roy, Partha, “Fingertip Detection and Tracking for Recognition of Air-Writing in Videos,” arXiv:1809.03016, September, 2018.
[14] Wei Liu and Dragomir Anguelov and Dumitru Erhan and Christian Szegedy and Scott E. Reed and Cheng-Yang Fu and Alexander C. Berg, “SSD: Single Shot MultiBox Detector,” ECCV, 2016.
[15] Simonyan, Karen & Zisserman, Andrew, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” arXiv 1409.1556, September, 2014.
[16] K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, 2016, pp. 770-778.
[17] G. Huang, Z. Liu, L. v. d. Maaten and K. Q. Weinberger, "Densely Connected Convolutional Networks," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp. 2261-2269.
[18] C. Szegedy et al., "Going deeper with convolutions," 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, 2015, pp. 1-9.
[19] F. Chollet, "Xception: Deep Learning with Depthwise Separable Convolutions," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp. 1800-1807.
[20] Howard, Andrew G.; Zhu, Menglong; Chen, Bo; Kalenichenko, Dmitry; Wang, Weijun; Weyand, Tobias; Andreetto, Marco; Adam, Hartwig, “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,” arXiv:1704.04861, April, 2017.
[21] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov and L. Chen, "MobileNetV2: Inverted Residuals and Linear Bottlenecks," 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018, pp. 4510-4520.
[22] A. Gupta, A. Vedaldi and A. Zisserman, "Synthetic Data for Text Localisation in Natural Images," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, 2016, pp. 2315-2324.
[23] Zimmermann, Christian; Brox, Thomas, “Learning to Estimate 3D Hand Pose from Single RGB Images,” arXiv:1705.01389, May, 2017
[24] J. Zhang, J. Jiao, M. Chen, L. Qu, X. Xu and Q. Yang, "A hand pose tracking benchmark from stereo matching," 2017 IEEE International Conference on Image Processing (ICIP), Beijing, 2017, pp. 982-986.
[25] von Zitzewitz, Gustav, “Deep Learning and Real-Time Computer Vision for Mobile Platforms,” 2018.
[26] Yin Guobing, “Deep Learning: Separable Convolution,” https://yinguobing.com/separable-convolution
[27] Joyce Xu, “Deep Learning for Object Detection: A Comprehensive Review,” https://towardsdatascience.com/deep-learning-for-object-detection-a-comprehensive-review-73930816d8d9
[28] Henriques, J. F., Caseiro, R., Martins, P., and Batista, J. “High-speed tracking with kernelized correlation filters,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3):583–596, 2015.
[29] Kalal, Z., Mikolajczyk, K., and Matas, J. “Tracking-learning-detection,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(7):1409–1422, 2012.
[30] Babenko, B., Yang, M.-H., and Belongie, S. “Robust object tracking with online multiple instance learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(8):1619–1632, 2011.