臺灣博碩士論文加值系統

隨著科技不斷發展及革新，現今社會已經發展出許多新穎且不需透過傳統搖桿或其他附加設備就能進行的操作方式。基於此，本篇論文旨在探索如何運用影像辨識的方式來控制系統。主要研究是採用Google的MediaPipe手部辨識系統，該系統擁有識別迅速手部骨架的優秀性能，並能透過推理算法來確定被遮蔽部位的手部骨架位置，進而提升辨識的精確度。此外，本研究還混合了YOLOv5的辨識方法，從而打造辨識率較高的辨識系統。利用MediaPipe來偵測手部骨架並識別手勢，並將其與YOLO的穩定辨識能力相混合。首先在辨識開始中為手部建立一個的定位框，然後由MediaPipe負責在框內進行骨架偵測和手勢辨識，以創建出一個即時且穩定的手部辨識系統。接著再透過Socket通訊，將辨識到的訊息即時傳送到車輛，實現精準的車輛控制。在最後的比較中，MediaPipe單獨使用時的整體平均F1分數為96%，而YOLO的平均F1分數為81.2%。然而，當MediaPipe和YOLO混合使用時，平均F1分數可以達到97%以上。根據這些結果，可以清楚地看出，藉由混合不同辨識方法，成功地提升了系統的精確率，從而提高了辨識能力。

With the continuous development and innovation of technology, modern society has developed many novel operation methods that do not require traditional joysticks or other additional equipment. Based on this, this paper aims to explore how to control systems using image recognition methods. The main focus of study is the use of Google's MediaPipe hand recognition system. This system possesses excellent performance in rapidly identifying hand skeletons and can use inferential algorithms to determine the positions of obscured parts of the hand skeleton, thereby improving the accuracy of recognition. In addition, this research also combines the YOLOv5 recognition method to create a recognition system with a higher recognition rate. Use MediaPipe to detect hand skeletons and recognize gestures, and combine it with the stable recognition ability of YOLO. At the beginning of recognition, a positioning frame is first established for the hand, and then MediaPipe is responsible for detecting the skeleton and recognizing gestures within the frame, thereby creating a real-time and stable hand recognition system. Next, through Socket communication, the recognized information is sent to the vehicle in real-time, achieving precise vehicle control. In the final comparison, when used individually, MediaPipe achieved an overall average F1 score of 96%, while YOLO achieved an average F1 score of 81.2%. However, when MediaPipe and YOLO are combined, the average F1 score can surpass 97%. According to these results, it is clear that by combining different recognition methods, we have successfully improved the accuracy rate of the system, thereby enhancing recognition performance.

摘要...i
Abstract...ii
誌謝...iii
目錄...iv
表目錄...vi
圖目錄...vii
第1章 緒論...1
1.1 研究背景...1
1.2 研究動機與目的...1
1.3 論文架構...2
第2章 文獻探討...3
2.1 Mediapipe...3
2.1.1 手部追蹤模型...5
2.1.2 手掌檢測器...6
2.1.3 手部關鍵點...8
2.2 深度學習...8
2.2.1 卷積神經網路...9
2.2.2 YOLO...11
第3章 系統規劃與製作...14
3.1 實驗硬體設備...14
3.2 系統規劃...15
3.2.1 系統流程...15
3.2.2 系統架構...16
3.2.3 資料收集...16
3.2.4 手勢樣本數據與前置作業...17
3.2.5 模型訓練...19
3.2.6 模型辨識...19
3.2.7 傳輸控制...20
3.3 MediaPipe手部偵測...21
3.3.1 辨識流程...21
3.3.2 手部辨識...21
3.3.3 手勢狀態...22
3.4 YOLOv5手部偵測...22
3.4.1 訓練流程...22
3.4.2 影像辨識...23
3.5 MediaPipe混合Yolov5手部偵測...23
3.5.1 辨識方法...23
第4章 實驗結果與分析...25
4.1 手勢偵測模型方法...25
4.1.1 自定義數據集...25
4.1.2 數據標籤...25
4.2 手勢偵測模型評估方法...26
4.3 Mediapipe實驗結果...28
4.4 YOLOv5實驗結果...32
4.5 YOLOv5混合MediaPipe實驗結果...36
4.6 三種模型實驗結果比較...38
第5章 結論與未來研究...42
參考文獻...43
Extended Abstract...45

[1]arm. "xR、AR、VR 與 MR：在實境上的區別." Jacob Hooker. https://www.arm.com/zh-TW/resources/blueprint/xr-ar-vr-mr-difference (accessed June 12, 2023).
[2]techcrunch. "This hand-tracking algorithm could lead to sign language recognition." Devin Coldewey. https://techcrunch.com/2019/08/19/this-hand-tracking-algorithm-could-lead-to-sign-language-recognition/ (accessed June 12, 2023).
[3]Google. "什麼是人工智慧 (AI)？." Google. https://cloud.google.com/learn/what-is-artificial-intelligence?hl=zh-tw (accessed June 12, 2023).
[4]Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," nature, vol. 521, no. 7553, pp. 436-444, 2015.
[5]教育部. "全國特殊教育資訊網 - 英文字母手勢." 教育部. https://special.moe.gov.tw/articleInfo.php?guid=A27DBF16-4BE9-5E14-1831-77143CC39B0E&paid=254&token=ef9bdfab969d688e780687a5457539d8 (accessed June 12, 2023).
[6]工業技術研究院. "量測提琴手肌肉訊號驚艷黑客松評審團." https://www.itri.org.tw/ListStyle.aspx?DisplayStyle=18_content&SiteID=1&MmmID=1036452026061075714&MGID=710155513041752777 (accessed June 12, 2023).
[7]C. Lugaresi et al., "Mediapipe: A framework for building perception pipelines," arXiv:1906.08172 [cs.DC], 2019.
[8]ultralytics. ultralytics. https://github.com/ultralytics/yolov5 (accessed June 12, 2023).
[9]J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 779-788.
[10]Google. "MediaPipe " Google https://developers.google.com/mediapipe (accessed June 12, 2023).
[11]F. Zhang et al., "Mediapipe hands: On-device real-time hand tracking," arXiv:2006.10214 [cs.CV], 2020.
[12]L. Ge, H. Liang, J. Yuan, and D. Thalmann, "Real-time 3D hand pose estimation with 3D convolutional neural networks," IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 4, pp. 956-970, 2018.
[13]V. Bazarevsky, Y. Kartynnik, A. Vakunov, K. Raveendran, and M. Grundmann, "Blazeface: Sub-millisecond neural face detection on mobile gpus," arXiv:1907.05047 [cs.CV], 2019.
[14]A. H. Mark Sandler, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen, "MobileNetV2: Inverted Residuals and Linear Bottlenecks," arXiv:1801.04381v4 [cs.CV], 2018.
[15]W. Liu et al., "SSD: Single Shot MultiBox Detector," arXiv:1512.02325v5 [cs.CV], pp. 21-37, 2016.
[16]T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature Pyramid Networks for Object Detection," in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017: IEEE, pp. 936-944.
[17]A. Neubeck and L. Van Gool, "Efficient non-maximum suppression," in 18th international conference on pattern recognition (ICPR'06), 2006, vol. 3: IEEE, pp. 850-855.
[18]X. Z. Kaiming He, Shaoqing Ren, Jian Sun, "Deep Residual Learning for Image Recognition," arXiv:1512.03385v1 [cs.CV], 2015.
[19]Y. Zhang, D. Zhao, J. Zhang, R. Xiong, and W. Gao, "Interpolation-dependent image downsampling," IEEE Transactions on Image Processing, vol. 20, no. 11, pp. 3291-3296, 2011.
[20]T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, "Focal loss for dense object detection," in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2980-2988.
[21]iT邦幫忙. "Python 演算法 Day 17 - Neural Network 神經網路." https://ithelp.ithome.com.tw/articles/10283558 (accessed June 12, 2023).
[22]D. A. Forsyth et al., "Object recognition with gradient-based learning," Shape, contour and grouping in computer vision, pp. 319-345, 1999.
[23]R. M. Schmidt, "Recurrent Neural Networks (RNNs): A gentle Introduction and Overview," arXiv:1912.05911v1 [cs.LG], 2019.
[24]Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
[25]T.-Y. Lin et al., "Microsoft coco: Common objects in context," in Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, 2014: Springer, pp. 740-755.
[26]NVIDIA. "NVIDIA Tesla P100." NVIDIA https://www.nvidia.com/zh-tw/data-center/tesla-p100/ (accessed June 12, 2023).
[27]Roboflow. "Responding to the Controversy about YOLOv5." Joseph Nelson, Jacob Solawetz. https://blog.roboflow.com/yolov4-versus-yolov5/ (accessed June 12, 2023).