從HCI(Human Computer Interaction, HCI)人機互動轉變到HRI(Human Robot Interaction, HRI)人機互動的應用中，更著重與機器人的互動上。機器人透過感測器感知人類身體姿勢特徵，對其進行分析並給予適當的反饋，是一種比較直覺的HRI方式。本論文提出三種基於人類身體姿勢特徵的HRI應用，其中第一種和第二種是與移動機器人的互動應用，第三種則是與機械手臂的互動應用。
移動機器人在自主導航中，通常都是點到點之間的移動，雖然效率很高但缺乏應用的彈性，所以本論文在與移動機器人的互動中提供兩種可行的應用來增加其彈性。本論文在第一種HRI的應用中提出在移動機器人導航的途中，切換到利用穿戴感測裝置的手勢辨識系統來控制移動機器人移動到定位，以五種手勢命令來控制移動機器人的「前進」、「後退」、「左轉」、「右轉」、「停止」，此手勢辨識系統使用人工神經網路(Artificial Neural Network, ANN)當作分類器來辨識這五種手勢，利用ANN模型對手勢進行訓練和辨識。而在控制移動平台時，設計兩種擷取手勢特徵方法，並利用控制到定位的時間以及在規定時間內控制移動平台與定位的距離和方位角度誤差來比較兩種方法的性能;而本論文在的第二種HRI的應用中提出在移動機器人導航的途中，切換到利用穿戴感測裝置和RGB-D影像感測器的模糊推論系統結合人臉偵測系統來完成定位，透過事先訓練的兩個模糊模型的推論來分別控制移動平台的轉動角度和移動距離，此移動平台的不斷推論控制方式將使得控制平台之人員之人臉偵測的預測bounding box逐漸與理想bounding box接近而達到平台對人員的最佳影像擷取。
本論文的第三種HRI應用是透過RGB-D影像感測器的身體姿勢辨識系統來與機械手臂完成一個組裝任務，以十種身體姿勢動作對應到四種機械手臂反饋的動作，此身體姿勢辨識系統提出一個使用卷積神經網路(Convolutional Neural Network, CNN)和長短期記憶模型(Long Short-Term Memory, LSTM)的三通道深度學習模型來辨識身體姿勢動作，第一個通道是使用彩色圖像訓練CNN-LSTM深度學習模型，第二個通道是使用深度圖像訓練CNN-LSTM深度學習模型，第三個通道是使用人體骨架關節點raw data訓練LSTM深度學習模型，並將三個通道的預測結果經過權重分配決策融合在一起，來達到提升辨識率的效果。
在本論文三種人機互動(HRI)的實驗中，第一種HRI應用的手勢辨識實驗，運用ANN對五種手勢進行辨識，其辨識率為100%，並且在此實驗中比較兩種手勢特徵擷取方法，其中變動時間特徵擷取方法的性能較佳;在第二種HRI應用的模糊系統與人臉偵測定位的實驗，我們實驗了五種不同距離的定位，運用兩個模糊模型控制移動平台的運動，實驗結果發現人臉偵測的預測bounding box與理想的bounding box的誤差會越來越小(最小的誤差值僅為9);在第三種HRI應用的身體姿勢辨識實驗，運用CNN-LSTM深度學習之彩色圖像的辨識率為95%，深度圖像為87.66%，運用LSTM深度學習之人體骨架關節點raw data之辨識率為87.33%，而權重分配決策融合後的辨識率為96%，此辨識率與任一個單一通道的辨識率相比皆有微幅的提升。此方法亦可有效使用於實際人機互動。本論文研究的實驗結果顯示本論文所發展的三個基於人類身體姿勢特徵的人機互動應用皆為可行。

From human computer interaction (HCI) to human robot interaction (HRI), more emphasis is placed on the interaction with robots. Robots perceive human body gesture features through sensors, analyze them and give appropriate feedback. HRI is a more intuitive way. This paper proposes three HRI applications based on human body gesture features. The first and the second applications are interactive applications with mobile robots, and the third is an interactive application with robotic arms.
In autonomous navigation, mobile robots usually move from point to point. Although the efficiency is high, it lacks application flexibility. Therefore, this paper provides two feasible applications in the interaction with mobile robots to increase its flexibility. In the first application of HRI, this paper proposes to switch to the gesture recognition system using the wearable sensing device to control the mobile robot to move to positioning during the navigation of the mobile robot. Five gesture commands are used to control the "forward", "backward", "left turn", "right turn" and "stop" of the mobile robot. This gesture recognition system that uses artificial neural network (ANN) as a classifier is to identify these five gesture actions, and ANN model is used to train and identify the gesture model. When controlling the mobile platform, two methods of capturing gesture features are designed, and the performance of the two methods is compared by using the time from control to positioning and the distance and azimuth angle errors between the control and positioning within the specified time; in the second application of HRI of this thesis, it is proposed to switch to the fuzzy inference system using wearable sensing devices and RGB-D image sensors combined with the face detection system to complete the positioning on the way of mobile robot navigation. The inference of two fuzzy models is used to control the rotation angle and moving distance of the mobile platform, so that the parameter value of the predict bounding box of face detection will be closer to the ideal bounding box parameter value because the fuzzy inference system controls the movement of the mobile platform.
The third HRI application of this paper is to complete an assembly task with the robotic arm through the body gesture recognition system of the RGB-D image sensor. In this work, ten body gesture actions correspond to four feedback actions of the robotic arm. In body gesture recognition, a three-channel deep learning model using convolutional neural network (CNN) and long short-term memory (LSTM) is proposed to identify body gestures. The first channel uses color image to train the CNN-LSTM deep learning model, and the second channel uses the depth image to train the CNN-LSTM deep learning model. The third channel uses the raw data of the human skeleton joint points to train the LSTM deep learning model. Finally, three channels are combined. The prediction results are fused together through the weight distribution decision to achieve the effect of improving the recognition rate.
In the experiment of three kinds of HRI applications in this paper, the first gesture recognition experiment of HRI application uses ANN to recognize five gestures, and the recognition rate is 100%. The feature extraction of two gestures is compared in this experiment. Among them, the performance of the variable time feature extraction method is better; in the experiment of the second HRI application of the fuzzy system and face detection and positioning, we conduct five different distance positioning, using two fuzzy models to control. During the movement of the mobile platform, the error between the predicted bounding box of the face detection and the ideal bounding box is getting smaller and smaller, and there is even an error value of 9; in the third HRI application, the body gesture recognition rate using the color image of CNN-LSTM deep learning is 95%, the depth image of CNN-LSTM deep learning is 87.66%. The recognition rate of raw data of human skeleton joint points using LSTM deep learning is 87.33%. The recognition rate after weight allocation decision fusion is 96%. When this method is used in actual human-robot interaction, the recognition rate is still high. Experimental results show the feasibility of three HRI applications developed in this thesis.

摘要...i
Abstract...iii
誌謝...v
目錄...vi
表目錄...viii
圖目錄...xi
第一章 緒論...1
1.1 研究動機...1
1.2 文獻回顧與探討...1
1.3 章節概要...2
第二章 相關技術介紹...3
2.1 穿戴式感測裝置之手勢特徵數據擷取...3
2.1.1 穿戴式感測裝置硬體架構與系統架構...3
2.1.2 穿戴感測裝置之表面肌電訊號特徵擷取...4
2.1.3 穿戴感測裝置之慣性量測單元特徵擷取...6
2.2 人工神經網路(Artificial Neural Network, ANN)...10
2.3 RealSense人體骨架追蹤系統...11
2.3.1 OpenNI2...12
2.3.2 NITE2介紹...12
2.4 長短期記憶(Long Short-Term Memory, LSTM)...13
2.4.1 LSTM內部運作...13
2.5 卷積神經網路(Convolutional Neural Network, CNN)...17
2.5.1 VGGNet...18
第三章 基於穿戴感測手勢特徵的人機互動應用系統設計...19
3.1 基於ROS的SLAM與導航介紹...19
3.2 基於命令式手勢辨識控制移動平台定位系統設計...20
3.2.1 AIOT手勢辨識控制移動平台(主從式)系統設計...21
3.2.2 手勢命令喚醒與結束機制...22
3.2.3 ANN手勢辨識系統...23
3.2.4 命令式手勢辨識控制移動平台定位示範應用...24
3.3 結合模糊系統與深度學習的移動平台定位方法設計...26
3.3.1 移動平台(邊緣計算式)...26
3.3.2 模糊系統之基礎架構...27
3.3.3 一種使用穿戴手勢感測及平台影像感測之模糊推論結合人臉偵測的移動平台導引設計 ...30
第四章 基於影像感測身體姿勢辨識的人機互動應用系統設計...41
4.1 基於影像感測裝置之身體姿勢辨識的機械手臂互動系統設計...41
4.2 影像感測之三通道身體姿勢辨識的深度學習模型...42
4.2.1 身體骨架關節點raw data 之LSTM深度學習模型...43
4.2.2 CNN-LSTM深度學習模型...44
4.3 多通道深度學習模型決策之融合方法設計...46
4.4 影像感測裝置之身體姿勢辨識的機械手臂互動系統應用...47
第五章 實驗結果與分析...49
5.1 SLAM導航實驗...49
5.1.1 建立地圖與初始定位...49
5.1.2 路徑規劃和導航...52
5.2 手勢辨識之人機互動系統實驗...53
5.2.1 手勢辨識資料庫建置...53
5.2.2 ANN手勢辨識實驗...55
5.2.3 手勢辨識控制移動平台到定位實驗...55
5.3 模糊系統與深度學習的移動平台定位實驗...63
5.3.1 Fuzzy model1和Fuzzy model2推論的實驗結果...65
5.3.2 模糊系統結合人臉偵測定位的實驗結果...66
5.4 身體姿勢辨識之人機互動系統實驗...72
5.4.1 身體姿勢辨識資料庫建立...73
5.4.2 三通道深度學習模型的決策融合實驗...84
5.4.3 身體姿勢辨識人機互動實驗與展示...88
第六章 結論...90
參考文獻...91
Extended Abstract...96

[1]X. Huang, Q. Wang, S. Zang, J. Wan, G. Yang, Y. Huang and X. Ren, “Tracing the Motion of Finger Joints for Gesture Recognition via Sewing RGO-Coated Fibers Onto a Textile Glove,” IEEE Sensors Journal, vol. 19, no. 20, pp. 9504-9511, 15 Oct.15, 2019.
[2]M. Lee and J. Bae, “Deep Learning Based Real-Time Recognition of Dynamic Finger Gestures Using a Data Glove,” IEEE Access, vol. 8, pp. 219923-219933, 2020.
[3]P. Pławiak, T. Sośnicki, M. Niedźwiecki, Z. Tabor and K. Rzecki, “Hand Body Language Gesture Recognition Based on Signals from Specialized Glove and Machine Learning Algorithms,” IEEE Transactions on Industrial Informatics, vol. 12, no. 3, pp. 1104-1113, June 2016.
[4]S. Jiang, B. Lv, W. Guo, C. Zhang, H. Wang, X. Sheng and P.B. Shull, “Feasibility of Wrist-Worn, Real-Time Hand, and Surface Gesture Recognition via sEMG and IMU Sensing,” IEEE Transactions on Industrial Informatics, vol. 14, no. 8, pp. 3376-3385, Aug. 2018.
[5]C. Shen, Y. Chen, G. Yang and X. Guan, “Toward Hand-Dominated Activity Recognition Systems with Wristband-Interaction Behavior Analysis,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 50, no. 7, pp. 2501-2511, July 2020.
[6]Y. Zhang, B. Liu and Z. Liu, “Recognizing Hand Gestures with Pressure-Sensor-Based Motion Sensing,” IEEE Transactions on Biomedical Circuits and Systems, vol. 13, no. 6, pp. 1425-1436, Dec. 2019.
[7]T. Simões Dias, J. J. A. M. Júnior and S. F. Pichorim, “An Instrumented Glove for Recognition of Brazilian Sign Language Alphabet,” IEEE Sensors Journal, vol. 22, no. 3, pp. 2518-2529, 1 Feb.1, 2022.
[8]T. Zhao, J. Liu, Y. Wang, H. Liu and Y. Chen, “Towards Low-Cost Sign Language Gesture Recognition Leveraging Wearables,” IEEE Transactions on Mobile Computing, vol. 20, no. 4, pp. 1685-1701, 1 April 2021.
[9]H. Rewari, V. Dixit, D. Batra and N. Hema, “Automated Sign Language Interpreter,” Proc. 2018 Eleventh International Conference on Contemporary Computing (IC3), 2018, pp. 1-5.
[10]A. Sengupta, T. Mallick and A. Das, “A Cost Effective Design and Implementation of Arduino Based Sign Language Interpreter,” Proc. 2019 Devices for Integrated Circuit (DevIC), 2019, pp. 12-15.
[11]S. P. Y. Jane and S. Sasidhar, “Sign Language Interpreter: Classification of Forearm EMG and IMU Signals for Signing Exact English,” Proc. 2018 IEEE 14th International Conference on Control and Automation (ICCA), 2018, pp. 947-952.
[12]D. L. Arsenault and A. D. Whitehead, “Quaternion Based Gesture Recognition Using Worn Inertial Sensors in a Motion Tracking System,” Proc. 2014 IEEE Games Media Entertainment, 2014, pp. 1-7.
[13]Y. Liu, F. Jiang and M. Gowda, “Application Informed Motion Signal Processing for Finger Motion Tracking Using Wearable Sensors,” Proc. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 8334-8338.
[14]P. Bellitti, A.D. Angelis, M. Dionigi, E. Sardini, M. Serpelloni, A. Moschitta and P. Carbone, “A Wearable and Wirelessly Powered System for Multiple Finger Tracking,” IEEE Transactions on Instrumentation and Measurement, vol. 69, no. 5, pp. 2542-2551, May 2020.
[15]Y. Chen, Y. Tsai, K. Huang and P. H. Chou, “MobiRing: A Finger-Worn Wireless Motion Tracker,” Proc. 2014 IEEE International Conference on Internet of Things (iThings), and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom), 2014, pp. 316-323.
[16]H. S. Chudgar, S. Mukherjee and K. Sharma, “S Control: Accelerometer-Based Gesture Recognition for Media Control,” Proc. 2014 International Conference on Advances in Electronics Computers and Communications, 2014, pp. 1-6.
[17]J. Yao, H. Chen, Z. Xu, J. Huang, J. Li, J. Jia and H. Wu, “Development of a Wearable Electrical Impedance Tomographic Sensor for Gesture Recognition with Machine Learning,” IEEE Journal of Biomedical and Health Informatics, vol. 24, no. 6, pp. 1550-1556, June 2020.
[18]X. Liang, R. Ghannam and H. Heidari, “Wrist-Worn Gesture Sensing with Wearable Intelligence,” IEEE Sensors Journal, vol. 19, no. 3, pp. 1082-1090, 1 Feb.1, 2019.
[19]F. -T. Liu, Y. -T. Wang and H. -P. Ma, “Gesture Recognition with Wearable 9-Axis Sensors,” Proc. 2017 IEEE International Conference on Communications (ICC), 2017, pp. 1-6.
[20]K. S. Krishnan, A. Saha, S. Ramachandran and S. Kumar, “Recognition of Human Arm Gestures Using Myo Armband for the Game of Hand Cricket,” Proc. 2017 IEEE International Symposium on Robotics and Intelligent Sensors (IRIS), 2017, pp. 389-394.
[21]Y. -T. Liu, Y. -A. Zhang and M. Zeng, “Novel Algorithm for Hand Gesture Recognition Utilizing a Wrist-Worn Inertial Sensor,” IEEE Sensors Journal, vol. 18, no. 24, pp. 10085-10095, 15 Dec.15, 2018.
[22]K. Yamagishi, L. Jing and Z. Cheng, “A System for Controlling Personal Computers by Hand Gestures Using a Wireless Sensor Device,” Proc. 2014 IEEE International Symposium on Independent Computing (ISIC), 2014, pp. 1-7,.
[23]D. Moazen, S. A. Sajjadi and A. Nahapetian, “AirDraw: Leveraging Smart Watch Motion Sensors for Mobile Human Computer Interactions,” Proc. 2016 13th IEEE Annual Consumer Communications & Networking Conference (CCNC), 2016, pp. 442-446.
[24]X. Liu, J. Sacks, M. Zhang, A. G. Richardson, T. H. Lucas and J. Van der Spiegel, “The Virtual Trackpad: An Electromyography-Based, Wireless, Real-Time, Low-Power, Embedded Hand-Gesture-Recognition System Using an Event-Driven Artificial Neural Network,” IEEE Transactions on Circuits and Systems II: Express Briefs, vol. 64, no. 11, pp. 1257-1261, Nov. 2017.
[25]S. R. Kurniawan and D. Pamungkas, “MYO Armband Sensors and Neural Network Algorithm for Controlling Hand Robot,” Proc. 2018 International Conference on Applied Engineering (ICAE), 2018, pp. 1-6.
[26]F. S. Sayin, S. Ozen and U. Baspinar, “Hand Gesture Recognition by Using sEMG Signals for Human Machine Interaction Applications,” Proc. 2018 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA), 2018, pp. 27-30.
[27]S. Hickman, A. S. Mirzakhani, J. Pabon and R. Alba-Flores, “A Case Study on Tuning Artificial Neural Networks to Recognize Signal Patterns of Hand Motions,” Proc. SoutheastCon 2015, 2015, pp. 1-4.
[28]U. Baspinar, H. Selcuk Varol and K. Yildiz, “Classification of Hand Movements by Using Artificial Neural Network,” Proc. 2012 International Symposium on Innovations in Intelligent Systems and Applications, 2012, pp. 1-4.
[29]D. Qu, Z. Huang, Z. Gao, Y. Zhao, X. Zhao and G. Song, “An Automatic System for Smile Recognition Based on CNN and Face Detection,” Proc. 2018 IEEE International Conference on Robotics and Biomimetics (ROBIO), 2018, pp. 243-247.
[30]Y. Zhou, H. Ni, F. Ren and X. Kang, “Face and Gender Recognition System Based on Convolutional Neural networks,” Proc. 2019 IEEE International Conference on Mechatronics and Automation (ICMA), 2019, pp. 1091-1095.
[31]R. Szmurlo and S. Osowski, “Deep CNN Ensemble for Recognition of Face Images,” Proc. 2021 22nd International Conference on Computational Problems of Electrical Engineering (CPEE), 2021, pp. 1-4.
[32]A. Song, Q. Hu, X. Ding, X. Di and Z. Song, “Similar Face Recognition Using the IE-CNN Model,” IEEE Access, vol. 8, pp. 45244-45253, 2020.
[33]D. S. Breland, A. Dayal, A. Jha, P. K. Yalavarthy, O. J. Pandey and L. R. Cenkeramaddi, “Robust Hand Gestures Recognition Using a Deep CNN and Thermal Images,” IEEE Sensors Journal, vol. 21, no. 23, pp. 26602-26614, 1 Dec.1, 2021.
[34]G. Lingyun, Z. Lin and W. Zhaokui, “Hierarchical Attention-Based Astronaut Gesture Recognition: A Dataset and CNN Model,” IEEE Access, vol. 8, pp. 68787-68798, 2020.
[35]D. Fan, H. Lu, S. Xu and S. Cao, “Multi-Task and Multi-Modal Learning for RGB Dynamic Gesture Recognition,” IEEE Sensors Journal, vol. 21, no. 23, pp. 27026-27036, 1 Dec.1, 2021.
[36]F. Zhan, “Hand Gesture Recognition with Convolution Neural Networks,” Proc. 2019 IEEE 20th International Conference on Information Reuse and Integration for Data Science (IRI), 2019, pp. 295-298.
[37]Z. Liu, X. Wang, M. SU and K. Lu, “A Method to Recognize Sleeping Position Using an CNN Model Based on Human Body Pressure Image,” Proc. 2019 IEEE International Conference on Power, Intelligent Computing and Systems (ICPICS), 2019, pp. 219-224.
[38]M. Atikuzzaman, T. R. Rahman, E. Wazed, M. P. Hossain and M. Z. Islam, “Human Activity Recognition System from Different Poses with CNN,” Proc. 2020 2nd International Conference on Sustainable Technologies for Industry 4.0 (STI), 2020, pp. 1-5.
[39]M. CUI, J. FANG and Y. ZHAO, “Emotion Recognition of Human Body's Posture in Open Environment,” Proc. 2020 Chinese Control And Decision Conference (CCDC), 2020, pp. 3294-3299.
[40]S. Mohsen, A. Elkaseer and S. G. Scholz, “Industry 4.0-Oriented Deep Learning Models for Human Activity Recognition,” IEEE Access, vol. 9, pp. 150508-150521, 2021.
[41]E. Martinez-Martin and M. Cazorla, “A Socially Assistive Robot for Elderly Exercise Promotion,” IEEE Access, vol. 7, pp. 75515-75529, 2019.
[42]Z. Yu and W. Q. Yan, “Human Action Recognition Using Deep Learning Methods,” Proc. 2020 35th International Conference on Image and Vision Computing New Zealand (IVCNZ), 2020, pp. 1-6.
[43]T. Nyajowi, N. Oyie and M. Ahuna, “CNN Real-Time Detection of Vandalism Using a Hybrid -LSTM Deep Learning Neural Networks,” Proc. 2021 IEEE AFRICON, 2021, pp. 1-6.
[44]D. Wang, J. Yang and Y. Zhou, “Human Action Recognition Based on Multi-Mode Spatial-Temporal Feature Fusion,” Proc. 2019 22th International Conference on Information Fusion (FUSION), 2019, pp. 1-7.
[45]H. El-Ghaish, M. E. Hussien, A. Shoukry and R. Onai, “Human Action Recognition Based on Integrating Body Pose, Part Shape, and Motion,” IEEE Access, vol. 6, pp. 49040-49055, 2018.
[46]U. Haroon, A. Ullah, T. Hussain, W. Ullah, M. Sajjad, K. Muhammad, M.Y. Lee and S.W. Baik, “A Multi-Stream Sequence Learning Framework for Human Interaction Recognition,” IEEE Transactions on Human-Machine Systems, vol. 52, no. 3, pp. 435-444, June 2022.
[47]X. Weiyao, W. Muqing, Z. Min and X. Ting, “Fusion of Skeleton and RGB Features for RGB-D Human Action Recognition,” IEEE Sensors Journal, vol. 21, no. 17, pp. 19157-19164, 1 Sept.1, 2021.
[48]Y. Tang, Z. Wang, J. Lu, J. Feng and J. Zhou, “Multi-Stream Deep Neural Networks for RGB-D Egocentric Action Recognition,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 29, no. 10, pp. 3001-3015, Oct. 2019.
[49]J. Yu, H. Gao, W. Yang, Y. Jiang, W. Chin, N. Kubota and Z. Ju, “A Discriminative Deep Model with Feature Fusion and Temporal Attention for Human Action Recognition,” IEEE Access, vol. 8, pp. 43243-43255, 2020.
[50]A. Kamel, B. Sheng, P. Yang, P. Li, R. Shen and D. D. Feng, “Deep Convolutional Neural Networks for Human Action Recognition Using Depth Maps and Postures,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 49, no. 9, pp. 1806-1819, Sept. 2019.
[51]J. He, H. Xia, C. Feng and Y. Chu, “CNN-Based Action Recognition Using Adaptive Multiscale Depth Motion Maps and Stable Joint Distance Maps,” Proc. 2018 IEEE Global Conference on Signal and Information Processing (GlobalSIP), 2018, pp. 439-443.
[52]顏承宇，2019，“運用視訊及音訊感測之非正常人顏面族群的發音照護系統設計研究”，國立虎尾科技大學電機工程學系碩士論文，2014。