本研究旨在解決現有人機互動應用中的限制，即機器需要透過指令才能執行任務，缺乏主動協助的能力。為此，本研究提出了一種結合深度學習策略之隱藏式馬可夫模型的人類行為辨識方法。這種方法使用雙通道輸入，其中一個通道為人體骨架關節點特徵，另一個通道為雙手手勢特徵，包含圖像和幾何特徵。通過去手勢遮蔽方法，非手勢資訊被移除，以提高辨識的準確性。深度學習技術在本研究中被廣泛應用，包括長短期記憶網路（LSTM）和卷積神經網路（CNN），用於提取人體骨架、雙手手勢圖像和幾何的深度學習特徵。針對手勢圖像特徵和手勢幾何特徵，本研究再進一步提出並行特徵融合和串接特徵融合方法以將這兩種手勢特徵融合在一起，此方式將能再提升辨識率。隱藏式馬可夫模型（HMM）在這個方法中被用來進行辨識，並使用決策融合方法整合多個辨識結果，以獲得最終的行為辨識結果。除此之外，本研究還設計了具有語意的隱藏式馬可夫模型（Semantic-HMM），這種模型可以由多個類別的HMM的行為辨識結果組成一個具有語意的完整行為的辨識結果。本研究所提出的結合深度學習策略之隱藏式馬可夫模型的人類行為辨識方法可以進一步再與狀態機進行整合以實現更高具有效率的人機協作實現。實驗結果顯示，本研究提出的雙手手勢並行式特徵融合辨識通道與骨架的深度學習特徵辨識通道之決策融合的86.41%辨識率和雙手手勢串接式特徵融合辨識通道與骨架的深度學習特徵辨識通道之決策融合的90.77%的辨識率都明顯高於雙通道都使用非深度學習特徵之決策融合的72.82%的辨識率要高出許多；本研究設計的Semantic-HMM得到的79.52%的整體平均辨識準確率可以宣稱既使行為動作中包含有由多類別的HMM所組成的前後行為相關之具有語意的行為的可行性。最終，本研究將所提出的結合深度學習策略之隱藏式馬可夫模型的人類行為辨識方法更進一步與機械人裝置進行系統整合，其凸顯出本研究所提出方法可以有效運用於生活中之人機協作應用。

This thesis aims to address the limitations in current human-machine interaction applications where machines require explicit instructions to perform tasks and lacked proactive assistance. To solve this issue, the research proposes a novel approach that combines deep learning strategies with hidden Markov model (HMM) for human behavior recognition. The proposed method employs a dual-channel input framework, one channel to capture human body skeletal joint features and the other channel to represent hand gestures with image and geometric features. To improve recognition accuracy, this thesis uses the removal occlusion method to remove non-gesture information. Deep learning techniques, including long short-term memory (LSTM) and convolutional neural networks (CNN), are extensively utilized to extract deep learning features from human body skeletal data, hand gesture images and hand geometric information in this study. For hand gesture images and hand geometric information, this study proposes parallel and cascade feature fusion methods to effectively combine these two types of gesture features for enhancing recognition performance. HMM is employed for recognition, and decision fusion is utilized to integrate multiple recognition results to finally output the overall behavior recognition outcome. Furthermore, this thesis also introduces a semantic hidden Markov model (Semantic-HMM), which represents a semantic and complete behavior recognition result from multiple categories of HMM. The human behavior recognition method that combines deep learning strategies and HMM proposed in this thesis can be further integrated with state machines to achieve more efficiency in human-machine collaboration.
The experimental results demonstrate the effectiveness of the proposed method in this thesis, the decision fusion of dual channels of the dual-hand gesture channel on parallel feature fusion and the skeleton channel using deep learning features achieving a recognition rate of 86.41% and the decision fusion of the dual-hand gesture channel on cascade feature fusion and the skeleton channel using deep learning features reaching a recognition rate of 90.77%. Both recognition rates are significantly higher than the recognition rate of 72.82% achieved by the decision fusion with only non-deep learning features in both channels. The designed Semantic-HMM achieves an overall average recognition accuracy of 79.52%, indicating the feasibility of recognizing semantically meaningful behavior actions containing pre- and post-behavioral correlations composed of multi-class HMMs. Ultimately, this research further integrates the proposed human behavior recognition method with a robotic system, highlighting the proposed method to be able to be effectively used in real-life human-machine collaborative applications.

摘要....i
Abstract........ii
誌謝....iv
目錄....v
表目錄...vii
圖目錄...x
第一章 緒論......1
1.1 研究動機.....1
1.2 文獻回顧與探討.......2
1.3 章節概要.....3
第二章 背景技術...4
2.1 RGB-D影像感測器影像擷取...5
2.2 NiTE2手部以及人體骨架追蹤.....6
2.2.1 人體骨架追蹤.......6
2.2.2 手部影像擷取以及影像處理....7
2.3 手勢特徵擷取....8
2.4 卷積神經網路(Convolution neural network, CNN).......11
2.5 長短期記憶網路(Long short-term memory, LSTM).......12
2.6 隱藏式馬可夫模型(hidden Markov model, HMM).......13
第三章 結合深度學習策略之隱藏式馬可夫模型的人類行為辨識.......15
3.1 去手勢遮蔽方法.......16
3.2 深度學習網路之特徵擷取以及特徵融合.......18
3.2.1 基於卷積神經網路VGG16架構的特徵擷取.......19
3.2.2 長短期記憶模型特徵擷取.......21
3.2.3 雙手手勢深度學習特徵融合.......22
3.3 Semantic-HMM.......25
3.4 雙通道決策融合方法.......27
第四章 實驗結果與分析比較.......30
4.1 實驗環境設定與資料庫建置(特定應用情境).......30
4.2 人類行為辨識實驗設定.......39
4.3 人類行為辨識結果.......40
4.3.1 雙手手勢深度學習特徵與骨架深度學習特徵人類行為辨識結果.......40
4.3.2 深度學習特徵與非深度學習特徵之人類行為辨識結果比較.......45
4.3.3 Semantic-HMM 辨識結果.......47
4.4 基於人類行為辨識的人機協作應用(特定情境).......48
4.4.1 狀態機1：消費者要求咖啡加糖和加奶精.......49
4.4.2 狀態機2：消費者要求咖啡只加糖.......53
4.4.3 狀態機3：消費者要求咖啡只加奶精.......56
4.4.4 狀態機4：消費者要求咖啡不加糖和不加奶精.......59
第五章 結論.......61
參考文獻.......62
附錄一.......67
附錄二.......91
Extended Abstract.......100

[1]A. A. Patil, S. A, A. R. R, N. N. V and G. R, “Human Action Recognition Using Skeleton Features,” Proc. IEEE Int. Symp. Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), pp. 289–296, 2022.
[2]T. Lu, L. Peng and S. Miao, “Human Action Recognition of Hidden Markov Model Based on Depth Information,” Proc. 15th Int. Symp. Parallel and Distributed Computing (ISPDC), pp. 354–357, 2016.
[3]Y. Fan, S. Weng, Y. Zhang, B. Shi and Y. Zhang, “Context-Aware Cross-Attention for Skeleton-Based Human Action Recognition,” IEEE Access, vol. 8, pp. 15280–15290, 2020.
[4]J. Liu, N. Akhtar and A. Mian, “Adversarial Attack on Skeleton-Based Human Action Recognition,” IEEE Transactions on Neural Networks and Learning Systems, vol. 33, no. 4, pp. 1609–1622, 2022.
[5]Z. Deng, Q. Gao, Z. Ju and X. Yu, “Skeleton-Based Multifeatures and Multistream Network for Real-Time Action Recognition,” IEEE Sensors Journal, vol. 23, no. 7, pp. 7397–7409, 2023.
[6]M.-F. Tsai and C.-H. Chen, “Spatial Temporal Variation Graph Convolutional Networks (STV-GCN) for Skeleton-Based Emotional Action Recognition,” IEEE Access, vol. 9, pp. 13870–13877, 2021.
[7]Y. Han, S.-L. Chung, Q. Xiao, W. Y. Lin and S.-F. Su, “Global Spatio-Temporal Attention for Action Recognition Based on 3D Human Skeleton Data,” IEEE Access, vol. 8, pp. 88604–88616, 2020.
[8]X. Jiang, K. Xu and T. Sun, “Action Recognition Scheme Based on Skeleton Representation with DS-LSTM Network,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 7, pp. 2129–2140, 2020.
[9]L. Wang, X. Zhao and Y. Liu, “Skeleton Feature Fusion Based on Multi-Stream LSTM for Action Recognition,” IEEE Access, vol. 6, pp. 50788–50800, 2018.
[10]P. T. Hai and H. H. Kha, “An Efficient Star Skeleton Extraction for Human Action Recognition Using Hidden Markov Models,” Proc. IEEE Sixth Int. Conf. Commun. and Electron. (ICCE), pp. 351–356, 2016.
[11]T. Z. Wint Cho, M. T. Win and A. Win, “Human Action Recognition System Based on Skeleton Data,” Proc. IEEE Int. Conf. Agents (ICA), pp. 93–98, 2018.
[12]林宗毅，2018，「一個運用貝式網路模型機制於三維感測器影像應用之最佳定位方法之方法研究」，國立虎尾科技大學碩士論文。
[13]張育瑞，2016，「一種人體姿態命令辨識及其身分識別的強化式方法之研究」，國立虎尾科技大學碩士論文。
[14]張哲維，2014，「運用3D運動特徵於人體姿態辨識及學習方法之研究」，國立虎尾科技大學碩士論文。
[15]D. Qu, Z. Huang, Z. Gao, Y. Zhao, X. Zhao and G. Song, “An Automatic System for Smile Recognition Based on CNN and Face Detection,” Proc. IEEE Int. Conf. Robotics and Biomimetics (ROBIO), pp. 243–247, 2018.
[16]Y. Zhou, H. Ni, F. Ren and X. Kang, “Face and Gender Recognition System Based on Convolutional Neural networks,” Proc. IEEE Int. Conf. Mechatronics and Automation (ICMA), pp. 1091–1095, 2019.
[17]R. Szmurlo and S. Osowski, “Deep CNN Ensemble for Recognition of Face Images,” Proc. 22nd Int. Conf. Comput. Probl. Electr. Eng. (CPEE), pp. 1–4, 2021.
[18]A.-P. Song, Q. Hu, X.-H. Ding, X.-Y. Di and Z.-H. Song, “Similar Face Recognition Using the IE-CNN Model,” IEEE Access, vol. 8, pp. 45244–45253, 2020.
[19]D. S. Breland, A. Dayal, A. Jha, P. K. Yalavarthy, O. J. Pandey and L. R. Cenkeramaddi, “Robust Hand Gestures Recognition Using a Deep CNN and Thermal Images,” IEEE Sensors Journal, vol. 21, no. 23, pp. 26602–26614, 2021.
[20]G. Lingyun, Z. Lin and W. Zhaokui, “Hierarchical Attention-Based Astronaut Gesture Recognition: A Dataset and CNN Model,” IEEE Access, vol. 8, pp. 68787–68798, 2020.
[21]D. Fan, H. Lu, S. Xu and S. Cao, “Multi-Task and Multi-Modal Learning for RGB Dynamic Gesture Recognition,” IEEE Sensors Journal, vol. 21, no. 23, pp. 27026–27036, 2021.
[22]S. Meshram, R. Singh, P. Pal and S. K. Singh, “Convolution Neural Network based Hand Gesture Recognition System,” Proc. Third Int. Conf. Adv. Electrl. Comput. Commun. Sustain. Technol. (ICAECT), pp. 1–5, 2023.
[23]D. Kollias and S. Zafeiriou, “Exploiting Multi-CNN Features in CNN-RNN Based Dimensional Emotion Recognition on the OMG in-the-Wild Dataset,” IEEE Transactions on Affective Computing, vol. 12, no. 3, pp. 595–606, 2021.
[24]J. Zhang, M. Xing and Y. Xie, “FEC: A Feature Fusion Framework for SAR Target Recognition Based on Electromagnetic Scattering Features and Deep CNN Features,” IEEE Transactions on Geoscience and Remote Sensing, vol. 59, no. 3, pp. 2174–2187, 2021.
[25]J. Zhang, M. Xing, G.-C. Sun and Z. Bao, “Integrating the Reconstructed Scattering Center Feature Maps With Deep CNN Feature Maps for Automatic SAR Target Recognition,” IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2022.
[26]A. I. Shahin and S. Almotairi, “An Accurate and Fast Cardio-Views Classification System Based on Fused Deep Features and LSTM,” IEEE Access, vol. 8, pp. 135184–135194, 2020.
[27]S. Mohsen, A. Elkaseer and S. G. Scholz, “Industry 4.0-Oriented Deep Learning Models for Human Activity Recognition,” IEEE Access, vol. 9, pp. 150508–150521, 2021.
[28]H. Riaz, M. Uzair, H. Ullah and M. Ullah, "Anomalous Human Action Detection Using a Cascade of Deep Learning Models," Proc. 9th Eur. Workshop Visual Inf. Process. (EUVIP), pp. 1–5, 2021.
[29]T. Nyajowi, N. Oyie and M. Ahuna, "CNN Real-Time Detection of Vandalism Using a Hybrid -LSTM Deep Learning Neural Networks," Proc. IEEE AFRICON, pp. 1–6, 2021.
[30]Y. -H. Byeon, D. Kim, J. Lee and K. -C. Kwak, “Ensemble Three-Stream RGB-S Deep Neural Network for Human Behavior Recognition Under Intelligent Home Service Robot Environments,” IEEE Access, vol. 9, pp. 73240–73250, 2021
[31]E. Martinez-Martin and M. Cazorla, “A Socially Assistive Robot for Elderly Exercise Promotion,” IEEE Access, vol. 7, pp. 75515–75529, 2019.
[32]莊亞澄，2022，「一個基於人類身體姿勢特徵的人機互動應用系統設計之研究」，國立虎尾科技大學碩士論文。
[33]Gong, C. Chen and M. Peng, “Human Interaction Recognition Based on Deep Learning and HMM,” IEEE Access, vol. 7, pp. 161123–161130, 2019.
[34]M. Ramadan and A. El-Jaroudi, “Action Detection and Classification in Kitchen Activities Videos Using Graph Decoding,” The Visual Computer, vol. 39, pp. 799–812, 2022.
[35]Y. Gu et al., “Sensor Fusion Based Manipulative Action Recognition,” Auton Robot, vol. 45 pp. 1–13, 2020.
[36]J. Yu et al., “A Discriminative Deep Model with Feature Fusion and Temporal Attention for Human Action Recognition,” IEEE Access, vol. 8, pp. 43243–43255, 2020.
[37]鄭乃瑋，2021，「一個考量彩色攝影機及深度攝影機之手勢影像感測融合的深度學習辨識研究」，國立虎尾科技大學碩士論文。
[38]Y. -H. Byeon, D. Kim, J. Lee and K. -C. Kwak, “Ensemble Three-Stream RGB-S Deep Neural Network for Human Behavior Recognition Under Intelligent Home Service Robot Environments,” IEEE Access, vol. 9, pp. 73240–73250, 2021.
[39]O. Koller, N. C. Camgoz, H. Ney and R. Bowden, “Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 42, no. 9, pp. 2306–2320, 2020.
[40]U. Haroon et al., “A Multi-Stream Sequence Learning Framework for Human Interaction Recognition,” IEEE Transactions on Human-Machine Systems, vol. 52, no. 3, pp. 435–444, 2022.
[41]A. K. Talukdar and M. K. Bhuyan, “Vision-Based Continuous Sign Language Spotting Using Gaussian Hidden Markov Model,” IEEE Sensors Letters, vol. 6, no. 7, pp. 1–4, 2022.
[42]T. Simões Dias, J. J. A. M. Júnior and S. F. Pichorim, “An Instrumented Glove for Recognition of Brazilian Sign Language Alphabet,” IEEE Sensors Journal, vol. 22, no. 3, pp. 2518–2529, 2022.
[43]蘇俊麟，2016，「基於影像之深度資訊的手勢辨識方法研究」，國立虎尾科技大學碩士論文。
[44]Y. Liu, F. Jiang and M. Gowda, “Application Informed Motion Signal Processing for Finger Motion Tracking Using Wearable Sensors,” Proc. IEEE Int. Conf. Acoust. Speech and Signal Process. (ICASSP), pp. 8334–8338, 2020.
[45]C. Mizera, T. Delrieu, V. Weistroffer, C. Andriot, A. Decatoire and J.-P. Gazeau, “Evaluation of Hand-Tracking Systems in Teleoperation and Virtual Dexterous Manipulation,” IEEE Sensors Journal, vol. 20, no. 3, pp. 1642–1655, 2020.
[46]J. P. Sahoo, S. P. Sahoo, S. Ari and S. K. Patra, “Hand Gesture Recognition Using Densely Connected Deep Residual Network and Channel Attention Module for Mobile Robot Control,” IEEE Transactions on Instrumentation and Measurement, vol. 72, pp. 1–11, 2023.
[47]K. Haratiannejadi and R. R. Selmic, “Smart Glove and Hand Gesture-Based Control Interface for Multi-Rotor Aerial Vehicles in a Multi-Subject Environment,” IEEE Access, vol. 8, pp. 227667–227677, 2020.
[48]H. He and Y. Dan, “The Research and Design of Smart Mobile Robotic Arm Based on Gesture Controlled,” Proc. Int. Conf. Adv. Mechatronic Systems (ICAMechS), pp. 308–312, 2020.
[49]X. Zhao, Y. He, X. Chen, and Z. Liu, “Human–Robot Collaborative Assembly Based on Eye-Hand and a Finite State Machine in a Virtual Environment,” Applied Sciences, vol. 11, no. 12, pp. 5754–5772, 2021.
[50]王信傑，2022，「一個基於RGB-D影像感測之手勢語言辨識系統設計之研究」，國立虎尾科技大學碩士論文。
[51]Y. Lecun, L. Bottou, Y. Bengio and P. Haffner, “Gradient-Based Learning Applied to Document Recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
[52]S. Hochreiter and J. Schmidhuber, “Long Short-Term Memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.
[53]G. D. Forney, “The Viterbi Algorithm,” Proceedings of the IEEE, vol. 61, no. 3, pp. 268–278, 1973.
[54]張育瑞，2016，「一種人體姿態命令辨識及其身分識別的強化式方法之研究」，國立虎尾科技大學碩士論文。