臺灣博碩士論文加值系統

傳統的手勢辨識當中，常以RGB影像來辨別膚色進行切割與擷取，但此種方法很容易受到環境的影響造成誤判，而本文改用使用Realsense Sensor的深度影像(Depth,D)，深度影像是計算紅外線發射出去回來時的時間差，將間誤差轉換成深度值，用此方法將不必考慮環境光影和複雜背景干擾，以減少對實驗場地的各種條件限制，獲得更加準確的擷取結果，此即為Depth影像感測的優勢。然而，RGB影像感測仍然有Depth影像感測最為不足的地方，RGB影像感測具有較豐富的影像色彩特徵參數量，可以做更複雜的影像分析及辨識(如深度學習之手勢辨識等)，因此RGB影像感測仍是現今主流。本論文是建立在基於一個RGB-D影像感測的手勢語言辨識系統，而所謂的RGB-D 影像感測就是RGB 影像與RGB影像對應的Depth影像的組合。其中Depth影像每個像素與影像平面與RGB圖像中對應對象之間的距離有關，也就是像素值表達的是感測器距離物體的實際距離。本論文在做手勢語言辨識系統的開發是先運用RGB-D的Depth感測進行台灣手語識別與手部追蹤演算法，在論文的最後同時運用RGB-D的RGB與Depth影像進行融合雙深度學習通道的手語識別。
在台灣手語識別方面，所發展的基於深度感測的台灣手語識別方法是將設計好的手部深度資料之幾何特徵運用在多分類的動態時間校正(Dynamic Time Warping, DTW)中，所設計的多分類的DTW可以依據所輸入的不同程度之動作時間長度的動作資料而選擇出適合的語意樹進行DTW台灣手語辨識。
在手部追蹤演算法方面，所發展的運用於人機互動的基於深度感測之手的追蹤演算法是基於在進行影像感測時，手部可以移動而不需要如傳統方式之只能在固定區域中進行手勢動作擷取，此擷取方式為一種較為理想的人機互動方式。本論文所設計的演算法能讓有影像感測器的機械手臂追隨使用者的移動手部，並且，感測器進行影像感測時又能保持於水平，上述的人機互動影像感測有別於目前固定式感測器建立的影像感測，本論文將此人機互動設計與手語辨識進行更進一步的整合應用發展。
在融合雙深度學習通道的手語識別方面，所發展的融合靜態手部姿態與動態手部姿勢之基於RGB-D感測深度學習的手語識別方法是左手靜態手部姿態進行卷積神經網路(Convolution Neural Network, CNN)深度學習辨識與右手動態手部姿勢進行長期循環卷積網路(Long-term Recurrent Convolutional Networks, LRCN)深度學習辨識，之後將兩類(CNN與LRCN)的RGB深度學習結果與Depth深度學習結果進行融合決策而最後得到手語辨識結果。
本論文提出的基於深度感測的台灣手語識別，設定一種問答方式的應用情境，在這應用情境具有12種台灣手語動作，此部分手語辨識的平均辨識率為83%。在運用於人機互動的手的追蹤演算法，有依據教育界與業界常用的機械手臂分別設計兩種演算法，在性能評估實驗情境中設計4個不同須放置的位置，運用手的追蹤演算法控制移動機械手臂至正確位置將做為兩種手的追蹤演算法評估標準，而兩種機械手臂經演算法控制正確移至任務位置的成功率均80%以上。在融合雙深度學習通道的手語識別，RGB-LRCN深度學習上機辨識率為25%，Depth-LRCN深度學習上機辨識率為95%，這與大多數文獻所敘述的RGB辨識率優於Depth辨識率不符，而之所以相差之大可能是因為本論文皆採用深度感測的手心追蹤來擷取特徵，使深度學習認為RGB影像重要的手部變化為僅稍微變動的背景，導致學習錯誤的特徵，但經過雙深度學習通道的融合決策方法，LRCN雙通道融合的上機辨識率提升至97%(靜態手部姿態的CNN雙通道融合的上機辨識率亦達99%以上)，實驗結果顯示通道融合決策後的辨識率均有所提升。

In traditional gesture recognition, RGB images are often used to identify skin color for cutting and capturing, but this method is easily affected by the environment and causes misjudgment. In this paper, the depth image (Depth, D) of Realsense Sensor is used instead. Depth image is to calculate the time difference when infrared rays are emitted and return, and convert the error into a depth value. Using this method, it is not necessary to consider ambient light and shadow and complex background interference, to reduce various conditions on the experimental site and obtain more accurate acquisition results. That is the advantage of Depth image sensing. However, RGB image sensing still has many advantages that the Depth image sensing cannot reach. RGB image sensing has richer image color feature parameters and can do more complex image analysis and recognition (such as deep learning gesture recognition, etc.). Therefore, RGB image sensing is still mainstream. This thesis is to develop a gesture language recognition system based on RGB-D image sensing. The so-called RGB-D image sensing is the combination of RGB image and Depth image corresponding to RGB image. Each pixel of the Depth image is related to the distance between the image plane and the corresponding object in the RGB image, that is, the pixel value expresses the actual distance between the sensor and the object. In the development of the gesture language recognition system in this thesis, we first use RGB-D Depth sensing to perform Taiwanese sign language recognition and hand tracking algorithms, and at the end of the thesis, we use RGB-D RGB and Depth image fusion to construct dual deep learning channel sign language recognition.
In terms of Taiwan sign language recognition, the developed Taiwan sign language recognition method based on depth sensing is to apply the designed geometric features in the multi-class dynamic time warping (DTW). According to the input data, the system can select different semantic trees for DTW Taiwan sign language recognition.
In terms of hand tracking algorithm, the developed hand tracking algorithm based on depth sensing for human-computer interaction is based on the fact that during image sensing, the hand can move without capturing gestures in a fixed area. This method is an ideal human-computer interaction method. The algorithm designed in this paper allows the robotic arm with the image sensor to follow the user's moving hand and keeps the sensor horizontal during image sensing. For the image sensing of the sensor, this paper integrates the human-computer interaction design and sign language recognition for application development.
In terms of sign language recognition that integrates dual deep learning channels, the developed RGB-D sensing deep learning-based sign language recognition method that integrates static hand gestures and dynamic hand gestures is a convolutional neural network (CNN) deep learning recognition and right-hand dynamic hand posture for the long-term recurrent convolutional network (LRCN) deep learning recognition, and then two types (CNN and LRCN) RGB deep learning results are fused with depth deep learning results by decision fusion to obtain final sign language recognition results.
The depth-sensing-based Taiwan sign language recognition proposed in this paper sets a question-and-answer application scenario. In this application scenario, there are 12 Taiwan sign language actions, and the average recognition rate of the sign language recognition is 83%. In the hand tracking algorithm used in human-computer interaction, two algorithms are designed based on the robotic arms commonly used in education and the industry. In the performance evaluation experimental situation, 4 different positions to be placed are designed, and the hand tracking algorithm is used. The method of controlling the moving robotic arm to the correct position will be used as the evaluation standard for the tracking algorithm of the two hands, and the success rate of the two robotic arms being controlled by the algorithm to move to the task position correctly is over 80%. In the sign language recognition using combinations of dual deep learning channels, the recognition rate of RGB-LRCN deep learning is 25%, and the recognition rate of Depth-LRCN deep learning is 95%, which is in line with the RGB recognition rate described in most literatures. The recognition rate does not match, and the reason for the big difference is that this paper uses depth-sensing palm tracking to extract features, so that deep learning considers that the important hand of RGB image changes to a slightly changed background, resulting in learning errors. However, through the decision fusion of dual deep learning channels, the recognition rate of LRCN dual-channel fusion is 97% and the recognition rate of CNN dual-channel fusion is also more than 99%, which demonstrates that the decision fusion of dual deep learning channels can improve performances.

摘要...i
Abstract...iii
誌謝...v
目錄...vi
表目錄...viii
圖目錄...xi
第一章 緒論...1
1.1研究動機...1
1.2文獻回顧與探討...2
1.3章節概要...3
第二章 背景技術...4
2.1 Realsense影像感測(RGB-D)...5
2.2 NITE2手部追蹤...6
2.3 雜訊處理...7
2.4 動態時間校正(Dynamic Time Warping,DTW)...8
2.5 卷積神經網路(Convolution Neural Network,CNN)...9
2.6 長短期記憶(Long Short-Term Memory, LSTM)...10
第三章 基於RGB-D影像感測之手勢語言識別...13
3.1基於深度感測的台灣手語識別方法...13
3.1.1 基於NiTE2手心的改善...14
3.1.2 特徵值提取...15
3.1.3 動態連續時間性之幾何特徵參數 16
3.1.4 DTW動態連續時間性手語識別方法...20
3.1.5 多分類的DTW語意決策樹設計...21
3.2 運用於人機互動的基於深度感測之手的追蹤演算法設計...26
3.3 融合靜態手部姿態與動態手部姿勢之基於RGB-D感測深度學習的手語識別方法...37
3.3.1 VGG16...38
3.3.2 長期循環卷積網路(Long-term Recurrent Convolutional Networks, LRCN)...39
3.3.3 雙深度學習通道(RGB通道與DEPTH通道)的融合決策方法...40
3.4 基於RGB-D影像感測的手勢語言識別示範應用...44
第四章 實驗結果與分析...46
4.1 實驗環境設置與台灣手語影像資料庫建置...46
4.1.1 動態連續時間性之台灣手語辨識實驗...52
4.1.2 DTW台灣手語辭彙分類實驗結果...53
4.1.3 DTW台灣手語語意決策實驗結果(特定應用情境)...55
4.2 深度感測之機械手臂追蹤手心演算法實驗結果 56
4.3 實驗環境設置與雙手手勢影像資料庫建置...60
4.3.1 動態連續時間性之手勢影像辨識時實驗設定...66
4.3.2 CNN手勢影像辨識實驗結果...67
4.3.3 LRCN手勢影像辨識實驗結果...69
4.3.4 雙深度學習通道的融合決策實驗結果...71
4.4 RDB-D影像感測的手勢語言應用(特定應用情境)...78
第五章 結論...82
參考文獻...83
Extended Abstract...88

[1]K. -Y. Lian, C. -C. Chiu, Y. -J. Hong and W. -T. Sung, “Wearable armband for real time hand gesture recognition”, Proc. 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 2992-2995, 2017.
[2]Makarov, N. Veldyaykin, M. Chertkov and A. Pokoev, “Russian sign language dactyl recognition”, Proc. 2019 42nd International Conference on Telecommunications and Signal Processing (TSP), pp. 726-729, 2019.
[3]Zhibo Wang, Tengda Zhao, Jinxin Ma, Hongkai Chen, Kaixin Liu, Huajie Shao, Qian Wang and Ju Ren, “Hear sign language: a real-time end-to-end sign language recognition system”, IEEE Transactions on Mobile Computing, 1 July 2022, vol. 21, no. 7, pp. 2398-2410.
[4]T. Du, X. Ren and H. Li, “Gesture recognition method based on deep learning”, Proc. 2018 33rd Youth Academic Annual Conference of Chinese Association of Automation (YAC), pp. 782-787, 2018.
[5]R. Chanu, A. Pillai, S. Sinha and P. Das, “Comparative study for vision based and data based hand gesture recognition technique”, Proc. 2017 International Conference on Intelligent Communication and Computational Techniques (ICCT), pp. 26-31, 2017.
[6]K. N. Lavanya, D. R. Shree, B. R. Nischitha, T. Asha and C. Gururaj, “Gesture controlled robot”, Proc. 2017 International Conference on Electrical, Electronics, Communication, Computer, and Optimization Techniques (ICEECCOT), pp. 465-469, 2017.
[7]Kar and P. S. Chatterjee, “An approach for minimizing the time taken by video processing for translating sign language to simple sentence in english”, Proc. 2015 International Conference on Computational Intelligence and Networks, pp. 172-177, 2015.
[8]Choudhury and K. K. Sarma, “A novel approach for gesture spotting in an assamese gesture-based character recognition system using a unique geometrical feature set”, Proc. 2018 5th International Conference on Signal Processing and Integrated Networks (SPIN), pp. 98-104, 2018.
[9]S. Bhowmick, A. K. Talukdar and K. K. Sarma, “Continuous hand gesture recognition for English alphabets”, Proc. 2015 2nd International Conference on Signal Processing and Integrated Networks (SPIN), pp. 443-446, 2015.
[10]Y. Ji, S. Kim and K. -B. Lee, “Sign language learning system with image sampling and convolutional neural network”, Proc. 2017 First IEEE International Conference on Robotic Computing (IRC), pp. 371-375, 2017.
[11]V. Joseph, A. Talpade, N. Suvarna and Z. Mendonca, “Visual gesture recognition for text writing in air”, Proc. 2018 Second International Conference on Intelligent Computing and Control Systems (ICICCS), pp. 23-26, 2018.
[12]X. Ji, Z. Wang and X. Zhang, “Design of human machine interactive system based on hand gesture recognition”, Proc. 2019 6th International Conference on Information Science and Control Engineering (ICISCE), pp. 250-253, 2019.
[13]J. Hossain Gourob, S. Raxit and A. Hasan, “A robotic hand: controlled with vision based hand gesture recognition system”, Proc. 2021 International Conference on Automation, Control and Mechatronics for Industry 4.0 (ACMI), pp. 1-4, 2021.
[14]S. Song, D. Yan and Y. Xie, “Design of control system based on hand gesture recognition”, Proc. 2018 IEEE 15th International Conference on Networking, Sensing and Control (ICNSC), pp. 1-4, 2018.
[15]J. Sanchez-Riera, K. Srinivasan, K. -L. Hua, W. -H. Cheng, M. A. Hossain and M. F. Alhamid, “Robust RGB-D hand tracking using deep learning priors”, IEEE Transactions on Circuits and Systems for Video Technology, Sept. 2018, vol. 28, no. 9, pp. 2289-2301.
[16]F. Yang and H. Shi, “Research on static hand gesture recognition technology for human computer interaction system”, Proc. 2016 International Conference on Intelligent Transportation, Big Data & Smart City (ICITBS), pp. 459-463, 2016.
[17]Y. Yao and C. -T. Li, “Hand gesture recognition and spotting in uncontrolled environments based on classifier weighting”, Proc. 2015 IEEE International Conference on Image Processing (ICIP), pp. 3082-3086, 2015.
[18]Xinwu Liang, Hesheng Wang, Yun-Hui Liu, Weidong Chen and Jie Zhao, 2015, “A unified design method for adaptive visual tracking control of robots with eye-in-hand/fixed camera configuration”, Automatica, 2015, vol. 59, pp. 97-105.
[19]Y. Wang, B. Zhang and C. Peng, “SRHandNet: real-time 2D hand pose estimation with simultaneous region localization”, IEEE Transactions on Image Processing, 2020, vol. 29, pp. 2977-2986.
[20]F. Wang, L. Zhou, Z. Cui, H. Li and M. Li, “Gesture recognition based on BoF and its application in human-machine interaction of service robot”, 2016 IEEE International Conference on Cyber Technology in Automation, Control, and Intelligent Systems (CYBER), pp. 115-120, 2016.
[21]Chen F, Zhong Q, Cannella F, Sekiyama K and Fukuda T, “Hand gesture modeling and recognition for human and robot interactive assembly using hidden markov models”, International Journal of Advanced Robotic Systems, 2015.
[22]H. He and Y. Dan, “The research and design of smart mobile robotic arm based on gesture controlled”, Proc. 2020 International Conference on Advanced Mechatronic Systems (ICAMechS), pp. 308-312, 2020.
[23]H. R. Vanamala, A. S. M, V. A, S. Kumar and M. Rathod, “Gesture and voice controlled robot for industrial applications”, Proc. 2022 International Conference for Advancement in Technology (ICONAT), pp. 1-8, 2022.
[24]D. Zhi, T. E. A. de Oliveira, V. P. d. Fonseca and E. M. Petriu, “Teaching a robot sign language using vision-based hand gesture recognition”, Proc. 2018 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA), pp. 1-6, 2018.
[25]P. Qian and D. Zhang, “Design and implementation of human computer interactive gesture recognition system”, Proc. 2021 IEEE International Conference on Advances in Electrical Engineering and Computer Applications (AEECA), pp. 179-182, 2021.
[26]S. Nitnaware and A. Bagde, “Hand gesture recognition to facilitate tasks for the disabled”, Proc. 2022 Second International Conference on Artificial Intelligence and Smart Energy (ICAIS), pp. 949-953, 2022.
[27]S. Nikam and A. G. Ambekar, “Sign language recognition using image based hand gesture recognition techniques”, Proc. 2016 Online International Conference on Green Engineering and Technologies (IC-GET), pp. 1-5, 2016.
[28]Q. Yang, W. Ding and X. Zhou, 2020, D. Zhao and S. Yan, “Leap motion hand gesture recognition based on deep neural network”, Proc. 2020 Chinese Control And Decision Conference (CCDC), pp. 2089-2093.
[29]K. H. M. Cheng and A. Kumar, “Deep feature collaboration for challenging 3D finger knuckle identification”, IEEE Transactions on Information Forensics and Security, 2021, vol. 16, pp. 1158-1173.
[30]Chan-Su Lee, SungYong Chun and Shin Won Park, “Tracking hand rotation and various grasping gestures from an IR camera using extended cylindrical manifold embedding”, Computer Vision and Image Understanding, 2013, vol. 117, Issue 12, pp. 1711-1723.
[31]Yuanjie Wu, Yu Wang, Sungchul Jung, Simon Hoermann and Robert W. Lindeman, “Towards an articulated avatar in VR: Improving body and hand tracking using only depth cameras”, Entertainment Computing, 2019, vol 31, August, 100303.
[32]Liao, J. Li, Z. Ju and G. Ouyang, “Hand Gesture recognition with generalized hough transform and DC-CNN using realsense”, Proc. 2018 Eighth International Conference on Information Science and Technology (ICIST), pp. 84-90, 2018.
[33]J. -H. Sun, T. -T. Ji, S. -B. Zhang, J. -K. Yang and G. -R. Ji, “Research on the hand gesture recognition based on deep learning”, Proc. 2018 12th International Symposium on Antennas, Propagation and EM Theory (ISAPE), pp. 1-4, 2018.
[34]S. Liang, L. Wang, L. Zhang and Y. Wu, “Research on recognition of nine kinds of fine gestures based on adaptive adaboost algorithm and multi-feature combination”, IEEE Access, 2019, vol. 7, pp. 3235-3246.
[35]Gong, C. Chen and M. Peng, “Human interaction recognition based on deep learning and HMM”, IEEE Access, 2019, vol. 7, pp. 161123-161130.
[36]S. Tornay, M. Razavi and M. Magimai.-Doss, “Towards multilingual sign language recognition”, Proc. ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6309-6313, 2020.
[37]Di Wu, Lionel Pigou, Pieter-Jan Kindermans, Nam Do-Hoang Le, Ling Shao, Joni Dambre and Jean-Marc Odobez, “Deep dynamic neural networks for multimodal gesture segmentation and recognition”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 1 Aug. 2016, vol. 38, no. 8, pp. 1583-1597.
[38]S. Rakesh, G. Kovács, H. Mokayed, R. Saini and U. Pal, “Static palm sign gesture recognition with leap motion and genetic algorithm”, Proc. 2021 Swedish Artificial Intelligence Society Workshop (SAIS), pp. 1-5, 2021.
[39]N. H. Dardas and N. D. Georganas, “Real-Time hand gesture detection and recognition using bag-of-features and support vector machine techniques”, IEEE Transactions on Instrumentation and Measurement, 2011, vol. 60, no. 11, pp. 3592-3607.
[40]Rodrigo Ibañez, Álvaro Soria, Alfredo Teyseyre and Marcelo Campo, “Easy gesture recognition for kinect”, Advances in Engineering Software, 2014, vol. 76, pp. 171-180.
[41]S. M. Aslam and S. Samreen, “Gesture recognition algorithm for visually blind touch interaction optimization using crow search method”, IEEE Access, 2020, vol. 8, pp. 127560-127568.
[42]Fang Liang, Chunhua Cai, Kun Zhang, Lili Zhang, Junda Li, Hengchang Bi, Peisong Wu, He Zhu, Chaolun Wang, Hailu Wang, Zuoyuan Dong, Chen Luo, Zewei Luo, Chongxin Shan, Weida Hu and Xing Wu, “Infrared gesture recognition system based on near-sensor computing”, IEEE Electron Device Letters, July 2021, vol. 42, no. 7, pp. 1053-1056.
[43]Papastratis, K. Dimitropoulos, D. Konstantinidis and P. Daras, “Continuous sign language recognition through cross-modal alignment of video and text embeddings in a joint-latent space”, IEEE Access, 2020, vol. 8, pp. 91170-91180.
[44]H. Long, M. Liu and M. Li, “Real-time gesture recognition in complex background based on deep learning”, Proc. 2020 IEEE 9th Joint International Information Technology and Artificial Intelligence Conference (ITAIC), pp. 2137-2140, 2020.
[45]F. Patrona, I. Mademlis and I. Pitas, “Self-Supervised convolutional neural networks for fast gesture recognition in Human-Robot interaction”, Proc. 2021 10th International Conference on Information and Automation for Sustainability (ICIAfS), pp. 88-93, 2021.
[46]J. Cheng and Z. Li, “Gesture recognition for human-computer interaction based on CNN model”, Proc. 2021 International Conference on Intelligent Computing, Automation and Applications (ICAA), pp. 241-244, 2021.
[47]Haojie Zeng, Bin Wang, Xiaoping Zhou, Xiaojing Sun, Longxiang Huang, Qian Zhang and Yang Wang, “TSFE-Net: two-stream feature extraction networks for active stereo matching”, IEEE Access, 2021, vol. 9, pp. 33954-33962.
[48]M. Zheng, Y. Tie, L. Qi and S. Jiang, “Dynamic gesture recognition based on the multimodality fusion temporal segment networks”, Proc. 2019 8th International Symposium on Next Generation Electronics (ISNE), pp. 1-3, 2019.
[49]S. He, “Research of a sign language translation system based on deep learning”, Proc. 2019 International Conference on Artificial Intelligence and Advanced Manufacturing (AIAM), pp. 392-396, 2019.
[50]W. Pan, X. Zhang and Z. Ye, “Attention-based sign language recognition network utilizing keyframe sampling and skeletal features”, IEEE Access, 2020, vol. 8, pp. 215592-215602.
[51]C. -L. Huang and B. -L. Tsai, “A vision-based taiwanese sign language recognition”, Proc. 2010 20th International Conference on Pattern Recognition, pp. 3683-3686, 2010.
[52]Y. Yan, Z. Li, Q. Tao, C. Liu and R. Zhang, “Research on dynamic sign language algorithm based on sign language trajectory and key frame extraction”, Proc. 2019 IEEE 2nd International Conference on Electronics Technology (ICET), pp. 509-514, 2019.
[53]X. Zhang and X. Wu, “Robotic control of dynamic and static gesture recognition”, Proc. 2019 2nd World Conference on Mechanical Engineering and Intelligent Manufacturing (WCMEIM), pp. 474-478, 2019.
[54]Jacob, M. Koshy and K. K. Nisha, “Real time static and dynamic hand gestures cognizance for human computer interaction”, Proc. 2021 International Conference on Advances in Computing and Communications (ICACC), pp. 1-6, 2021.
[55]OpenNI2, “OpenNI2 UserGuide”, online available at: http:openni.ru/index.html
[56]蘇秀芬，“台灣手語的呼應方式”，百川匯海：李壬癸先生七秩壽慶論文集，頁341-363，2016 8月。