臺灣博碩士論文加值系統

本論文發展Face Identification and Gesture Answering Recognition for Interactive Response System ，簡稱 FIGARIRS。此系統可辨識學生臉部與手勢，應用於即時反饋系統。首先，將教室拍攝影片輸入FIGARIRS，進行每一幀影像的人形、臉形、手形偵測，接著，進行辨識臉部與手勢，最後輸出學生學號及手勢答題選項(1-5)。同時，在答題過程，FIGARIRS會動態地顯示即時答題分佈長條圖。此系統主要使用You only look once (YOLO)深度學習模型進行偵測人形、臉形與手形，接著使用Convolutional Neural Network (CNN)深度學習模型辨識臉部與手勢答題選項， 並以WebSocket及Chart.js呈現即時答題分佈長條圖。由實驗結果顯示三個班級的平均準確率為91%。此系統的確可以用於傳統教室內學生手勢答題活動，可以解決傳統IRS系統可能遭遇問題，例如:學生忘記帶手機、網路環境較差、手機軟體操作不熟、手機電力不足等。

This paper proposes Face Identification and Gesture Answering Recognition (FIGAR) technique which can recognize students’ faces and gestures simutaneously. It can be applied to construct an Interactive Response System (IRS). Here the system is called FIGARIRS. First, the video is inputted into FIGARIRS which detects persons, faces and hands in each frame. Then, it recognizes face and gesture for each person. FIGARIRS outputs student’ identifiction number and corresponding gesture with number (1-5). Additionally, another feature of the system is to displayreal-time answers histogram dynamically while students are answering questions with gestures. You only look once (YOLO), a deep learning module, is utilized in the system for detection of persons, faces and hands. Convolutional Neural Network (CNN) is employed to recognize faces and gestures to get student identification numbers and answer choices with number, respectively. WebSocket and Chart.js are involved in the process of displaying real-time answer histogram dynamically. The experiment results show that the average accuracy of face and guestue recognition reaches 91%. Thus, FIGARIRS can be in integrated in the IRS to reduce its construction cost in traditional classroom. Moreover, it can solve the problems traditional IRS, requring ansering questions with mobile phones, may encounter such as forgetting to carry mobile phones, unskilled APP operations, insufficient power supply of mobile phones, or poor quality of network environments.

摘要....................................................i
Abstract....................................................iii
誌謝....................................................v
目錄....................................................vi
表目錄....................................................vii
圖目錄....................................................viii
符號說明....................................................x
第一章 研究動機目的....................................................1
1.1 研究動機....................................................1
1.2 研究目的 ....................................................4
1.3 研究流程 ....................................................5
第二章 相關文獻回顧....................................................6
2.1 臉形偵測與臉部辨識相關文獻.............................................6
2.2 IRS相關文獻.............................................8
2.3 手形偵測與手勢辨識相關文獻.............................................8
2.4 WebSocket.............................................9
2.5 Chart.js.............................................10
2.6 Backpropagation.............................................10
2.7 深度學習工具.............................................13
2.8 深度學習物件偵測模型.............................................16
2.9 深度學習物件辨識模型.............................................21
2.10 深度學習模型效能評比指標.............................................25
第三章 偵測與辨識技術說明.............................................32
3.1 影像前處理.............................................32
3.2 基於深度學習模型設計人形、臉形與手形物件偵測技術.......................41
3.3 基於深度學習模型設計臉部與手勢辨識技術.......................53
第四章 實驗結果.............................................61
4.1 系統需求.............................................61
4.2 影像資料庫.............................................62
4.3 YOLO物件偵測深度模型實驗結果........................................65
4.4 CNN物件辨識深度學習模型實驗結果.......................................71
第五章 IRS系統建置.............................................84
5.1 即時反饋系統建置(資料庫).............................................84
5.2 呈現答題結果.............................................86
第六章 結論與未來研究.............................................92
6.1 結論.............................................92
6.2 未來研究.............................................92
參考文獻.............................................94

[1] Cheng, K.-H., & Tsai, C.-C. (2019). A case study of immersive virtual field trips in an elementary classroom: Students’ learning experience and teacher-student interaction behaviors. Computers & Education, 140, 103600. doi: 10.1016/j.compedu.2019.103600.
[2] Chen, Y.-C., Hwang, R.-H., & Wang, C.-Y. (2012). Development and evaluation of a Web 2.0 annotation system as a learning tool in an e-learning environment. Computers & Education, 58(4), 1094–1105. doi: 10.1016/j.compedu.2011.12.017.
[3] Hwang, G.-J., Sung, H.-Y., & Yen, Y.-F. (2014). Development of a Contextual Decision-Making Game for Improving Students Learning Performance in a Health Education Course. 2014 International Conference of Educational Innovation through Technology. doi: 10.1109/eitt.2014.17.
[4] Parsazadeh, N., Ali, R., & Rezaei, M. (2018). A framework for cooperative and interactive mobile learning to improve online information evaluation skills. Computers & Education, 120, 75–89. doi: 10.1016/j.compedu.2018.01.010.
[5] López-Faican, L., & Jaen, J. (2020). EmoFindAR: Evaluation of a mobile multiplayer augmented reality game for primary school children. Computers & Education, 149, 103814. doi: 10.1016/j.compedu.2020.103814.
[6] Abeysekera, L., & Dawson, P. (2014). Motivation and cognitive load in the flipped classroom: definition, rationale and a call for research. Higher Education Research & Development, 34(1), 1–14. doi: 10.1080/07294360.2014.934336.
[7] Lai, C.-L., & Hwang, G.-J. (2016). A self-regulated flipped classroom approach to improving students’ learning performance in a mathematics course. Computers & Education, 100, 126–140. doi: 10.1016/j.compedu.2016.05.006.
[8] Chiou, C.-C., Tien, L.-C., & Tang, Y.-C. (2020). Applying structured computer-assisted collaborative concept mapping to flipped classroom for hospitality accounting. Journal of Hospitality, Leisure, Sport & Tourism Education, 26, 100243. doi: 10.1016/j.jhlste.2020.100243.
[9] IRS即時反饋系統-網奕資訊. https://www.habook.com.tw/eTeaching/products.aspx?BookNo=IRS_01.
[10] Wang, A. I. (2015). The wear out effect of a game-based student response system. Computers & Education, 82, 217–227. doi: 10.1016/j.compedu.2014.11.004.
[11] Zuvio-大學課堂教學互動工具.(2013). https://www.zuvio.com.tw/.
[12] Tsai, H.-H., Chang, C.-T., Hou, X.-Y., Yong, Y.-M., Chiou, K.-C., & Yu, P.-T. (2019). Interactive student response system with iBeacon and web-socket for flipped classroom learning. Journal of Computing in Higher Education, 31(2), 340–361. doi: 10.1007/s12528-019-09226-x.
[13] Tsai, H.-H., & Chang, Y.-C. (2017). Facial expression recognition using a combination of multiple facial features and support vector machine. Soft Computing, 22(13), 4389–4405. doi: 10.1007/s00500-017-2634-3.
[14] Galarza, Eddie E., et al. “Real Time Driver Drowsiness Detection Based on Driver’s Face Image Behavior Using a System of Human Computer Interaction Implemented in a Smartphone.” Proceedings of the International Conference on Information Technology & Systems (ICITS 2018) Advances in Intelligent Systems and Computing, 2018, pp. 563–572., doi:10.1007/978-3-319-73450-7_53.
[15] Faria, D. R., Vieira, M., Faria, F. C., & Premebida, C. (2017). Affective facial expressions recognition for human-robot interaction. 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). doi: 10.1109/roman.2017.8172395.
[16] Chang, W.-J., Schmelzer, M., Kopp, F., Hsu, C.-H., Su, J.-P., Chen, L.-B., & Chen, M.-C. (2019). A Deep Learning Facial Expression Recognition based Scoring System for Restaurants. 2019 International Conference on Artificial Intelligence in Information and Communication (ICAIIC). doi: 10.1109/icaiic.2019.8668998.
[17] Sajjad, M., Nasir, M., Ullah, F. U. M., Muhammad, K., Sangaiah, A. K., & Baik, S. W. (2019). Raspberry Pi assisted facial expression recognition framework for smart security in law-enforcement services. Information Sciences, 479, 416–431. doi: 10.1016/j.ins.2018.07.027.
[18] Sebastian, A. G., Singh, S., Manikanta, P. B. T., Ashwin, T. S., & Reddy, G. R. M. (2018). Multimodal Group Activity State Detection for Classroom Response System Using Convolutional Neural Networks. Advances in Intelligent Systems and Computing Recent Findings in Intelligent Computing Techniques, 245–251. doi: 10.1007/978-981-10-8639-7_25.
[19] Neto, L. D. S. B., Maike, V. R. M. L., Koch, F. L., Baranauskas, M. C. C., Rocha, A. D. R., & Goldenstein, S. K. (2015). A Wearable Face Recognition System Built into a Smartwatch and the Visually Impaired User. Proceedings of the 17th International Conference on Enterprise Information Systems. doi: 10.5220/0005370200050012.
[20] Sajjad, M., Nasir, M., Muhammad, K., Khan, S., Jan, Z., Sangaiah, A. K., … Baik, S. W. (2017). Raspberry Pi assisted face recognition framework for enhanced law-enforcement services in smart cities. Future Generation Computer Systems. doi: 10.1016/j.future.2017.11.013.
[21] Liu, X., Geng, J., Ling, H., & Cheung, Y.-M. (2019). Attention guided deep audio-face fusion for efficient speaker naming. Pattern Recognition, 88, 557–568. doi: 10.1016/j.patcog.2018.12.011.
[22] Surekha, B., Nazare, K. J., Raju, S. V., & Dey, N. (2016). Attendance Recording System Using Partial Face Recognition Algorithm. Intelligent Techniques in Signal Processing for Multimedia Security Studies in Computational Intelligence, 293–319. doi: 10.1007/978-3-319-44790-2_14.
[23] Son, N. T., Chi, L. P., Lam, P. T., & Dinh, T. V. (2019). Combination of Facial Recognition and Interaction with Academic Portal in Automatic Attendance System. Proceedings of the 2019 8th International Conference on Software and Computer Applications - ICSCA 19. doi: 10.1145/3316615.3316638.
[24] Panditpautra, V., Goswami, A., Aishwarya, K., & Ambadekar, S. (2019). Biometric Attendance Management System Using Raspberry Pi. SSRN Electronic Journal. doi: 10.2139/ssrn.3368163.
[25] Lee, S.-H., Sohn, M.-K., Kim, D.-J., Kim, B., & Kim, H. (2013). Smart TV interaction system using face and hand gesture recognition. 2013 IEEE International Conference on Consumer Electronics (ICCE). doi: 10.1109/icce.2013.6486845.
[26] Rautaray, S. S., & Agrawal, A. (2011). Interaction with virtual game through hand gesture recognition. 2011 International Conference on Multimedia, Signal Processing and Communication Technologies. doi: 10.1109/mspct.2011.6150485.
[27] Radkowski, R. & Stritzke, C.. (2012). Interactive hand gesture-based assembly for augmented reality applications. ACHI 2012 - 5th International Conference on Advances in Computer-Human Interactions. 303-308.
[28] Bergh, M. V. D., Koller-Meier, E., Bosche, F., & Gool, L. V. (2009). Haarlet-based hand gesture recognition for 3D interaction. 2009 Workshop on Applications of Computer Vision (WACV). doi: 10.1109/wacv.2009.5403103.
[29] Bellarbi, A., Benbelkacem, S., Zenati-Henda, N., & Belhocine, M. (2011). Hand gesture interaction using color-based method for tabletop interfaces. 2011 IEEE 7th International Symposium on Intelligent Signal Processing. doi: 10.1109/wisp.2011.6051717.
[30] Haria, A., Subramanian, A., Asokkumar, N., Poddar, S., & Nayak, J. S. (2017). Hand Gesture Recognition for Human Computer Interaction. Procedia Computer Science, 115, 367–374. doi: 10.1016/j.procs.2017.09.092.
[31] Canal, G., Escalera, S., & Angulo, C. (2016). A real-time Human-Robot Interaction system based on gestures for assistive scenarios. Computer Vision and Image Understanding, 149, 65–77. doi: 10.1016/j.cviu.2016.03.004.
[32] Si, J., Lin, J., Jiang, F., & Shen, R. (2019). Hand-raising gesture detection in real classrooms using improved R-FCN. Neurocomputing, 359, 69–76. doi: 10.1016/j.neucom.2019.05.031.
[33] Introduction to WebSockets : https://linode.com/docs/development/introduction-to-websockets/.
[34] WebSocket通訊協定簡介 : https://blog.gtwang.org/web-development/websocket-protocol/.
[35] Chart.js : https://www.chartjs.org/.
[36]李宏毅 — Backpropagation :
https://sakura-gh.github.io/ML-notes/ML-notes-html/9_Backpropagation.html.
[37] Tensorflow Github : https://github.com/tensorflow/tensorflow.
[38] Tensorflow : https://www.tensorflow.org/.
[39] TensorFlow基本使用與分散式概念 : https://k2r2bai.com/2017/04/10/tensorflow/introduction/.
[40] Tensorflow Lite : https://www.tensorflow.org/lite.
[41] Tensorflow.js : https://www.tensorflow.org/js.
[42] Keras Github : https://github.com/keras-team/keras.
[43] OpenCv : https://opencv.org/.
[44] Imgaug Github : https://github.com/aleju/imgaug.
[45] LabelImg : https://github.com/tzutalin/labelImg.
[46] 深度學習系列: 什麼是AP/mAP : https://medium.com/@chih.sheng.huang821/%E6%B7%B1%E5%BA%A6%E5%AD%B8%E7%BF%92%E7%B3%BB%E5%88%97-%E4%BB%80%E9%BA%BC%E6%98%AFap-map-aaf089920848.
[47] 機器/深度學習 : 物件偵測 Non-Maximum Suppression (NMS) :
https://medium.com/@chih.sheng.huang821/%E6%A9%9F%E5%99%A8-%E6%B7%B1%E5%BA%A6%E5%AD%B8%E7%BF%92-%E7%89%A9%E4%BB%B6%E5%81%B5%E6%B8%AC-non-maximum-suppression-nms-aa70c45adffa.
[48] Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You Only Look Once: Unified, Real-Time Object Detection. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). doi: 10.1109/cvpr.2016.91.
[49] 深度學習-物件偵測:You Only Look Once (YOLO) : https://medium.com/@chih.sheng.huang821/%E6%B7%B1%E5%BA%A6%E5%AD%B8%E7%BF%92-%E7%89%A9%E4%BB%B6%E5%81%B5%E6%B8%AC-you-only-look-once-yolo-4fb9cf49453c.
[50] Real-time Object Detection with YOLO, YOLOv2 and now YOLOv3 :
https://medium.com/@jonathan_hui/real-time-object-detection-with-yolo-yolov2-28b1b93e2088.
[51] Redmon, J., & Farhadi, A. (2017). YOLO9000: Better, Faster, Stronger. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). doi: 10.1109/cvpr.2017.690.
[52] Redmon, Joseph & Farhadi, Ali. (2018). YOLOv3: An Incremental Improvement.
[53] Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324. doi: 10.1109/5.726791.
[54] 卷積神經網路的運作原理 : https://brohrer.mcknote.com/zh-Hant/how_machine_learning_works/how_convolutional_neural_networks_work.html.
[55] 入門深度學習 — 2 :
https://medium.com/@syshen/%E5%85%A5%E9%96%80%E6%B7%B1%E5%BA%A6%E5%AD%B8%E7%BF%92-2-d694cad7d1e5.
[56] 卷積神經網絡介紹 : https://medium.com/jameslearningnote/%E8%B3%87%E6%96%99%E5%88%86%E6%9E%90-%E6%A9%9F%E5%99%A8%E5%AD%B8%E7%BF%92-%E7%AC%AC5-1%E8%AC%9B-%E5%8D%B7%E7%A9%8D%E7%A5%9E%E7%B6%93%E7%B6%B2%E7%B5%A1%E4%BB%8B%E7%B4%B9-convolutional-neural-network-4f8249d65d4f.
[57] 機器學習\統計方法: 模型評估-驗證指標(validation index) : https://medium.com/@chih.sheng.huang821/%E6%A9%9F%E5%99%A8%E5%AD%B8%E7%BF%92-%E7%B5%B1%E8%A8%88%E6%96%B9%E6%B3%95-%E6%A8%A1%E5%9E%8B%E8%A9%95%E4%BC%B0-%E9%A9%97%E8%AD%89%E6%8C%87%E6%A8%99-b03825ff0814.
[58] Oxford Hand Dataset : http://www.robots.ox.ac.uk/~vgg/data/hands/.
[59] VIVA Hand Detection Dataset : http://cvrr.ucsd.edu/vivachallenge/index.php/hands/hand-detection/.
[60] YOLO object detection with OpenCV.
https://www.pyimagesearch.com/2018/11/12/yolo-object-detection-with-opencv/.
[61] OpenCV Face Recognition.
https://www.pyimagesearch.com/2018/09/24/opencv-face-recognition/.