臺灣博碩士論文加值系統

工作流程是由事理邏輯與任務情境同步搭配且相互支援而定義，使藉由執行流程各步驟來完美完成工作任務。然而於傳統生產環境之服務流程中，經常面對現場人員因經驗或知識不足而需專人臨場指導之情形，故流程的機能應該更緊密地與工作者及任務銜接。為了提升效率及降低錯誤率，本研究提出一套融合視覺畫面、語言交談與工作知識架構之整合智慧教學系統於服務工作流程中。藉由擴增實境(Augmented Reality, AR)技術、AR頭戴式顯示器及聊天機器人(Chatbot)所組成，以物件辨識技術來識別視線範圍內的零組件，並以視覺方式呈現組裝步驟，過程中尚可藉由語音向系統發問，系統並以動畫或文字方式回饋使用者，達到相當於以專人臨場指導並協助解決問題之機能。研究實驗顯示，使用此視語輔助互動介面的操作者可以更快速地認知工作並解決問題，大幅解決了經驗與知識不足的人員狀況。

A workflow is defined by incorporating knowledge logics and task scenario, and is accomplished by fulfilling the steps and segments along the path. A Standard Operation Procedure (SOP) may desire a sound workflow structure which is both efficient and reliable. Inherently, a service may encounter unfamiliar situations that cannot be solved without consulting, but such timely assistance may not always available at hand. In this research, an AR-based service workflow is created with visual and verbal interaction capability. By wearing the AR helmet, a worker may request instant assistance or instructions, by using gestures or verbal expressions, from the work intelligent system. The system will reply with voice or intuitive graphic animations to strengthen the cognition for solving the problem. It is believed that such interaction is more intuitive and adaptable, and may prevail in the scenario of man-machine collaboration for intelligent manufacturing.

摘要...........i
Abstract.......ii
誌謝...........iii
目錄...........iv
表目錄.........vii
圖目錄.........viii
第1章 緒論......1
1.1研究背景......1
1.2研究動機......1
1.3研究目的......2
1.4研究限制......3
1.5章節描述......3
第2章 文獻探討...4
2.1擴增實境定義...4
2.1.1 Microsoft Hololens...5
2.1.2 Meta2......7
2.2物件辨識......8
2.3自然語言處理...12
2.3.1 BERT.......13
2.3.2 Slot-Filling And Intent...14
2.3.3 Encoder-Decoder...15
2.3.4 Seq2Seq...15
2.3.5 ImageCaption...17
2.3.6 VQA.......18
2.4 AR相關應用...19
2.5 文獻總結......22
第3章 研究方法......23
3.1方法概述......23
3.2模型建構......24
3.3 目標檢測.....26
3.4 語音識別.....32
3.5 系統回饋.....35
3.5.1 會員系統...35
3.5.2聊天機器人...36
3.5.3即時狀態顯示...43
3.5.4系統知識庫...44
3.6 歷程記錄......44
第4章 研究實作....45
4.1 實作介紹......46
4.2 影像識別......50
4.3 語音辨識......51
4.4 會員系統......52
4.5 自然語言處理API...52
4.6 即時狀態顯示...56
4.7 歷程記錄......57
4.8實作探討......57
第5章 結論.......59
5.1研究貢獻......59
5.2未來展望......60
參考文獻.........61
Extended Abstract...64

Azuma, R. T. (1997), “A survey of augmented reality”, Teleoperators Virtual Environments, Vol. 6, No. 4, pp. 355-385.
Bimber, O. & Raskar, R. (2005), “Spatial augmented reality: merging real and virtual worlds”, CRC press.
Bochkovskiy, A., Wang, C.-Y. & Liao, H.-Y. M. (2020), “Yolov4: Optimal speed and accuracy of object detection”, arXiv e-prints, pp. arXiv:2004.10934.
Calonder, M., Lepetit, V., Strecha, C. & Fua, P. (2010), “Brief: Binary robust independent elementary features”. Paper presented at the European conference on computer vision, pp. 778-792.
Chen, Q., Zhuo, Z. & Wang, W. (2019), “Bert for joint intent classification and slot filling”, arXiv e-prints, pp. arXiv:1902.10909.
Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. (2018), “Bert: Pre-training of deep bidirectional transformers for language understanding”. Paper presented at the Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1, pp 4171-4186.
Faccio, M., Ferrari, E., Galizia, F. G., Gamberi, M. & Pilati, F. (2019), “Real-time assistance to manual assembly through depth camera and visual feedback”, Procedia CIRP, Vol. 81, pp. 1254-1259.
Fiorentino, M., Uva, A. E., Gattullo, M., Debernardis, S. & Monno, G. (2014), “Augmented reality on large screen for interactive maintenance instructions”, Computers in Industry, Vol. 65, No. 2, pp. 270-278.
Hochreiter, S. & Schmidhuber, J. (1997), “Long short-term memory”, Neural computation, Vol. 9, No. 8, pp. 1735-1780.
Lai, Z.-H., Tao, W., Leu, M. C. & Yin, Z. (2020), “Smart augmented reality instructional system for mechanical assembly towards worker-centered intelligent manufacturing”, Journal of Manufacturing Systems, Vol. 55, pp. 69-81.
Lowe, D. G. (2004), “Distinctive image features from scale-invariant keypoints”, International journal of computer vision, Vol. 60, No. 2, pp. 91-110.
Malinowski, M. & Fritz, M. (2014), “A multi-world approach to question answering about real-world scenes based on uncertain input”. Paper presented at the Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, pp 1682-1690.
Milgram, P., Takemura, H., Utsumi, A. & Kishino, F. (1994), “Augmented reality: A class of displays on the reality-virtuality continuum ”, International Society for Optics and Photonics, Vol. 2351, pp. 282-292.
Prabhavalkar, R., Rao, K., Sainath, T. N., Li, B., Johnson, L. & Jaitly, N. (2017), “A Comparison of Sequence-to-Sequence Models for Speech Recognition”. Paper presented at the Interspeech, pp. 939-943.
Rosten, E. & Drummond, T. (2006), “Machine learning for high-speed corner detection”. Paper presented at the European conference on computer vision, pp. 430-443.
Rublee, E., Rabaud, V., Konolige, K. & Bradski, G. (2011), “ORB: An efficient alternative to SIFT or SURF”. Paper presented at the 2011 International conference on computer vision, pp. 2564-2571.
Shao, C. C., Liu, T., Lai, Y., Tseng, Y. & Tsai, S. (2018), “Drcd: a chinese machine reading comprehension dataset”, arXiv e-prints, pp. arXiv:1806.00920.
Sutskever, I., Vinyals, O. & Le, Q. V. (2014), “Sequence to sequence learning with neural networks”. Paper presented at the Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, pp. 3104-3112.
Vinyals, O., Toshev, A., Bengio, S. & Erhan, D. (2015). “Show and tell: A neural image caption generator”. Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3156-3164
Wu, Q., Wang, P., Shen, C., Dick, A. & Van Den Hengel, A. (2016a), “Ask me anything: Free-form visual question answering based on knowledge from external sources”. Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4622-4630.
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W. & Macherey, K. (2016b), “Google's neural machine translation system: Bridging the gap between human and machine translation”, arXiv e-prints, pp. arXiv:1609.08144.
Xiang, Y., Schmidt, T., Narayanan, V. & Fox, D. (2017), “Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes”, arXiv e-prints, pp. arXiv:1711.00199.
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhudinov, R. & Bengio, Y. (2015), “Show, attend and tell: Neural image caption generation with visual attention”. Paper presented at the International conference on machine learning, pp. 2048-2057.
Zheng, L., Liu, X., An, Z., Li, S. & Zhang, R. (2020), “A smart assistance system for cable assembly by combining wearable augmented reality with portable visual inspection”, Virtual Reality Intelligent Hardware, Vol. 2, No. 1, pp. 12-27.