臺灣博碩士論文加值系統

接受培訓的現場工作人員通常需要幫助才能定位、識別和熟悉不同
的機器和設備。此外，在沒有通過通信媒介提供交互式幫助的情況下，工
人應嚴格遵守多種工業機器的說明手冊。每當需要對工人進行進一步指導
時，專家都應使用視覺和聽覺手段提供幫助。因此，本研究旨在採用智能
語音助手系統，通過增強現實 (AR) 和自然語言處理 (NLP) 的集成來指導
現場工作人員，以實現一種交互式方法，其中使用指導系統來協助執行任
務。

Microsoft HoloLens 2 被用作頭戴式顯示器 (HMD)，以覆蓋必要
的 AR 視覺效果和語音合成以提供用戶幫助。多項式樸素貝葉斯算法在整
個過程中的不同順序運行時間被實施和重複使用以進行分類。使用輕量級
分類方法，無論何時說出描述機器或機械零件的獨特特徵，都將保持訓練
速度，同時保持高精度。為了通過使用交互式現場指導系統為工人提出解
決方案，本研究可以進一步發展，增強用戶界面、交互和整體體驗。

Field workers often need assistance locating, recognizing, and familiarizing
themselves with different machines and equipment. Moreover, workers are
expected to strictly follow the instructional manuals of several industrial machines
without the presence of an interactive assistance through a medium of
communication. Whenever further guidance among workers is required, experts
are expected to assist using both visual and auditory means. Thus, this research
aims to employ an intelligent voice assistant system that guides onsite workers
through the integration of Augmented Reality (AR) and Natural Language
Processing (NLP) to achieve an interactive approach where a guidance system is
used to assist in conducting tasks.
Microsoft HoloLens 2 was used as the Head Mounted Display (HMDs) to
overlay necessary AR visuals and speech synthesis for user assistance. The
multinomial Naive Bayes algorithm was implemented and used repeatedly at
different sequential run times throughout processes for classifications to be made.
With the lightweight classification method, training speeds will be preserved
while a high accuracy will be maintained whenever a unique feature describing a
machine or mechanical part is uttered. With the objective to propose a solution
for workers by using an interactive onsite guidance system, further developments
can be made in this research enhancing the user interface, interaction, and overall
experience.

Abstract .... i
摘要... ii
Acknowledgment ... iii
Table of Contents... iv
List of Figures ... vii
List of Tables... viii
Chapter 1 Introduction ... 1
1.1 Research Background... 1
1.2 Problem Statement... 2
1.3 Purpose of Study... 3
1.4 Research Limitations... 3
1.5 Research Outline... 4
Chapter 2 Literature Review ... 5
2.1 Voice Assistance... 5
2.1.1 Pattern Recognition... 5
2.1.2 Speech Recognition... 7
2.2 Augmented and Mixed Reality... 9
2.2.1 AR Devices... 10
2.2.2 Microsoft HoloLens 2... 11
2.2.3 Instruction-based AR... 11
2.3 Speech Interactions with AR... 13
2.4 Natural Language Processing... 14
2.4.1 Tokenizer... 15
2.4.2 Part of Speech Tagger... 16
2.4.3 Topical Classification... 18
2.5 Summary of Literature... 19
Chapter 3 Research Methodology... 21
3.1 Research Application Scenario... 21
3.2 Augmented Reality Voice Assistance (ARVA)... 22
3.3 System Architecture of ARVA... 24
3.3.1 Workflow of ARVA... 25
3.3.2 The Multinomial Naïve Bayes Classification... 29
3.4 Setup and Experimentation... 32
3.4.1 User Interface Design... 34
3.4.2 Software and Tools Prerequisites... 35
3.4.3 Experimentation... 35
Chapter 4 Experiment Results... 38
4.1 Program Result ... 38
4.2 Discussion... 41
Chapter 5 Conclusion... 42
5.1 Conclusion... 42
5.2 Problem Solution Summary... 43
5.3 Limitation and Future Work... 44
Reference... 45
Appendix ... 48
Appendix I - Tokenizer... 48
Appendix II - POS Tagger... 52
Appendix III - Naïve Bayes Classifier... 59
Appendix IV - Main Script (MainBL)... 62
Extended Abstract ... 67

Akçayır, M., & Akçayır, G. (2017). Advantages and challenges associated with augmented
reality for education: A systematic review of the literature. Educational Research
Review, 20, 1–11. https://doi.org/10.1016/j.edurev.2016.11.002

Bahja, M. (2021). Natural language processing applications in business. E-Business - Higher
Education and Intelligence Applications. https://doi.org/10.5772/intechopen.92203

Bahl, L. R., Brown, P. F., de Souza, P. V., Mercer, R. L., & Picheny, M. A. (1993). A method
for the construction of acoustic Markov models for words. IEEE Transactions on
Speech and Audio Processing, 1(4), 443–452. https://doi.org/10.1109/89.242490

Burchfiel, A. (2022, October 11). What is NLP (Natural Language Processing) tokenization?
tokenex. Retrieved December 6, 2022, from https://www.tokenex.com/blog/ab-what-isnlp-natural-language-processingtokenization/#:~:text=Tokenization%20is%20used%20in%20natural,into%20understan
dable%20parts%20(words)

Cambria, E., & White, B. (2014). Jumping NLP curves: A review of Natural Language
Processing Research [review article]. IEEE Computational Intelligence Magazine, 9(2),
48–57. https://doi.org/10.1109/mci.2014.2307227

Carmigniani, J., & Furht, B. (2011). Augmented reality: An overview. Handbook of
Augmented Reality, 3–46. https://doi.org/10.1007/978-1-4614-0064-6_1

Cheng, D., Wang, Q., Liu, Y., Chen, H., Ni, D., Wang, X., Yao, C., Hou, Q., Hou, W., Luo,
G., & Wang, Y. (2021). Design and manufacture AR head-mounted displays: A review
and outlook. Light: Advanced Manufacturing, 2(3), 336.
https://doi.org/10.37188/lam.2021.024

Dodiya, T. (2021). Using term frequency - inverse document frequency to find the relevance
of words in Gujarati language. International Journal for Research in Applied Science
and Engineering Technology, 9(4), 378–381.https://doi.org/10.22214/ijraset.2021.33625

Dubois, E., & Nigay, L. (2000). Augmented reality. Proceedings of DARE 2000 on Designing
Augmented Reality Environments - DARE '00. https://doi.org/10.1145/354666.354695

Dutsinma, F. L., Pal, D., Funilkul, S., & Chan, J. H. (2022). A systematic review of voice
assistant usability: An ISO 9241–11 approach. SN Computer Science, 3(4).
https://doi.org/10.1007/s42979-022-01172-3

Guo, Q. (2015). Learning in a Mixed Reality System in the Context of Industry 4.0. Journal
of Technical Education. vol. 3, no. 2, pp. 92–115, ISSN 2198-0306.

Hoy, M. B. (2018). Alexa, Siri, Cortana, and more: An introduction to voice assistants.
Medical Reference Services Quarterly, 37(1), 81–88.
https://doi.org/10.1080/02763869.2018.1404391
46

Irawati, S., Green, S., Billinghurst, M., Duenser, A., & Ko, H. (2006). An evaluation of an
augmented reality multimodal interface using speech and paddle gestures. Advances in
Artificial Reality and Tele-Existence, 272–283. https://doi.org/10.1007/11941354_28

Lagi, M. (2019, December 13). Natural language processing – business applications. Emerj
Artificial Intelligence Research. Retrieved December 6, 2022, from
https://emerj.com/ai-sector-overviews/natural-language-processing-businessapplications/

Loizeau, Q., Danglade, F., Ababsa, F., & Merienne, F. (2019). Evaluating added value of
augmented reality to assist aeronautical maintenance workers—experimentation on onfield use case. Virtual Reality and Augmented Reality, 151–169.
https://doi.org/10.1007/978-3-030-31908-3_10

Low, D. Y., Poh, P. E., & Tang, S. Y. (2022). Assessing the impact of augmented reality
application on students’ learning motivation in chemical engineering. Education for
Chemical Engineers, 39, 31–43. https://doi.org/10.1016/j.ece.2022.02.004

Meng, J., Zhang, J., & Zhao, H. (2012). Overview of the speech recognition technology. 2012
Fourth International Conference on Computational and Information Sciences.
https://doi.org/10.1109/iccis.2012.202

Milgram, P., & Kishino, F. (1994). A Taxonomy of Mixed Reality Visual Displays. IEICE
Trans. Information Systems. vol. E77-D, no. 12. 1321-1329.

Moawad, A., Mohammed, F., El-Mansy, A., Mohammed, N., & Abd-Ghaffar, A. (2012).
Speech Recognition System. In Smart Blind Stick (pp. 7–8). Mansoura University:
Department of Electronics and Communication Engineering

Naziya S., S. and Deshmukh, R.R. (2016) “Speech recognition system – A Review,” IOSR
Journal of Computer Engineering, 18(04), pp. 01–09. Available at:
https://doi.org/10.9790/0661-1804020109.

Neb, A., & Strieg, F. (2018). Generation of AR-enhanced assembly instructions based on
Assembly features. Procedia CIRP, 72, 1118–1123.
https://doi.org/10.1016/j.procir.2018.03.210

Piumsomboon, T., Altimira, D., Kim, H., Clark, A., Lee, G., & Billinghurst, M. (2014).
Grasp-shell vs gesture-speech: A comparison of direct and indirect natural interaction
techniques in augmented reality. 2014 IEEE International Symposium on Mixed and
Augmented Reality (ISMAR). https://doi.org/10.1109/ismar.2014.6948411

Priya B, Nandhini J.M, & Gnanasekaran T. (2021). An analysis of the applications of natural
language processing in various sectors. Advances in Parallel Computing.
https://doi.org/10.3233/apc210109

Ras, E., Wild, F., Stahl, C., & Baudet, A. (2017). Bridging the skills gap of workers in
Industry 4.0 by Human Performance Augmentation Tools. Proceedings of the 10th
International Conference on Pervasive Technologies Related to Assistive Environments.
https://doi.org/10.1145/3056540.3076192
47

Ratz, A. V. (2022, April 8). Multinomial NAЇVE Bayes' for documents classification and
Natural Language Processing (NLP). Medium. Retrieved December 15, 2022, from
https://towardsdatascience.com/multinomial-na%C3%AFve-bayes-for-documentsclassification-and-natural-language-processing-nlp-e08cc848ce6

Sharma, A., Mehtab, R., Mohan, S., & Mohd Shah, M. K. (2021). Augmented reality – an
important aspect of industry 4.0. Industrial Robot: the International Journal of Robotics
Research and Application, 49(3), 428–441. https://doi.org/10.1108/ir-09-2021-0204

Speechmatics. (2022). The Voice Report 2022. Retrieved November 8, 2022, from
https://assets.ctfassets.net/yze1aysi0225/1LwFuhCeZKZQlGDQLPoIoN/2e5459f45aa7
1de3284aedb56553bcd4/The-Speechmatics-Voice-Report-2022.pdf

Tang, Y. M., Chau, K. Y., Kwok, A. P., Zhu, T., & Ma, X. (2022). A systematic review of
immersive technology applications for medical practice and education - trends,
application areas, recipients, teaching contents, evaluation methods, and performance.
Educational Research Review, 35, 100429.
https://doi.org/10.1016/j.edurev.2021.100429

Taylor, A., Marcus, M., & Santorini, B. (2003). The penn treebank: An overview. Treebanks,
5–22. https://doi.org/10.1007/978-94-010-0201-1_1

Wang, Z., Bai, X., Zhang, S., Billinghurst, M., He, W., Wang, P., Lan, W., Min, H., & Chen,
Y. (2022). A comprehensive review of Augmented Reality-based instruction in manual
assembly, training and Repair. Robotics and Computer-Integrated Manufacturing, 78,
102407. https://doi.org/10.1016/j.rcim.2022.102407

Wang, Z., Bai, X., Zhang, S., He, W., Zhang, X., Zhang, L., Wang, P., Han, D., & Yan, Y.
(2019). Information-level AR instruction: A novel assembly guidance information
representation assisting user cognition. The International Journal of Advanced
Manufacturing Technology, 106(1-2), 603–626. https://doi.org/10.1007/s00170-019-
04538-9

Washani, N. and Sharma, S. (2015) “Speech recognition system: A Review,” International
Journal of Computer Applications, 115(18), pp. 7–10. Available at:
https://doi.org/10.5120/20249-2617.

Werrlich, S., Daniel, A., Ginger, A., Nguyen, P.-A., & Notni, G. (2018). Comparing HMDbased and paper-based training. 2018 IEEE International Symposium on Mixed and
Augmented Reality (ISMAR). https://doi.org/10.1109/ismar.2018.00046

Wu, J. (2020). Overview of a Pattern Recognition System. In Essentials of Pattern
Recognition: An Accessible Approach (pp. 44-62). Cambridge: Cambridge University
Press. doi:10.1017/9781108650212.005