臺灣博碩士論文加值系統

自然語言處理旨在賦予機器理解並應用人類語言的能力。隨著深度學習的進步和普及，一種新的策略已經出現，旨在以更有效的方式教導機器如何理解和使用語詞。這種策略就是將語詞轉換為一種稱為「詞向量」的形式，或者更具體地說，這是一種被稱為 Word Embedding 的技術。其核心思想是讓機器閱讀大量的文本，並從這些文本中學習語詞的上下文相關性。這樣，每個語詞都會獲得一個向量表示，這個表示捕獲了詞彙的語義和它在語言中的用法，而無需依賴專門的語言學知識。Bidirectional Encoder Representations from Transformers (BERT) 是 Google 在2018年以無監督的方式利用大量無標註文本訓練成的語言代表模型，其架構為 Transformer 中的 Encoder。傳統中文BERT 在Fine-tune的四種下游任務分別為單一句子分類任務、單一句子標註任務、成對句子分類任務、問答任務都是以句子或斷落或文章來做輸入，而沒有以單字或詞對的下游任務。本論文提出以轉移學習之BERT對同義詞、反義詞和不相關詞之預測分類模型開發。我們收集人工智慧和尖端科技的新聞標題做為語料庫。我們以轉移學習的方式來對中文BERT做同義詞、反義詞和不相關詞之多類別分類任務。其次提出基於中文BERT對同義詞、反義詞與不相關詞之多類別分類任務之結構重建與開發。針對BERT內部的架構來做結構重建，針對我們資料量的規模較小，所以將內部的結構減少，最後由實驗結果去驗證我們提出的方法之穩健性與有效性。第三是提出中文BERT結合Skip-gram對同義詞與反義詞之相似度比較開發，上下文嵌入的方法會導致參數極大且複雜，使得計算成本提高。我們採用靜態嵌入的方法並且以BERT用做上下文嵌入來替換模型的中心詞嵌入，來降低計算成本，並且達到上下文嵌入的效果。

Natural Language Processing (NLP) aims to compose machines with the capability to understand and utilize human language. With the advancement and proliferation of deep learning, a novel strategy has emerged, intending to teach machines to comprehend and utilize words more effectively. This strategy involves transforming words into a form known as 'word vectors,' or more specifically, a technique called Word Embedding. The core idea is to have machines read extensive text volumes, learning the context correlations of words from these texts. Consequently, each word acquires a vector representation that captures the semantic of the word and its usage in language, without the reliance on specialized linguistic knowledge. Bidirectional Encoder Representations from Transformers (BERT) is a linguistic representation model trained by Google in 2018 in an unsupervised way using a large amount of unmarked text, and its architecture is the Encoder in Transformer. The four downstream tasks of BERT Fine-tune are single-sentence classification task, single-sentence annotation task, paired-sentence classification task, and question-answer task, all of which are based on sentences or broken sentences or articles as input, but not on single word or word pairs. This thesis proposed the development of a predictive classification model for synonyms, antonyms and irrelevant words using BERT of transfer learning. We collect news headlines of artificial intelligence and cutting-edge technology as a corpus. We use transfer learning to do multi-category classification tasks for Chinese BERT with synonyms, antonyms and irrelevant words. Secondly, the structure reconstruction and development of multi-category classification tasks for synonyms, antonyms and irrelevant words based on Chinese BERT are proposed. For the structural reconstruction of BERT's internal architecture, the scale of our data is small, so the internal structure is reduced, and finally the experimental results verify the robustness and effectiveness of this thesis proposed method. The third is to propose Chinese BERT combined with Skip-gram to compare and develop the similarity between synonyms and antonyms. The method of context embedding will lead to extremely large and complex parameters, which will increase the calculation cost. We adopt the method of static embedding and use BERT as context embedding to replace the model's center word embedding to reduce the computational cost and achieve the effect of context embedding.

摘要...i
Abstract...iii
誌謝...v
Table of Contents...vi
List of Tables...ix
List of Figures...xi
Chapter 1 Introduction...1
1.1 Natural Language Processing...1
1.2 AI and Machine Learning on Natural Language Processing...2
1.3 Corpus and Knowledge-Base...2
1.4 Motivations and Contributions...3
1.5 Organization of This Thesis...5
Chapter 2 Preliminaries...6
2.1 Review of WordNet...6
2.2 Review of Word2Set...6
2.3 Review of Chinese Word2Set...8
2.4 Review of Word2Vec...9
2.5 Review of E-HowNet...12
2.6 Review of Chinese Word2Set based on Corpus and Knowledge-Base...12
2.7 Review of Transfer Learning...14
2.8 Compare Synonyms and Antonyms on Chat Generative Pre-trained Transformer...15
2.9 Review of Transformer and Language Model...20
2.10 Review of BERT...23
Chapter 3 The Development of Chinese BERT Transfer Learning for Multi-Category Classification Tasks of Synonyms and Antonyms...29
3.1 Proposed Methods...29
3.2 Introduction of Training Dataset...32
3.2.1 Technology News Dataset...34
3.2.2 Wiki Dataset...37
3.2.3 Antonyms Dataset...38
3.3 Training Environment...39
3.4 Experimental Results...40
3.4.1 Wiki Dataset and Antonyms Dataset...40
3.4.2 Technology News Dataset and Antonyms Dataset...53
3.5 Discussion and Conclude Remarks...66
Chapter 4 Structural Reconstruction and Development of Multi-Category Classification Tasks for Synonyms and Antonyms Based on Chinese BERT...67
4.1 Fine-tune...67
4.2 Proposed Methods...67
4.3 Training Environment...75
4.4 Experimental Results...76
4.4.1 Wiki Dataset and Antonyms Dataset...77
4.4.2 Technology News Dataset and Antonyms Dataset...83
4.5 Discussion and Conclude Remarks...99
Chapter 5 The Development of Chinese Word2Set and Chinese BERT Combined with Skip-Gram Compares the Similarity Between Synonyms and Antonyms...100
5.1 Chinese Word2Set based on Corpus and Knowledge-Base...100
5.2 Proposed Methods...101
5.3 Introduction of Training Dataset...105
5.4 Experimental Results...107
5.4.1 Wiki Dataset and Antonyms Dataset...108
5.4.2 Technology News Dataset and Antonyms Dataset...117
5.5 Discussion and Conclude Remarks...122
Chapter 6 Conclusions and Future Research...123
6.1 Conclusions...123
6.2 Future Works...123
References...124
Extended Abstract...129

[1]D. Khurana, A. Koli, K. Khatter, and S. Singh, "Natural Language Processing: State of The Art, Current Trends and Challenges," Multimedia Tools and Applications, vol. 82, Jul 2022.
[2]T. Mikolov, K. Chen, G. Corrado, and J. Dean, "Efficient Estimation of Word Representations in Vector Space.," arXiv, 06 Sep 2013.
[3]A. Sherstinsky, "Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network," Physica D: Nonlinear Phenomena, vol. 404, p. 132306, Mar 2020.
[4]A. Vaswani et al., "Attention Is All You Need," arXiv, 05 Dec 2017.
[5]J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” arXiv, 24 May 2019.
[6]L. Floridi and M. Chiriatti, "GPT-3: Its Nature, Scope, Limits, and Consequences," Minds & Machines, vol. 30, no. 4, p. 681–694, 2020.
[7]Z.-H. Zhou, Machine Learning, Springer Nature, 2021.
[8]P. Cunningham, M. Cord, and S. J. Delany, Supervised Learning, Springer, 2008, pp. 21-49.
[9]H. B. Barlow, "Unsupervised Learning," Neural Computation, vol. 1, no. 3, pp. 295-311, Sep. 1989.
[10]L. Jiang, H. Zhang, and Z. Cai, "A Novel Bayes Model: Hidden Naive Bayes," IEEE Transactions on Knowledge and Data Engineering, vol. 21, no. 10, pp. 1361-1371, Oct. 2009.
[11]S. Suthaharan, Support Vector Machine, Boston: Springer US, 2016, pp. 207-235.
[12]T. G. Nick and K. M. Campbell, "Logistic Regression," Topics in Biostatistics, W. T. Ambrosius, Ed., in Methods in Molecular Biology™., pp. 273-301, 2007.
[13]A. J. Myles, R. N. Feudale, Y. Liu, N. A. Woody, and S. D. Brown, "An Introduction to Decision Tree Modeling," Journal of Chemometrics, vol. 18, no. 6, pp. 275-285, 2004.
[14]G. Biau and E. Scornet, "A Random Forest Guided Tour," Test, vol. 25, no. 2, pp. 197-227, Jun. 2016.
[15]A. O’Keeffe and M. McCarthy, The Routledge Handbook of Corpus Linguistics, Routledge, 2010.
[16]J. Fagerberg, M. Fosaas, and K. Sapprasert, "Innovation: Exploring the Knowledge Base," Research Policy, vol. 41, no. 7, pp. 1132-1153, Sep. 2012.
[17]J. L. Martinez-Rodriguez, I. Lopez-Arevalo, and A. B. Rios-Alvarado, "OpenIE-Based Approach for Knowledge Graph Construction from Text," Expert Systems with Applications, pp. 339-355, Dec. 2018.
[18]C. Goddard, Semantic Analysis: A Practical Introduction, OUP Oxford, 2011.
[19]Y. L. e. al., "Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models," arXiv, 10 May 2023.
[20]W.-T. Chen, S.-C. Lin, S.-L. Huang, Y.-S. Chung, and K.-J. Chen, "E-HowNet and Automatic Construction of a Lexical Ontology," Coling 2010: Demonstrations, pp. 45-48, Aug. 2010.
[21]C. Fellbaum, "WordNet," Theory and Applications of Ontology: Computer Applications, pp. 231-243, 2010.
[22]G. A. Miller, R. Beckwith, C. Fellbaum, D. Gross, and K. J. Miller, "Introduction to WordNet: An On-line Lexical Database," International Journal of Lexicography, vol. 3, no. 4, pp. 235-244, Dec. 1990.
[23]C.-R. Huang, E. I. J. Tseng, D. B. S. Tsai, and B. Murphy, "Cross-Lingual Portability of Semantic Relations," Language and Linguistics, vol. 4, no. 3, pp. 509-532, 1 7 2003.
[24]S. Jimenez, F. A. Gonzalez, A. Gelbukh, and G. Duenas, "Word2Set: WordNet-Based Word Representation Rivaling Neural Word Embedding for Lexical Similarity and Sentiment Analysis," IEEE Computational Intelligence Magazine, vol. 14, no. 2, p. 41–53, May 2019.
[25]Y.-T. Jeng, The Development of Chinese Word2Set and Its Applications on Similarity of Words and Short Text, National Formosa University, 2020.
[26]Y. Wang, L. Cui, and Y. Zhang, "Improving Skip-Gram Embeddings Using BERT," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, p. 1318–1328, 2021.
[27]Y.-J. Huang, Development of Chinese Word2Set Based on the Improvement of Corpus and Knowledge-Base, National Formosa University, 2022.
[28]S. Niu, Y. Liu, J. Wang, and H. Song, "A Decade Survey of Transfer Learning (2010–2020)," IEEE Transactions on Artificial Intelligence, vol. 1, no. 2, pp. 151-166, Oct. 2020.
[29]F. Zhuang et al., "A Comprehensive Survey on Transfer Learning," Proceedings of the IEEE, vol. 109, no. 1, pp. 43-76, Jan. 2021.
[30]K. Weiss, T. M. Khoshgoftaar, and D. Wang, "A Survey of Transfer Learning," Journal of Big Data, vol. 3, no. 1, p. 9, May. 2016.
[31]C. Sun, X. Qiu, Y. Xu, and X. Huang, "How to Fine-Tune BERT for Text Classification?," Chinese Computational Linguistics, pp. 194-206, 2019.
[32]J. R. Bellegarda, "Statistical Language Model Adaptation: Review and Perspectives," Speech Communication, vol. 42, no. 1, pp. 93-108, Jun. 2004.
[33]A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, "Language Models are Unsupervised Multitask Learners," OpenAI Blog, 2019.
[34]Y. Bengio, R. Ducharme, and P. Vincent, "A Neural Probabilistic Language Model," in Advances in Neural Information Processing Systems, 24 Jul. 2023.
[35]R. C. Staudemeyer and E. R. Morris, "Understanding LSTM -- a Tutorial into Long Short-Term Memory Recurrent Neural Networks," arXiv, 12 Sep. 2019.
[36]P. Shaw, J. Uszkoreit, and A. Vaswani, "Self-Attention with Relative Position Representations," arXiv, 12 Apr. 2018.
[37]C. Tao, S. Gao, M. Shang, W. Wu, D. Zhao, and R. Yan, "Get The Point of My Utterance! Learning Towards Effective Responses with Multi-Head Attention Mechanism," Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI-18), p. 4424, 2018.
[38]C. Xi, G. Lu, and J. Yan, "Multimodal Sentiment Analysis Based on Multi-Head Attention Mechanism," Proceedings of the 4th International Conference on Machine Learning and Soft Computing, pp. 34-39, Mar. 2020.
[39]S. González-Carvajal and E. C. Garrido-Merchán, "Comparing BERT Against Traditional Machine Learning Text Classification," arXiv, Jan. 2021.
[40]M. Joshi, O. Levy, D. S. Weld, and L. Zettlemoyer, "BERT for Coreference Resolution: Baselines and Analysis," arXiv, 22 Dec. 2019.
[41]S. Zheng and M. Yang, "A New Method of Improving BERT for Text Classification," Intelligence Science and Big Data Engineering, pp. 442-452, 2019.
[42]Y. Chang, L. Kong, K. Jia, and Q. Meng, "Chinese Named Entity Recognition Method Based on BERT," 2021 IEEE International Conference on Data Science and Computer Application (ICDSCA), pp. 294-299, Oct. 2021.
[43]J. Luoma and S. Pyysalo, "Exploring Cross-sentence Contexts for Named Entity Recognition with BERT," arXiv, 17 Dec. 2020.
[44]F. Alloatti, L. Di Caro, and G. Sportelli, "Real Life Application of a Question Answering System Using BERT Language Model," in Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pp. 250-253, Sep. 2019.
[45]K. Duan, S. Du, Y. Zhang, Y. Lin, H. Wu, and Q. Zhang, "Enhancement of Question Answering System Accuracy via Transfer Learning and BERT," Applied Sciences, vol. 12, no. 22, Jan. 2022.
[46]Y. Liu, "Fine-tune BERT for Extractive Summarization," arXiv, 05 Sep. 2019.
[47]H. Huo and M. Iwaihara, “Utilizing BERT Pretrained Models with Various Fine-Tune Methods for Subjectivity Detection,” Lecture Notes in Computer Science, pp. 270-284, 2020.
[48]K. Clark, U. Khandelwal, O. Levy, and C. D. Manning, "What Does BERT Look At? An Analysis of BERT’s Attention," arXiv, 10 2019 Jun..
[49]O. Kovaleva, A. Romanov, A. Rogers, and A. Rumshisky, "Revealing the Dark Secrets of BERT," arXiv, 11 Sep. 2019.
[50]D. Bahdanau, K. Cho, and Y. Bengio, "Neural Machine Translation by Jointly Learning to Align and Translate," arXiv, 19 May. 2016.
[51]G. Kobayashi, T. Kuribayashi, S. Yokoi, and K. Inui, "Attention is Not Only a Weight: Analyzing Transformers with Vector Norms," arXiv, 06 Oct. 2020.
[52]B. Wen, K. P. Subbalakshmi, and F. Yang, "Revisiting Attention Weights as Explanations from an Information Theoretic Perspective," arXiv, 31 Oct. 2022.