臺灣博碩士論文加值系統

本論文探討有關自然語言處理工具的硬體測試與應用。自然語言處理是在人工智慧領域中，實現人類與電腦之間使用自然語言來進行有效地溝通的研究理論與方法。由於它的理論與實作和探討人類的思維有關，使得每一項重大的突破往往都是經歷了幾十年的時間才得以突破。近年來自然語言處理在中文分詞、詞彙語義、詞性標記、句法解析這些技術都獲得了很大的突破。它的範疇包含了文字與語音這兩種應用，文字的應用包含機器翻譯、自動摘要、文字分類、語法校對甚至是應用在電腦程式設計上等。語音的部分則有語音辨識、聊天機器人、多媒體資訊分析等。
本次研究著重在自然語言處理的工具之應用，首先我們在不同的硬體環境上測試了三種自然語言的工具，分別是詞性標記(Part-of-Speech Tagging)、命名實體識別(Named Entity Recognition)、分塊(Chunking)，先是測試了訓練準確度(Accuracy)和F1-score，再來觀察不同的硬體環境下運行的差異性。其次我們將十類的論文主題短文經過分類處理得到一些結果。我們把這些資料劃分成兩個非對稱分布資料以及一個對稱分布資料，透過詞袋模型建模以及經由詞嵌入處理後再用t-SNE來降維後去顯示出分類結果。從結果中能得知，個別處理每一類的優點是對於那些出現次數較少的關鍵字來說比較不容易被其他類的詞給覆蓋過，缺點則是處理的過程繁複。而將所有類合併處理的優點是在顯現各類主題詞，使用處理兩個詞的模型會比處理三個詞的好。在處理三個詞的時候，有保留介係詞的模型能找到十類裡面大部分的關鍵詞，而移除介係詞的模型雖然能保留三個詞，但對於一些兩個字組成的詞來說是很難分類到的。隨後我們將十類的短文資料經由獨立的兩個字組成的詞之分類方法來分出關鍵字。接著在使用LDA (Latent Dirichlet Allocation)來訓練多群的短文，並將之依序分成10、5、3群，並將Test Data對訓練的模型進行測試。LDA為一個較完整的文字預測模型，此方法是將每一篇文件的機率視為潛在主題中隨機字詞的機率之混合模型，進而求得該篇文件出現的機率值。根據5.6.1的結果，10類的非對稱分布資料中訓練後我們得知，前者中較少的關鍵字不容易被找出關鍵字，對稱分布資料則能發現10類的所有關鍵字平均地分散在各10類中。至於5類與3類的LDA分析則能夠發現，因為都是在對稱分布資料下進行訓練，10類中個關鍵字依照各自的關聯性分類成一群。整體而言與非對稱分布資料相比，訓練後的LDA模型的對稱分布資料表現更好。 非對稱分布資料會導致一些關鍵字被合併。

This thesis explores the hardware test and application with tools of natural language processing (NLP). NLP is a research theory and method that uses natural language to communicate effectively between humans and computers in the field of artificial intelligence (AI). Because its theory is related to the implementation and discussion of human thinking, every major breakthrough often takes decades to break through. In recent years, NLP has made great breakthroughs in Chinese word segmentation, lexical semantics, part-of-speech tagging, and syntactic parsing. Its class includes both text and speech applications. The applications of text include machine translation, automatic summarization, text categorization, grammar proofing, and even programming. The applications of speech include voice recognition, chat robot, multimedia information analysis and so on.
This research focuses on the applications with tools of NLP. Firstly, we tested three tools of NLP in different hardware environments, namely Part-of-Speech Tagging (POS) and Named Entity Recognition (NER). Chunking, first tested the accuracy of the Accuracy and F1-score, and then observed the difference in the operation of different hardware environments. Secondly, we classify ten types of paper subject themes to get some results. We divide this data into two asymmetrically distributed data and one symmetrically distributed data. After word embedding processing, t-SNE is used to reduce the dimensionality to show the classification results. It can be seen from the results that the advantage of processing each class individually is that for those keywords that appear less frequently, it is less likely to be covered by other types of words. The disadvantage is that the processing process is complicated. The advantage of combining all classes is that in displaying various types of subject words, using a model that processes bigrams is better than trigrams. When dealing with trigrams, models with prepositions reserved can find most of the keywords in the ten classes, while models that remove prepositions retain trigrams, but for some bigram words are difficult to classify. Then we classify the ten types of essay data through independent bigram classification methods to separate keywords. Finally, we use Latent Dirichlet Allocation (LDA) to train many types of essays. And sequentially divided into the class 10, 5, 3. Then tested with pre-trained LDA topic models. The LDA is a more complete text prediction model. This method is a hybrid model that combines the probability of each document with the probability of random words in a potential topic. Then obtain the probability value of the document. According to the results of 5.6.1, after training in 10 classes of asymmetrically distributed data, we learned that the keywords of the lesser data class in the former are not easy to find keywords, and the symmetrically distributed data can find all the keywords of 10 classes. They are evenly distributed among the 10 classes. As for the LDA analysis on classes 5 and 3, because the training is performed on symmetrically distributed data, the keywords in the 10 classed into classes 5 and 3 are independently classified into a group according to their relevance, respectively. To sum up, compared to asymmetrically distributed data, the symmetrically distributed data for the LDA model after training performs better. Asymmetrically distributed data causes a few keywords being merged.

摘要....i
Abstract....iii
Acknowledgements....v
Table of Contents....vi
List of Tables....viii
List of Figures....xii
Chapter 1 Introduction....1
1.1 Review of Natural Language Processing....1
1.2 Review of Deep Learning....1
1.3 Motivation and Contributions....3
1.4 Organization of this Thesis....5
Chapter 2 Preliminaries....6
2.1 Word Representation....6
2.2 Bag-of-words model....7
2.3 Term Frequency-Inverse Document Frequency (TF-IDF)....8
2.4 N-gram model....9
2.5 Word Embedding....10
Chapter 3 Hardware Test with Tools of Natural Language Processing....11
3.1 Introduction....11
3.2 Properties of Different Hardware....13
3.3 Experimental Results....13
3.4 Concluding Remarks....15
Chapter 4 Keywords Analysis with Natural Language Processing Tools....16
4.1 Introduction....16
4.2 Architectures....17
4.2.1 Data Processing....17
4.2.2 Data Description....18
4.2.3 t-Distributed Stochastic Neighbor Embedding (t-SNE)....18
4.3 Experimental Results....19
4.3.1 Results of Individual Class....19
4.3.2 Results for Class Distribution of Asymmetrically Distributed Data (1000-9000)....60
4.3.3 Results for Class Distribution of Asymmetrically Distributed Data (3000-5000)....68
4.3.4 Results for Class Distribution of Symmetrically Distributed Data (3000)....76
4.4 Concluding Remarks....85
Chapter 5 Unsupervised Keywords Analysis with Natural Language Processing tools and Latent Dirichlet Allocation....86
5.1 Introduction....86
5.2 Mixture of Unigrams....86
5.3 Latent Dirichlet Allocation (LDA)....87
5.3.1 Topic model LDA....88
5.3.2 LDA model....89
5.3.3 LDA parameter estimation: Gibbs sampling....90
5.4 Architectures....96
5.5 Experiment Results....97
5.5.1 Results for the 10 topics LDA model Mixture with Unigram....97
5.5.2 Results for the 5 topics LDA model Mixture with Unigram....102
5.5.3 Results for the 3 topics LDA model Mixture with Unigram....105
5.6 Concluding Remarks....108
Chapter 6 Conclusions and Future Works....109
References....111
Extended Abstract....117

[1]W. Y. Zou, R. Socher, D. Cer and C. D. Manning, "Bilingual Word Embeddings for Phrase-Based Machine Translation", Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), vol.13, pp. 1393-1398, Oct. 2013.
[2]K. Hui, A. Yates, K. Berberich and G. de Melo, "PACRR: A Position-Aware Neural IR Model for Relevance Matching", Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1049-1058,Sep. 2017.
[3]M. Abadi, A. Agarwal, P. Barham, et al., "TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems", arXiv: 1603.04467, Mar. 2016.
[4]Y. Bengio, R. Ducharme, P. Vincent and C. Jauvin, "A Neural Probabilistic Language Model", Journal of Machine Learning Research, vol. 3, pp. 1137-1155, Feb. 2003.
[5]T. Mikolov, K. Chen, G. Corrado and J. Dean, "Efficient Estimation of Word Representations in Vector Space", arXiv: 1301.3781, Sep. 2013.
[6]J. Pennington, R. Socher and C. D. Manning, "Glove: Global Vectors for Word Representation", Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), vol. 14, pp. 1532-1543, Oct. 2014.
[7]M. Bansal, K. Gimpel and K. Livescu, "Tailoring Continuous Word Representations for Dependency Parsing", Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, vol. 2, pp. 809-815, June 2014.
[8]J. Turian, L. Ratinov and Y. Bengio, "Word Representations: A Simple and General Method for Semi-Supervised Learning", Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 384-394, July 2010.
[9]R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu and P. Kuksa, "Natural Language Processing (Almost) from Scratch", Journal of Machine Learning Research, vol. 12, pp. 2493-2537, Mar. 2011.
[10]R. Lebret and R. Collobert, "Word Emdeddings Through Hellinger PCA", Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pp. 482-490, Apr. 2014.
[11]O. Levy and Y. Goldberg, "Dependency-Based Word Embeddings", Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, vol. 2, pp. 302-308, June 2014.
[12]M. Marcus, B. Santorini and M. A. Marcinkiewicz, "Building a Large Annotated Corpus of English: The Penn Treebank", Computational Linguistics, vol. 19, pp. 313-330, Oct. 1993.
[13]E. F. T. K. Sang and F. D. Meulder, "Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition", Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003, vol. 4, pp. 142-147, 2003.
[14]E. F. T. K. Sang and S. Buchholz, "Introduction to the Conll-2000 Shared Task: Chunking", Proceedings of the 2nd workshop on Learning language in logic and the 4th conference on Computational natural language learning, vol. 7, pp. 127-132, 2000.
[15]S. Hochreiter and J. Schmidhuber, "Long Short-Term Memory", Neural Computation, vol. 9, pp. 1735-1780, Nov. 1997.
[16]S. Ji, H. Yun, P. Yanardag, S. Matsushima and S. V. N. Vishwanathan, "WordRank: Learning Word Embeddings via Robust Ranking", Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 658-668, Nov. 2016.
[17]B. Gao, J. Bian and T. Y. Liu, " WordRep: A Benchmark for Research on Learning Word Representations", arXiv:1407.1640, July 2014.
[18]O. Levy, Y. Goldberg and I. Dagan, "Improving Distributional Similarity with Lessons Learned from Word Embeddings", Transactions of the Association for Computational Linguistics, vol. 3, pp. 211-225, 2015.
[19]S. Ghannay, Y. Est`eve, N. Camelin, C. Dutrey, F. Santiago and M. A. Decker, "Combining Continuous Word Representation and Prosodic Features for ASR Error Prediction", International Conference on Statistical Language and Speech Processing, pp. 84-95, Nov. 2015.
[20]G. Mesnil, Y. Dauphin, K. Yao, Y. Bengio, L. Deng, D. H. Tur, X. He, L. Heck, G. Tur and D. Yu, et al., "Using Recurrent Neural Networks for Slot Filling in Spoken Language Understanding", Audio, Speech, and Language Processing IEEE/ACM Transactions, vol. 23, pp. 530-539, Mar. 2015.
[21]C. E. Shanon, "A Mathematical Theory of Communication", The Bell System Technical Journal, vol. 27, pp. 623-656, Oct. 1948.
[22]P. D. Turney and P. Pantel, "From Frequency to Meaning: Vector Space Models of Semantics", Journal of Artificial Intelligence Research, vol. 37, pp. 141-188, Mar. 2010.
[23]D. M. Blei, A. Y. Ng and M. I. Jordan, "Latent Dirichlet Allocation", Journal of Machine Learning Research, vol. 3, no. 5, pp. 993-1022, Mar. 2003.
[24]G. Salton and C. S. Yang, "On the Specification of Term Values in Automatic Indexing", Journal of Documentation, vol. 29, no. 4, pp. 351-372, June 1973.
[25]D. M. Blei and J. D. Lafferty, "Correlated Topic Model", Advances in Neural Information Processing Systems (NIPS), vol. 18, pp. 147-154, Aug. 2006.
[26]D. M. Blei and J. D. Lafferty, "Dynamic Topic Model", Proceedings of the 23rd International Conference on Machine Learning, pp. 113-120, June 2006.
[27]T. Hofmann, "Probabilistic Latent Semantic Analysis", UAI’99: Proceedings of the Fifteenth Annual Conference on Uncertainty in Artificial Intelligence, pp. 289-296, July 1999.
[28]T. Hofmann, "Unsupervised Learning by Probabilistic Latent Semantic Analysis", Machine Learning, vol. 42, pp. 177-196, 2001.
[29]D. Mrva and P. C. Woodland, "Unsupervised Language Model Adaptation for Mandarin Broadcast Conversation Transcription", Proceedings of International Conference on Spoken Language Processing, pp. 1961-1964, 2004.
[30]Y. C. Tam and T. Schultz, "Dynamic Language Model Adaptation Using Variational Bayes Inference", Proceedings of European Conference on Speech Communication and Technology, pp. 5-8, Jan. 2005.
[31]Y. Ko, J. Seo, "Automatic Text Categorization by Unsupervised Learning", Proceedings of the 18th conference on Computational linguistics, vol. 1, pp. 453-459, 2000.
[32]M. Caillet, J. F. Pessiot, M. R. Amini, P. Gallinari, "Unsupervised Learning with Term Clustering for Thematic Segmentation of Texts", Coupling Approaches, Coupling Media and Coupling Languages for Information Retrieval, pp. 648-657, Apr. 2004.
[33]R. A. G. Hernández, R. Montiel, Y. Ledeneva, E. Rendón, A. Gelbukh, R. Cruz, "Text Summarization by Sentence Extraction Using Unsupervised Learning", Mexican International Conference on Artificial Intelligence MICAI 2008, Lecture Notes in Computer Science, vol. 5317, pp. 133-143, Oct. 2008.
[34]T. Jo, "Text Categorization: Approaches", Text Categorization: Approaches. In: Text Mining. Studies in Big Data, vol. 45. Springer, pp. 101-127, 2019.
[35]Y. Bengio, A. Courville, P. Vincent, "Representation Learning: A Review and New Perspectives", IEEE Trans. PAMI, special issue Learning Deep Architectures, vol. 35, pp. 1798-1828, Aug. 2013.
[36]G. Hinton, S. Roweis, "Stochastic Neighbor Embedding", Proceedings of the Advances in Neural Information Processing System, pp. 857-864, 2002.
[37]L. van der Matten, G. Hinton, "Visualizing High-Dimensional Data Using t-SNE", Journal of Machine Learning Research, vol. 9, pp. 2579-2605, Nov. 2008.
[38]Matlab: Text Analytics Toolbox User's Guide, Mathworks, 2019.
[39]Matlab: Text Analytics Toolbox Reference, Mathworks, 2019.
[40]Y. Bengio, A. Courville and P. Vincent, "Representation Learning: A Review and New Perspectives", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 35, pp. 1798-1828, Aug. 2013.
[41]J. Schmidhuber, "Deep Learning in Neural Networks: An Overview", Neural Networks, vol. 61, pp. 85-117, Jan. 2015.
[42]Y. LeCun, Y. Bengio and G. Hinton, "Deep Learning", Nature, vol. 521, pp. 436-444, May. 2015.
[43]P. Langley, "The Changing Science of Machine Learning", Machine Learning, vol. 82, pp. 275-279, Feb. 2011.
[44] C. X. Zhai, Statistical Language Models for Information Retrieval, Morgan & Claypool Publishers, 2009.
[45] Meng-Sung Wu, Hsuan-Jui Hsu, Jen-Tzung Chien, “Bayesian Topic Mixture Model for Information Retrieval [In Chinese]”, Proceedings of the 19th Conference on Computational Linguistics and Speech Processing, 2007.
[46] Hamed Jelodar, Yongli Wang, Chi Yuan, Xia Feng, Xiahui Jiang, Yanchao Li and Liang Zhao, “Latent Dirichlet Allocation (LDA) and Topic modeling: models, applications, a survey”, arXiv:1711.04305v2, 2018.
[47] Nurzat Rakhmanberdieva, “Word Representation in Natural Language Processing Part II”, 2018. Website: https://towardsdatascience.com/word-representation-in-natural-language-processing-part-ii-1aee2094e08a