臺灣博碩士論文加值系統

政府制定專利法以保障創意，但申請專利過程繁複，在專利申請前，不僅需了解發明、新型和設計三類專利的區別，且各類別的申請文件撰寫特點不同，不僅要清楚表達創意，還需符合產業利用性、新穎性及進步性三大專利要件。此外，需使用中華民國專利資訊檢索系統（Taiwan Patent Search System）進行查詢，以避免與現有專利雷同。上述問題若因選擇不當、未符合專利要件或與現有專利雷同，都可能導致申請被駁回。為了改善撰寫問題及提升申請效率與成功率，本論文蒐集中華民國專利資訊檢索系統中的公開專利文本，建置專利領域預訓練模型（Patent domain pre-trained BERT, PBERT），提升BERT模型對專利領域的理解能力，根據實驗結果，PBERT模型的混淆度可達1.5，相比其他預訓練模型降低了1.03。預訓練完畢後，以PBERT模型分別建置專利類別預測模型（Patent Category Prediction BERT, PCPBERT）和專利要件預測模型（Patentability Prediction BERT, PPBERT），評估文本的專利類別與是否滿足專利要件特性，在多元分類任務中，PCPBERT的準確率達到93.19%，而在多標籤任務中，PPBERT模型的準確率達到96.72%，兩者模型顯著優於其他未經專利領域訓練的模型；當文本未滿足任意特性時，使用專利文本生成模型（PatentLLAMA）修改文本以滿足專利要件特性，根據實驗結果顯示，PatentLLAMA在生成滿足專利要件的文本方面，成功率達到95.2%，已充分提高了撰寫效率。隨後為提升申請效率及成功率，建置LSI模型，比對自己的文本或想法，找尋相似的專利文本，避免遭到駁回。上述所有模型建置完畢後，以聊天機器人作為使用介面，結合先前所建置的四大模型。當使用者輸入對應之文本時，便可根據設定之功能改善撰寫專利時常遇到的問題，提升申請效率及成功率，讓需要相關資源的發明人降低被駁回機率。

The government formulates patent laws to safeguard creativity, but the process of patent application is intricate. Prior to applying for a patent, one not only needs to understand the distinctions among invention, utility model, and design patents but also the unique characteristics of drafting documents for each category. It is imperative to clearly articulate the innovation while ensuring compliance with the three major patent criteria: industrial applicability, novelty, and inventive step.Additionally, one needs to utilize the Taiwan Patent Search System to conduct searches to avoid similarities with existing patents. Failure to choose appropriate options, meet patent criteria, or avoid similarity with existing patents can result in application rejection. To address these writing issues and enhance application efficiency and success rates, this paper collects publicly available patent texts from the Taiwan Patent Search System to establish a Patent domain pre-trained BERT (PBERT) model.This model enhances BERT's understanding of the patent domain. According to experimental results, the confusion degree of the PBERT model can reach 1.5, reducing by 1.03 compared to other pre-trained models. After pre-training, the PBERT model is used to build Patent Category Prediction BERT (PCPBERT) and Patentability Prediction BERT (PPBERT) models to evaluate text for patent category and patentability characteristics.In multi-class classification tasks, PCPBERT achieves an accuracy rate of 93.19%, while in multi-label tasks, PPBERT achieves an accuracy rate of 96.72%. Both models significantly outperform other models not trained in the patent domain. When the text does not meet any characteristic, the PatentLLAMA model is used to generate text that satisfies patent criteria. According to experimental results, PatentLLAMA achieves a success rate of 95.2% in generating text that meets patent criteria, significantly improving writing efficiency.Subsequently, an LSI model is constructed to compare one's own text or ideas with similar patent texts to avoid rejection. After establishing all the aforementioned models, a chatbot is employed as the user interface, integrating the four major models built previously. When users input corresponding text, they can improve common issues encountered in patent writing based on the set functions, enhancing application efficiency and success rates, thus reducing the probability of rejection for inventors in need of relevant resources.

摘要...........................i
Abstract...........................ii
誌謝...........................iii
目錄...........................iv
表目錄...........................vii
圖目錄...........................viii
第一章 緒論...........................1
1.1 研究背景...........................1
1.2 研究動機...........................2
1.3 研究目的...........................3
第二章 研究技術探討...........................4
2.1 專利法...........................4
2.1.1 發明專利...........................4
2.1.2 新型專利...........................5
2.1.3 設計專利...........................5
2.2 聊天機器人...........................6
2.3 BERT...........................6
2.3.1 Transformer...........................6
2.3.2 預訓練(Pre-training)...........................7
2.3.3 微調(Fine-Tuning)...........................8
2.4 Longformer BERT...........................9
2.4.1 Stride Attention...........................9
2.4.2 Local Attention...........................10
2.4.3 Global Attention...........................10
2.5 機器學習...........................11
2.5.1 監督式學習...........................11
2.5.2 非監督式學習...........................12
2.5.3 半監式督學習...........................13
2.6 分類器種類...........................14
2.6.1 二元分類/多元分類...........................14
2.6.2 多標籤分類...........................15
2.7 過擬合...........................16
2.8 評估指標...........................17
2.8.1 準確率(Accuracy)...........................17
2.8.2 精確率(Precision)...........................17
2.8.3 召回率(Recall)...........................18
2.8.4 F1-Score...........................18
2.8.5 混淆度...........................18
2.9 詞頻-逆向文本頻率指數...........................19
2.10 潛在語意索引...........................20
2.11 隱含狄利克雷分布...........................20
2.12 提示學習...........................21
2.13 LLaMA 2...........................21
2.14 相關研究...........................22
第三章 研究方法與步驟...........................23
3.1 資料收集模組...........................24
3.1.1 資料來源...........................24
3.1.2 資料篩選...........................25
3.2 專利領域預訓練模組...........................26
3.2.1 訓練資料...........................26
3.2.2 訓練方式...........................27
3.3 專利類別預測訓練模組...........................28
3.3.1 訓練資料...........................28
3.3.2 訓練方式...........................29
3.4 專利要件預測訓練模組...........................31
3.4.1 訓練資料...........................31
3.4.2 資料標記...........................32
3.4.3 訓練方式...........................33
3.5 專利文本生成訓練模組...........................34
3.5.1 資料生成...........................34
3.5.2 訓練方式...........................35
3.5.3 驗證方式...........................35
3.6 專利文本比對訓練模組...........................37
3.6.1 訓練資料...........................37
3.6.2 Jieba斷詞...........................38
3.6.3 移除停用字...........................38
3.6.4 SVD分解...........................39
第四章 研究結果...........................40
4.1 領域預訓練模組...........................40
4.2 專利類別預測模型...........................41
4.3 專利要件預測模型...........................43
4.4 專利文本生成模型...........................48
4.5 專利文本比對模型...........................49
第五章 系統應用展示...........................53
5.1 系統介面...........................53
5.2 系統功能...........................54
5.2.1 專利類別預測...........................54
5.2.2 專利三大要件預測...........................55
5.2.3 專利文本生成...........................56
5.2.4 專利相似度比對...........................57
5.2.5 歷史資料查詢...........................58
第六章 結論...........................60
第七章 未來展望...........................61
參考文獻...........................62
Extended Abstract...........................64
1. INTRODUCTION...........................65
2. BACKGROUND...........................65
3. METHOD...........................66
3.1 DCM...........................66
3.3 PCPTM...........................67
3.4 PPTM...........................68
3.5 PTGTM...........................69
3.6 PTCTM...........................70
4. Result...........................70
4.1 Patent Domain Pre-Trained BERT...........................70
4.2 Patent Category Prediction BERT...........................71
4.3 Patentability Prediction BERT...........................71
4.4 Patent Text Generation Model...........................73
4.5 Patent Text Comparison Model...........................74
5. Application...........................74
5.1 System Interface...........................75
5.2 System Functions...........................75
6. CONCLUSION...........................77
7. REFERENCE...........................77

[1]Yenduri, G., Srivastava, G., Maddikunta, P. K. R., Jhaveri, R. H., Wang, W., Vasilakos, A. V., & Gadekallu, T. R. (2023). Generative pre-trained transformer: A comprehensive review on enabling technologies, potential applications, emerging challenges, and future directions. arXiv preprint arXiv:2305.10435.
[2]Asudani, D. S., Nagwani, N. K., & Singh, P. (2023). Impact of word embedding models on text analytics in deep learning environment: a review. Artificial intelligence review, 56(9), 10345-10425.
[3]經濟部智慧財產局. 中華民國專利資訊檢索系統. from https://twpat1.tipo.gov.tw/twpatc/twpatkm
[4]經濟部智慧財產局. 認識專利. from https://topic.tipo.gov.tw/patents-tw/cp-784-873246-9d957-101.html
[5]Adamopoulou, E., & Moussiades, L. (2020). An overview of chatbot technology. Paper presented at the IFIP international conference on artificial intelligence applications and innovations.
[6]Zhang, C., & Lu, Y. (2021). Study on artificial intelligence: The state of the art and future prospects. Journal of Industrial Information Integration, 23, 100224.
[7]Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
[8]Chowdhary, K., & Chowdhary, K. (2020). Natural language processing. Fundamentals of artificial intelligence, 603-649.
[9]Lee, S., Jang, H., Baik, Y., Park, S., & Shin, H. (2020). Kr-bert: A small-scale korean-specific language model. arXiv preprint arXiv:2008.03979.
[10]Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., . . . Xu, Y. (2022). A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45(1), 87-110.
[11]Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., . . . Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30.
[12]Wettig, A., Gao, T., Zhong, Z., & Chen, D. (2022). Should you mask 15% in masked language modeling? arXiv preprint arXiv:2202.08005.
[13]Yang, M. (2020). ckiplab/bert-base-chinese. from https://huggingface.co/ckiplab/bert-base-chinese
[14]Yang, M. (2020). ckiplab/albert-base-chinese. from https://huggingface.co/ckiplab/albert-base-chinese
[15]Iz Beltagy, M. E. P., Arman Cohan. (2020). Longformer: The Long-Document Transformer. from https://huggingface.co/transformers/v3.0.2/model_doc/longformer.html
[16]Yang, M. (2020). bert-base-chinese. from https://huggingface.co/bert-base-chinese
[17]Beltagy, I., Peters, M. E., & Cohan, A. (2020). Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
[18]Nasteski, V. (2017). An overview of the supervised machine learning methods. Horizons. b, 4(51-62), 56.
[19]James, G., Witten, D., Hastie, T., Tibshirani, R., & Taylor, J. (2023). Unsupervised learning An Introduction to Statistical Learning: with Applications in Python (pp. 503-556): Springer.
[20]Yang, X., Song, Z., King, I., & Xu, Z. (2022). A survey on deep semi-supervised learning. IEEE Transactions on Knowledge and Data Engineering.
[21]Kumari, R., & Srivastava, S. K. (2017). Machine learning: A review on binary classification. International Journal of Computer Applications, 160(7).
[22]Grandini, M., Bagli, E., & Visani, G. (2020). Metrics for multi-class classification: an overview. arXiv preprint arXiv:2008.05756.
[23]Tan, Q., Yu, Y., Yu, G., & Wang, J. (2017). Semi-supervised multi-label classification using incomplete label information. Neurocomputing, 260, 192-202.
[24]Roelofs, R., Shankar, V., Recht, B., Fridovich-Keil, S., Hardt, M., Miller, J., & Schmidt, L. (2019). A meta-analysis of overfitting in machine learning. Advances in Neural Information Processing Systems, 32.
[25]Düntsch, I., & Gediga, G. (2019). Confusion matrices and rough set data analysis. Paper presented at the Journal of Physics: Conference Series.
[26]Havrlant, L., & Kreinovich, V. (2017). A simple probabilistic explanation of term frequency-inverse document frequency (tf-idf) heuristic (and variations motivated by this explanation). International Journal of General Systems, 46(1), 27-36.
[27]Hofmann, T. (1999). Probabilistic latent semantic indexing. Paper presented at the Proceedings of the 22nd annual international ACM SIGIR conference on Research and development in information retrieval.
[28]Brand, M. (2006). Fast low-rank modifications of the thin singular value decomposition. Linear Algebra and its Applications, 415(1), 20-30. doi: https://doi.org/10.1016/j.laa.2005.07.021
[29]Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of machine Learning research, 3(Jan), 993-1022.
[30]Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9), 1-35.
[31]Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., . . . Bhosale, S. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
[32]Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., & Sanghai, S. (2023). Gqa: Training generalized multi-query transformer models from multi-head checkpoints. arXiv preprint arXiv:2305.13245.
[33]蔡孟純. (2023). 基於BERT模型的專利相似度計算：以台灣金融科技專利為例. (碩士), 國立政治大學, 台北市. Retrieved from https://hdl.handle.net/11296/m6jxay 臺灣博碩士論文知識加值系統 database.
[34]吳柏成. (2022). 以BERT為基之中文文件相似度計算—應用於專利文件之分類與分群. (碩士), 國立屏東大學, 屏東縣. Retrieved from https://hdl.handle.net/11296/awwk2z 臺灣博碩士論文知識加值系統 database.
[35]戴余修. (2021). 基於BERT預訓練模型的專利檢索方法. (碩士), 國立臺灣大學, 台北市. Retrieved from https://hdl.handle.net/11296/6sc56k 臺灣博碩士論文知識加值系統 database.
[36]PyPI. Selenium. from https://pypi.org/project/selenium/
[37]經濟部智慧財產局. 專利公開資訊查詢. from https://tiponet.tipo.gov.tw/S092_OUT/out