臺灣博碩士論文加值系統

本文首先提出了雙向巢狀LSTM模型。雙向LSTM神經網路是由兩組獨立的LSTM神經網路結合而成，有時預測除了需要經由過去或前面的資料來決定之外，也會需要經由未來或之後的資訊來共同決定，透過雙向之結構，經由輸入序列正向和反向輸入將模型訓練的更加準確。此模型亦使用巢狀結構，由另一個完整的LSTM取代LSTM中的記憶單元，使得模型擁有了更好的長期依賴性。第二，由於LSTM在長時間跨度的時間序列預測表現良好以及SVM具有強大的泛化能力與強健性，過去的研究引入了SVM於LSTM，用於分類以完成預測結果。然而SVM在不平衡數據分類時，超平面會向少數類移位或傾斜，這種偏移現象會使得模型在少數類別中的表現降低。因此我們嘗試使用Bidirectional Nested LSTM RF模型結合SMOTE過採樣、OSS欠採樣以及隨機森林算法，來改善SVM分類器在數據不平衡時對少數類分類表現降低的問題。由於SMOTE過採樣與OSS欠採樣對數據類別提供平衡機制使分類性能得到提升。而隨機森林模型中，每顆決策樹判斷到的特徵與樣本的抽樣皆不同，使每顆決策樹之間的相關性減少，並且彼此間相互獨立。此特性也使得隨機森林在應用於不平衡數據時，較能避免少數類表現降低的情況。本文以天氣數據為例，訓練所使用的天氣資料取自Kaggle的New York City - Hourly Weather Data，並透過均方根誤差（RMSE）以評估預測結果的性能。此研究結果表明雙向巢狀LSTM模型在相同的參數數量下與其他LSTM模型相比擁有更高的預測準確率。此外，結合SVM後的Bidirectional Nested LSTM SVM精準度仍然優於Bidirectional Nested LSTM RF。

This paper first proposes a bidirectional nested LSTM model. The bidirectional LSTM neural network is a combination of two independent LSTM neural networks. Sometimes it is predicted that it needs to be determined by past or previous data, and it needs to be determined jointly by future or subsequent information. The structure, the model training is more accurate via the input sequence forward and reverse inputs. This model also uses a nested structure, replacing the memory cells in the LSTM with another complete LSTM, giving the model a better long-term dependence. Second, because LSTM performs well in long-term time series predictions and SVM has strong generalization and robustness, past research has introduced SVM in LSTM for classification to complete prediction results. However, when the SVM is unbalanced, the hyperplane will shift or tilt to a few classes. This offset will reduce the performance of the model in a few categories. Therefore, we try to use the Bidirectional Nested LSTM RF model combined with SMOTE oversampling, OSS undersampling and random forest algorithm to improve the SVM classifier's performance degradation for a few classes when data is unbalanced. As SMOTE oversampling and OSS undersampling provide a balancing mechanism for data categories, classification performance is improved. In the random forest model, the features judged by each decision tree are different from the sample samples, so that the correlation between each decision tree is reduced and independent of each other. This feature also makes it easier for random forests to avoid the degradation of a few classes when applied to unbalanced data. In this paper, weather data is used as an example. The weather data used in the training is taken from Kaggle's New York City - Hourly Weather Data, and the root mean square error (RMSE) is used to evaluate the performance of the prediction results. The results of this study show that the bidirectional nested LSTM model has higher prediction accuracy than other LSTM models with the same number of parameters. In addition, the Bidirectional Nested LSTM SVM combined with SVM is still superior to Bidirectional Nested LSTM RF.

中文摘要 …………………………………………………………..... i
英文摘要 …………………………………………………………..... ii
誌謝 …………………………………………………………..... iii
目錄 …………………………………………………………..... iv
表目錄 …………………………………………………………..... v
圖目錄 …………………………………………………………..... vi
第一章 緒論…………………………………………………......... 1
第二章 長短期記憶模型理論…………………………………..... 3
2.1 遞歸神經網路……………………………………............. 3
2.2 長短期記憶模型………………………………………..... 5
第三章 長短期記憶模型進展…………………………………..... 13
3.1 堆疊長短期記憶……………………………..................... 13
3.2 雙向長短期記憶………………………………................. 14
3.3 巢狀長短期記憶……………………………………......... 15
3.4 雙向堆疊長短期記憶…………………………………..... 17
3.5 雙向巢狀長短期記憶…………………………………..... 19
第四章 雙向長短期記憶結合隨機森林模型…………………..... 22
4.1 合成少數類過採樣技術……………………………......... 22
4.2 單面選擇………………………………............................. 22
4.3 支持向量機……………………………………................. 23
4.4 隨機森林…………………………………......................... 26
4.5 雙向巢狀長短期記憶結合隨機森林…………………..... 27
第五章 實驗結果與討論…………………………………………. 28
5.1 溫度預測結果分析………………………………............. 29
5.2 露點預測結果分析……………………………………..... 32
5.3 濕度預測結果分析……………………………………..... 35
5.4 氣壓預測結果分析………………………………...…...... 38
5.5 降雨預測結果分析………………………………...…...... 41
第六章 結論與未來研究方向……………………………………. 42
參考文獻 …………………………………………………………..... 43
Extended Abstract……………………………………………………………. 46

[1] Muhammad Amri Yahya, et al., 2018, "Short-Term Electric Load Forecasting Using Recurrent Neural Network (Study Case of Load Forecasting in Central Java and Special Region of Yogyakarta)", 2018 4th International Conference on Science and Technology (ICST), pp. 1-6, August.
[2] Yuanyuan Wang, et al., 2017, "Water Quality Prediction Method Based on LSTM Neural Network", 2017 12th International Conference on Intelligent Systems and Knowledge Engineering (ISKE), pp. 24-26, November.
[3] Songyang Zhang, et al., 2018, "Fusing Geometric Features for Skeleton-Based Action Recognition Using Multilayer LSTM Networks", IEEE Transactions on Multimedia, Volume: 20, Issue:9, pp. 2330-2343, September.
[4] Khaled A. Althelaya, et al., 2018, "Evaluation of Bidirectional LSTM for Short- and Long-Term Stock Market Prediction", 2018 9th International Conference on Information and Communication Systems (ICICS), pp. 151-156, April.
[5] Li Li, et al., 2013, "Freeway Travel-Time Estimation Based on Temporal–Spatial Queueing Model", IEEE Transactions on Intelligent Transportation Systems , vol. 14, no. 3, pp. 1536-1541, September.
[6] Joel Ruben Antony Moniz, David Krueger, 2017, "Nested LSTMs", Proceedings of the Ninth Asian Conference on Machine Learning, PMLR 77, pp. 530-544, January
[7] Tyler Wilson, Pang-Ning Tan, Lifeng Luo, 2018, "A Low Rank Weighted Graph Convolutional Approach to Weather Prediction", 2018 IEEE International Conference on Data Mining, pp. 17-20, November.
[8] 迴圈神經網路學習筆記:基礎理論，https://codertw.com/%E7%A8%8B%E5%BC%8F%E8%AA%9E%E8%A8%80/621114/
[9] 淺談遞歸神經網路 (RNN) 與長短期記憶模型 (LSTM)，https://medium.com/@tengyuanchang/%E6%B7%BA%E8%AB%87%E9%81%9E%E6%AD%B8%E7%A5%9E%E7%B6%93%E7%B6%B2%E8%B7%AF-rnn-%E8%88%87%E9%95%B7%E7%9F%AD%E6%9C%9F%E8%A8%98%E6%86%B6%E6%A8%A1%E5%9E%8B-lstm-300cbe5efcc3
[10] Baigal tugsSanjaa, Erdenebat Chuluun, 2013, "Malware Detection Using Linear SVM", Ifost, Pages: 136 – 138, July.
[11] Haibo He, Yunqian Ma, 2013, "Imbalanced Learning：Foundations Algorithms and Applications", Wiley-IEEE Press, pp.216, 2013.
[12] M. Kubat, Stan Matwin, 2000, "Addressing the Curse of Imbalanced Training Sets: One-Sided Selection." , Fourteenth International Conference on Machine Learning, 2000.
[13] Liliya Demidova, Irina Klyueva, 2017, "SVM classification: Optimization with the SMOTE algorithm for the class imbalance problem ", 2017 6th Mediterranean Conference on Embedded Computing (MECO), pp. 1-4, June.
[14] Yoga Pristyanto, Irfan Pratama, Anggit Ferdita Nugraha, 2018, "Data Level Approach for Imbalanced Class Handling on Educational Data Mining Multiclass Classification", 2018 International Conference on Information and Communications Technology (ICOIACT), pp. 310-314, March.
[15] Xin Li, et al., 2018, "Overdue Prediction of Bank Loans Based on LSTM-SVM", 2018 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation, pp. 1859-1863, October.
[16] Yuanli Wu, Hong Wang, Fei Wu , 2017, "Automatic Classification of Pulmonary Tuberculosis and Sarcoidosis based on Random Forest", 2017 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pp. 1-5, October.
[17]K. Veropoulos, C. Campbell, and N. Cristianini, 1999, "Controlling the sensitivity of support vector machines", Proceedings of the International Joint Conference on Artificial Intelligence (Stockholm, Sweden), pp. 55–60, 1999.
[18] G. Wu and E. Chang, "Adaptive feature-space conformal transformation for imbalanced-data learning", Proceedings of the 20th International Conference on Machine Learning (Washington, DC), pp. 816–823, 2003.
[19] R. Akbani, S. Kwek, N. Japkowicz, 2004, "Applying support vector machines to imbalanced datasets", Proceedings of the 15th European Conference on Machine Learning, pp. 39–50, 2004.
[20] Mohammed Khalilia, Sounak Chakraborty, Mihail Popescu, 2011, "Predicting disease risks from highly imbalanced data using random forest", BMC Med Inform Decis Mak, 2011.
[21] Y. Yan, et al., 2019, "A Parameter-Free Cleaning Method for SMOTE
in Imbalanced Classification", IEEE Access, Volume: 7, pp. 23537-23548, February.
[22]New York city taxi trip hourly weather data, www.kaggle.com/meinertsen/new-york-city-taxi-trip-hourly-weather-data/home
[23] 機器學習演算法-隨機森林(Random Forest), http://hn28082251.blogspot.com/2018/07/random-forest.html
[24] J. Zhou, Y. Lu, et al., "Sentiment Analysis of Chinese Microblog Based on Stacked Bidirectional LSTM", IEEE Access, vol. 7, pp. 38856-38866, 2019.
[25] 趙琨，2010，非標準支持向量機，對外經濟貿易大學出版社，北京。
[26] Vapnik, V. N., 1995, The nature of statistical learning theory, Springer, New York.
[27] 張俊妮，2009，數據挖掘與應用，北京大學出版社，北京。
[28] Leo Breiman, "Random Forests", Machine Learning, Volume 45, Issue 1, pp. 5–32, 2001.