臺灣博碩士論文加值系統

在人工智慧領域中，時間序列預測一直是一個熱門的討論議題。其目的是從大量的時間序列資料中去尋找規律，並從資料學習資料中的規則，藉由學習到的規律進行未來時間序列的預測。本論文的研究提出使用到TCN (Temporal Convolution Network: 時間卷積網路)以及BLS (Broad Learning System: 寬度學習系統)這兩種技術。本論文透過三個章節了解TCN在時間序列預測上的應用、TCN的可解釋性，以及結合BLS與TCN兩種技術的運用。相比常見的LSTM(Long Short-Term Memory: 長短期記憶模型)與GRU(Gate Recurrent Units: 門控循環單元)，TCN能更好地處理時間序列的問題。在時間序列預測的應用中，本論文使用兩種資料集進行測試，分別是美國的股票和台灣的天氣。使用上述兩種不同資料型態的資料集進行測試，並了解TCN的表現。在TCN的可解釋性中，本論文透過調整各種不同的超參數，並透過相關可解釋技術來了解如何正確使用超參數來最佳化TCN的訓練結果。以便能更好地使用TCN這個模型。本論文結合BLS與TCN兩種技術，透過BLS的特徵提取層替換為TCN，結合BLS與TCN兩種技術的能力，以增強BLS在時間序列預測的能力。最後，本論文總結可解釋TCN的研究結果與BLS與TCN在時間序列預測的應用與能力。

In the field of artificial intelligence, time series prediction has always been a popular topic of discussion. Its purpose is to find patterns in large amounts of time series data and learn the rules within the data, using the learned patterns to predict future time series. This thesis research employs two techniques, TCN (Temporal Convolution Network) and BLS (Broad Learning System). The thesis explores the application of TCN in time series prediction, the explainable of TCN, and the combination of BLS and TCN through three chapters. Compared to common LSTM (Long Short-Term Memory) and GRU (Gate Recurrent Units), TCN can better handle time series problems. In the time series prediction applications, this thesis tests two datasets, namely US stocks and Taiwan weather. This thesis uses these two datasets with different data types to test and understand TCN's performance. In the explainable of TCN, this thesis adjusts various hyperparameters and use related explainable techniques to understand how to correctly use hyperparameters to optimize TCN's training results, thereby better utilize the TCN model. This thesis proposes to combine BLS and TCN techniques by replacing the feature extraction layer of BLS with TCN, combining the capabilities of BLS and TCN to enhance BLS's ability in time series prediction. Finally, this thesis summarizes the research results of explainable TCN and the application and capabilities of BLS and TCN in time series prediction.

摘要...i
Abstract...iii
誌謝...v
Table of Contents...vi
List of Tables...viii
List of Figures...xi
Chapter 1 Introduction...1
1.1 Temporal Convolutional Network (TCN)...1
1.2 Broad Learning System (BLS)...2
1.3 Explainable AI (XAI)...3
1.4 Motivations and Contributions...4
1.5 Organization of This Thesis...5
Chapter 2 Mathematical Background and Definition of Keywords...6
2.1 Root Mean Squared Error...6
2.2 1D Convolution...6
2.3 Causal Convolution...8
2.4 Dilated Convolution...9
2.5 Residual Connection...10
2.6 ReLU...11
2.7 Adam Optimizer...11
2.8 Dropout...12
Chapter 3 Introduction to the Application of TCN and Dataset Description...14
3.1 The Architecture of TCN...14
3.2 System Model Architecture...16
3.3 Flow Chart...17
3.4 Experimental Environment...18
3.5 The Test Results of the Weather Dataset...18
3.5.1 Sequence to Point (near)...26
3.5.2 Sequence to Point (future)...32
3.5.3 Sequence to Sequence (near)...38
3.5.4 Sequence to Sequence (future)...43
3.6 The Test Results of the Stock Dataset...48
3.6.1 Sequence to Point (near)...53
3.6.2 Sequence to Point (future)...58
3.6.3 Sequence to Sequence (near)...63
3.6.4 Sequence to Sequence (future)...68
3.8 Concluding Remarks...74
Chapter 4 The Explainable AI of Temporal Convolutional Network...75
4.1 Dilated Causal Convolution...75
4.1.1 Receptive Field...75
4.1.2 Dilations is 1...78
4.1.3 Dilations is 2...81
4.1.4 Dilations is 4...83
4.2 Local Interpretable Model-Agnostic Explanations (LIME)...85
4.3 SHapley Additive exPlanations (SHAP)...87
4.4 Structural Explainable...89
4.4.1 Weather Dataset...90
4.4.2 Stock Dataset...96
4.5 Concluding Remarks...102
Chapter 5 Application of Explainable TCN in BLS...103
5.1 BLS Structure...103
5.2 Original BLS Regression Flow Chart...105
5.3 Proposed Method...105
5.4 TCN in BLS Regression Flow Chart...107
5.5 Experimental Results...108
5.5.1 Weather Dataset...108
5.5.2 Stock Dataset...110
5.6 Concluding Remarks...113
Chapter 6 Conclusions and Future Research...114
6.1 Conclusions...114
6.2 Future Works...114
References...115
Extended Abstract...120

[1] S. Bai, J. Z. Kolter and V. Koltun, 2018, “An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling.” arXiv1803.01271, Mar.
[2] C. L. P. Chen and Z. Liu, 2018, “Broad Learning System: An Effective and Efficient Incremental Learning System Without the Need for Deep Architecture.” IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 1, pp. 10-24, Jan, doi: 10.1109/TNNLS.2017.2716952.
[3] C. L. P. Chen, Z. Liu and S. Feng, 2019, “Universal Approximation Capability of Broad Learning System and Its Structural Variations.” IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 4, pp. 1191-1204, April, doi: 10.1109/TNNLS.2018.2866622.
[4] M. T. Ribeiro, S. Singh and C. Guestrin, 2016, “'Why Should I Trust You?': Explaining the Predictions of Any Classifier.” in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA, pp. 1135–1144. doi: 10.1145/2939672.2939778.
[5] S. M. Lundberg and S.-I. Lee, 2017, “A Unified Approach to Interpreting Model Predictions.” in Proceedings of the 31st International Conference on Neural Information Processing Systems, Red Hook, NY, USA, pp. 4768–4777.
[6] H. -J. Chang and J. -T. Jen, 2022, "Comparing the Prediction Accuracy with Passenger Time Series of Taipei MRT under Different Deep RNNs." in 2022 IET International Conference on Engineering Technologies and Applications, Changhua, Taiwan, pp. 1-2, doi: 10.1109/IET-ICETA56553.2022.9971668.
[7] S. Hochreiter and J. Schmidhuber, 1997, “Long Short-Term Memory.” Neural Computation, vol. 9, no. 8, pp. 1735-1780, Nov. 15, doi: 10.1162/neco.1997.9.8.1735.
[8] J. Chung, C. Gulcehre, K. Cho and Y. Bengio, 2014, “Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling.” arXiv1412.3555, Dec.
[9]CWB Observation Data Inquire Service, July 20, 2023. [Online]. Available: https://codis.cwb.gov.tw/StationData
[10]Yahoo finance, July 20, 2023. [Online]. Available: https://finance.yahoo.com/quote/0050.TW/history?p=0050.TW
[11]Yahoo finance, July 20, 2023. [Online]. Available: https://finance.yahoo.com/quote/%5EDJI?p=^DJI&.tsrc=fin-srch
[12] F. Doshi-Velez and B. Kim, 2017, “Towards a Rigorous Science of Interpretable Machine learning.” arXiv1702.08608, Feb.
[13]Explainable Artificial Intelligence (XAI), July 20, 2023. [Online]. Available: https://www.darpa.mil/program/explainable-artificial-intelligence
[14] G. E. P. Box and G. Jenkins, 1990, Time Series Analysis, Forecasting and Control. Holden-Day, Inc., USA.
[15] S. M. Idrees, M. A. Alam and P. Agarwal, 2019, "A Prediction Approach for Stock Market Volatility Based on Time Series Data." IEEE Access, vol. 7, pp. 17287-17298, doi: 10.1109/ACCESS.2019.2895252.
[16] Y. Lv, Y. Duan, W. Kang, Z. Li and F. -Y. Wang, 2015, "Traffic Flow Prediction with Big Data: A Deep Learning Approach." IEEE Transactions on Intelligent Transportation Systems, 16(2), 865-873. https://doi.org/10.1109/TITS.2014.2345663.
[17] M. Yu, F. Xu, W. Hu, J. Sun and G. Cervone, 2021, "Using Long Short-Term Memory (LSTM) and Internet of Things (IoT) for Localized Surface Temperature Forecasting in an Urban Environment." IEEE Access, 9, 137406-137418. https://doi.org/10.1109/ACCESS.2021.3116809.
[18] K. He, X. Zhang, S. Ren and J. Sun, 2016, "Deep Residual Learning for Image Recognition." Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 770-778. https://doi.org/10.1109/CVPR.2016.90.
[19] C. L. P. Chen and J. Z. Wan, 1999, "A Rapid Learning and Dynamic Stepwise Updating Algorithm for Flat Neural Networks and the Application to Time-Series Prediction." IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 29(1), 62-72. https://doi.org/10.1109/3477.740166.
[20] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi, 2019, "A Survey of Methods for Explaining Black Box Models." ACM Comput. Surv., 51(5), Article 93. https://doi.org/10.1145/3236009.
[21] A. Adadi and M. Berrada, 2018, "Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)." IEEE Access, 6, 52138-52160. https://doi.org/10.1109/ACCESS.2018.2870052.
[22] L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter and L. Kagal, 2018, "Explaining Explanations: An Overview of Interpretability of Machine Learning." Proceedings of the 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), Turin, Italy, 80-89. https://doi.org/10.1109/DSAA.2018.00018.
[23] T. Chai and R. R. Draxler, 2014, "Root Mean Square Error (RMSE) or Mean Absolute Error (MAE)?– Arguments Against Avoiding RMSE in the Literature." Geosci. Model Dev., 7, 1247–1250. https://doi.org/10.5194/gmd-7-1247-2014.
[24] F. Yu and V. Koltun, 2015, "Multi-Scale Context Aggregation by Dilated Convolutions." arXiv preprint arXiv:1511.07122.
[25] W. Luo, Y. Li, R. Urtasun and R. Zemel, 2016, "Understanding the Effective Receptive Field in Deep Convolutional Neural Networks." Advances in Neural Information Processing Systems, 29.
[26] H. Li, Z. Xu, G. Taylor, C. Studer and T. Goldstein, 2018, "Visualizing the Loss Landscape of Neural Nets." Advances in Neural Information Processing Systems, 31.
[27] A. Veit, M. J. Wilber and S. Belongie, 2016, "Residual Networks Behave Like Ensembles of Relatively Shallow Networks." Advances in neural information processing systems, 29.
[28] V. Nair and G. E. Hinton, 2010, "Rectified Linear Units Improve Restricted Boltzmann Machines." Proceedings of the 27th International Conference on International Conference on Machine Learning (ICML'10). Omnipress, Madison, WI, USA, 807–814.
[29] B. Xu, N. Wang, T. Chen and M. Li, 2015, "Empirical Evaluation of Rectified Activations in Convolutional Network." arXiv preprint arXiv:1505.00853.
[30] D. P. Kingma and J. Ba, 2014, "Adam: A Method for Stochastic Optimization." arXiv preprint arXiv:1412.6980.
[31] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov, 2014, "Dropout: A Simple Way to Prevent Neural Networks from Overfitting." The journal of machine learning research, 15(1), 1929-1958.
[32] C. Zhang, S. Bengio, M. Hardt, B. Recht and O. Vinyals, 2021, "Understanding Deep Learning (still) Requires Rethinking Generalization." Communications of the ACM, 64(3), 107-115.
[33] Unit8. "Temporal Convolutional Networks and Forecasting." Accessed from https://unit8.com/resources/temporal-convolutional-networks-and-forecasting/.
[34] Þórírmarsdóttir, L. "Temporal Convolutional Networks." Accessed from https://thorirmar.com/post/temporal_convolutional_networks/.
[35] A. V. D. Oord, et al., 2016, "Wavenet: A Generative Model for Raw Audio." arXiv preprint arXiv:1609.03499.
[36] M. Tan, S. Yuan, S. Li, Y. Su, H. Li and F. He, 2020, “Ultra-Short-Term Industrial Power Demand Forecasting Using LSTM Based Hybrid Ensemble Learning.” IEEE Transactions on Power Systems, vol. 35, no. 4, pp. 2937-2948, July, doi: 10.1109/TPWRS.2019.2963109.
[37] Q. Zhang, H. Wang, J. Dong, G. Zhong and X. Sun, 2017, “Prediction of Sea Surface Temperature Using Long Short-Term Memory.” IEEE Geoscience and Remote Sensing Letters, vol. 14, no. 10, pp. 1745-1749, Oct., doi: 10.1109/LGRS.2017.2733548.
[38] A. H. Bukhari, M. A. Z. Raja, M. Sulaiman, S. Islam, M. Shoaib and P. Kumam, 2020, “Fractional Neuro-Sequential ARFIMA-LSTM for Financial Market Forecasting.” IEEE Access, vol. 8, pp. 71326-71338, doi: 10.1109/ACCESS.2020.2985763.
[39] Q. Tao, F. Liu, Y. Li and D. Sidorov, 2019, “Air Pollution Forecasting Using a Deep Learning Model Based on 1D Convnets and Bidirectional GRU.” IEEE Access, vol. 7, pp. 76690-76698, doi: 10.1109/ACCESS.2019.2921578.
[40] C. Li, G. Tang, X. Xue, A. Saeed, and X. Hu, 2020, “Short-Term Wind Speed Interval Prediction Based on Ensemble GRU Model.” IEEE Transactions on Sustainable Energy, vol. 11, no. 3, pp. 1370-1380, July, doi: 10.1109/TSTE.2019.2926147.
[41] M. Kuzlu, U. Cali, V. Sharma and Ö. Güler, 2020, “Gaining Insight into Solar Photovoltaic Power Generation Forecasting Utilizing Explainable Artificial Intelligence Tools.” IEEE Access, vol. 8, pp. 187814-187823, doi: 10.1109/ACCESS.2020.3031477.
[42] X. -H. Li et al., 2022, “A Survey of Data-Driven and Knowledge-Aware eXplainable AI.” IEEE Transactions on Knowledge and Data Engineering, vol. 34, no. 1, pp. 29-49, Jan, doi: 10.1109/TKDE.2020.2983930.
[43] F. -L. Fan, J. Xiong, M. Li and G. Wang, 2021, “On Interpretability of Artificial Neural Networks: A Survey.” IEEE Transactions on Radiation and Plasma Medical Sciences, vol. 5, no. 6, pp. 741-760, Nov, doi: 10.1109/TRPMS.2021.3066428.
[44] F. Yu and V. Koltun, 2015,"Multi-scale Context Aggregation by Dilated Convolutions." arXiv:1511.07122.
[45] E. Štrumbelj and I. Kononenko, 2014, “Explaining Prediction Models and Individual Predictions with Feature Contributions.” Knowl Inf. Syst., vol. 41, pp. 647-665, [Online]. Available: https://doi.org/10.1007/s10115-013-0679-x.
[46] Y. H. Pao, G. H. Park and D. J. Sobajic, 1994, "Learning and Generalization Characteristics of the Random Vector Functional-link Net." Neurocomputing, vol. 6, no. 2, pp. 163-180.
[47] B. Igelnik and Y. H. Pao, 1995, "Stochastic Choice of Basis Functions in Adaptive Function Approximation and the Functional-link Net." IEEE Transactions on Neural Networks, vol. 6, no. 6, pp. 1320-1329.
[48] Y. Lecun, L. Bottou, Y. Bengio and P. Haffner, 1998, "Gradient-Based Learning Applied to Document Recognition." Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, Nov, doi: 10.1109/5.726791.
[49] W. Luo, Y. Li, R. Urtasun, and R. Zemel, 2016, “Understanding the Effective Receptive Field in Deep Convolutional Neural Networks.” Advances in Neural Information Processing Systems, vol. 29.
[50] H. Robbins and S. Monro, 1951, "A Stochastic Approximation Method." The Annals of Mathematical Statistics, pp. 400-407.
[51] J. Duchi, E. Hazan and Y. Singer, 2011, “Adaptive Subgradient Methods for Online Learning and Stochastic Optimization.” Journal of Machine Learning Research, vol. 12, no. 7.
[52] G. Hinton, “Neural Networks for Machine Learning.” Coursera course slides, [Online].