數據化時代來臨，各種分析技術及工具因應而出。但多數企業或研究人員大部分多專注於各種分析技術及模型，少有人關注資料集的品質。資料探勘為巨量資料分析中一個關鍵的過程。資料探勘透過資料的清理、整合、精簡、轉換，將資料準備就緒後，接下來經過挖礦的模式與方法得到有價值的資訊。其中清理、整合、精簡、轉換四步驟稱為「資料前處理(Data Preprocessing) 」，為影響資料集品質最深遠也最重要步驟。
本論文以社群及新聞計算後的情緒分數來預測大盤漲跌為例子，利用實例呈現資料探勘的過程及方法。整個研究的過程中，「資料前處理」作業佔了整個研究的百分之六十九，為研究中最繁瑣及費時的過程。該實例首先，透過python爬蟲程式蒐集社群及新聞資料。接著，執行資料前處理、中文斷詞處理、以情緒分析方法取得情緒分數並產生建模資料集。最後，以倒傳遞類神經網路演算法建立預測模型、評估及比較。經由實驗結果證實兩點，第一，資料集經由正規化及離群值處理產生的預測模型，準確度有明顯的改善；第二，使用情緒分數的模型對於預測股票上漲準確度較佳。

The era of big data is coming, and various analysis techniques and tools are produced. However, most companies or researchers mostly focus on various analysis techniques and models, and few people pay attention to the quality of data sets. Data Mining is a key process in the analysis of Big Data. Data Mining prepares data through the procedures of data cleaning, integration, reduction and transformation. Next, through mining models and methods to obtain valuable information. Among them, the four steps of cleaning, integration, reduction and transformation are called "Data Preprocessing", which are the most profound and important steps that affect the quality of the data set.
This paper takes the sentiment score calculated by the community and news as an example to predict the market's ups and downs, and uses examples to present the process and method of Data Mining.
In the entire research process, the "Data Preprocessing" operation accounted for 69% of the entire research, which was the most tedious and time-consuming process in the research. In this example, first of all, collect the community and news information through the python crawler program. Next, perform Data Preprocessing, Chinese Word-Segmentation Processing, Sentiment Analysis to obtain sentiment score, and generate modeling data sets. Finally, the Back-Propagation Neural Network Algorithm is used to build a predictive model, evaluate and compare. The experimental results confirmed two points. First, the accuracy of the prediction model generated by the data set through normalization and outlier processing has been significantly improved; second, the model using sentiment scores has better accuracy in predicting stock rises.

中文摘要.......................................................i
英文摘要.......................................................ii
致 謝.......................................................iv
目 錄.......................................................v
表 目 錄.....................................................vii
圖 目 錄.....................................................ix
第一章 緒論.....................................................1
1-1 研究背景與動機..............................................1
1-2 研究目的....................................................2
1-3 研究架構....................................................2
第二章 文獻探討.................................................3
2-1 資料前處理(Data Preprocessing)..............................3
2-1-1 為什麼需要資料前處理?......................................3
2-1-2 資料前處理步驟.............................................4
2-1-3 相關係數-皮爾遜積差相關係數.................................7
2-1-4 資料類別不平衡問題(Class Imbalance Problem)................10
2-2 情緒分析(Sentiment Analysis)................................14
2-3 結巴中文斷詞系統.............................................17
2-4 類神經網路...................................................20
第三章 資料前處理.................................................23
3-1 研究架構......................................................23
3-2 研究工具簡介..................................................24
3-3 資料集與資料前處理.............................................27
3-3-1資料的蒐集....................................................27
3-3-2網路爬蟲設計與實做............................................30
3-3-3資料的整合....................................................37
3-3-4情緒的處理及資料集的產生.......................................39
3-3-5相關係數分析..................................................44
3-3-6離群值檢視....................................................47
第四章 實驗結果與分析...............................................51
4-1 實驗環境與數據..................................................51
4-2 實驗的設計......................................................53
4-3 實驗結果與分析..................................................55
4.3.1 實驗一:情緒分數於股票漲跌預測之BPN模型..........................55
4.3.2 實驗二:無情緒分數於股票漲跌預測之BPN模型........................60
4.3.3 實驗三:情緒分數拆分成正負於股票漲跌預測之BPN模型.................65
4.3.4 實驗四:情緒分數(離群值處理)於股票漲跌預測之BPN模型...............70
4.3.5 實驗五:無情緒分數(離群值處理)於股票漲跌預測之BPN模型.............76
4.3.6 實驗六:情緒分數拆分成正負(離群值處理)於股票漲跌預測之BPN模型......82
4.3.7 實驗結果分析...................................................88
第五章 結論與未來研究方向.............................................92
5-1 結論.............................................................92
5-2 未來研究方向.....................................................94
參考文獻.............................................................95

[1]Han, J., M. Kamber, and J. Pei, "資料探勘 Data Mining Concepts and Techniques 3/e. ", 高立, 新北市, 2014.
[2]簡禎富 and 許嘉裕, "資料挖礦與大數據分析 = Data mining & big data analytics. ", 前程文化, 新北市, 2014.
[3]曾新穆, "巨量資料分析前處理程序之關鍵議題及挑戰,", http://www.twcloud.org.tw/files/file_pool/1/0i235514714386617720/5.pdf (accessed June 30,2020)
[4]Pang, B. and L. Lee, 2008, "Opinion Mining and Sentiment Analysis," Foundations and Trends in Information Retrieval, 2(1-2), pp. 1–135.
[5]Pang, B., L. Lee, and S. Vaithyanathan., "Thumbs up?: sentiment classification using machine learning techniques," in Proceedings of the ACL-02 conference on Empirical methods in natural language processing, Association for Computational Linguistics ,vol. 10, pp.79-86,2002.
[6]Snyder, B. and R. Barzilay. "Multiple aspect ranking using the good grief algorithm," in Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pp.300-307,2007.
[7]張琇茵, 2018, "基於社群與新聞內容之情感分析預測股價指數趨勢," 碩士論文, 國立臺北科技大學.
[8]張偉德, 2018, "應用情感分析從媒體評論推測企業聲譽之研究," 碩士論文, 國立中央大學.
[9]許家銘,2017,"情感分析應用於社群媒體輿論分析之研究," 碩士論文, 國立中央大學.
[10]陳岳群,2014,"使用情緒分析於公眾行為預測之研究," 碩士論文,樹德科技大學,高雄市,台灣.
[11]IT閱讀, "最全中文停用詞表整理（1893個）,", https://www.itread01.com/content/1494661154.html (accessed June 30,2020)