臺灣博碩士論文加值系統

本論文研究利用Arduino擷取電力訊號之DAQ技術與Node.js處理電力訊號品質之相關演算法，探討如何設定雲端伺服器主機並調整Arduino傳輸時間間距，以適當地將電力品質相關資訊傳輸到遠端主機進行分散式儲存與運算任務。本論文以Nginx平衡負載器配合Node.js網站伺服器作為資料傳輸之網路連線入口，接受多受測地點同時傳輸；並以MongoDB作為後端雲端資料庫，以穩定執行運算與儲存相關工作；Hadoop Hive 配合Hadoop HDFS執行效能分析，使得整體系統的儲存方式、運算模式及工作分派等機制更為有效率。Hadoop及MongoDB是大數據(Big Data)處理技術中同時具備運算與儲存功能之雲端系統，因此可以建置大量(Volume)、多樣(Variety)與快速(Velocity)的電力訊號品質監測所需大數據處理功能平台，本論文研究設計雲端分散式運算平台之統計分析技術，所執行的雲端系統建置架構也可以適用目前物聯網的普遍應用環境，同時，相關系統效能分析原理也可以適用一般大數據應用可能遭遇的問題。

In this paper, by using the Arduino controller with the DAQ technology to capture the power signal, a Node.js web platform has been established to analyze power quality which was accompanied with the Nginx load-balanced portal system. Nginx and Node.js can accept request connections simultaneously from multiple Arduino sites. All the received power quality information is stored into the MongoDB database after power quality execution from Node.js. Moreover, configuring the MongoDB parameters and analyzing the Hadoop framework efficiency could make the storage approach, computing procedure and job assignment more acceptable. Since the data volume, variety, and velocity of the power quality information have significantly increased, cloud-based platforms such as Hadoop framework and MongoDB database are required to complete the power quality monitoring. The Hadoop framework and MongoDB database can provide distributed storage and computing capability for big data analysis. Meanwhile, the analysis and principle of the power quality design can be applied to general problems for big data application or IoT (Internet of Things) environment.

目錄
第一章 緒論 1
1.1 研究動機 1
1.2 問題描述 2
1.2.1電力品質監測系統運算與存取效能 2
1.2.2系統設計擴充性與可用性 3
1.2.3線上營運型資料與離線分析型資料 3
1.2.4系統設計穩健性與可靠性 4
1.3.系統設計 4
1.3.1 Arduino 電力訊號感測系統 5
1.3.2 電力品質運算平台 6
1.3.4 分散式雲端儲存與運算系統 7
1.4 章節結構 9
第二章 文獻回顧 11
2.1 電力訊號擷取設計重點 11
2.2 利用雲端運算系統平台執行電力品質統計分析 14
第三章 研究方法與系統架構 20
3.1 Apache Hadoop 系統架構 24
3.2 MongoDB雲端資料庫系統架構 26
3.3 Apache Hadoop 與 Hive 系統架構 27
3.4 Nginx負載平衡器與Node.js網站平台系統架構 29
第四章 實驗設計與結果分析 32
4.1 Node.js + Nginx處理電力品質演算法運算效能 32
4.2 Apache Hadoop + Hive與MongoDB資料查詢時間比較 35
4.3 Apache Hadoop + Hive 與 MongoDB資料寫入時間比較 37
4.4 電力訊號監測網站平台 39
第五章 未來研究 41
5.1 設計彈性切換的負載平衡系統 41
5.2 改變Apache MapReduce為 Apache Tez 處理引擎 42
5.3 調校Hive系統設定參數 43
5.4 設計MongoDB之副本叢集系統(Replica Set) 44
第六章 貢獻與結論 46
參考文獻 48
誌謝 53



圖目錄
圖1.1 電力品質監測系統運算與存取效能示意圖 3
圖1.2 系統設計垂直擴充與水平擴充示意圖 3
圖1.3 即時性資料與非即時性資料示意圖 4
圖1.4 系統設計穩健性與可靠性示意圖 4
圖1.5 Arduino電力訊號感測系統方塊圖 5
圖1.6電壓異常事件波形圖 6
圖1.7電力品質運算平台功能方塊圖 7
圖1.8 MongoDB 分散式統架構圖 8
圖1.9 Apache Hadoop YARN 2.0分散式系統架構圖 9
圖2.1 電力訊號感測網路功能設計圖 12
圖2.2 電力訊號傳輸網路功能設計圖 13
圖2.3 擷取電力訊號設計架構圖 14
圖2.4資料中心系統功能方塊圖 15
圖2.5 Hadoop MapReduce分散式運算原理 15
圖2.6 Hadoop MapReduce之電壓驟降運算流程圖 16
圖2.7智慧電網之電力訊號資料分類圖 17
圖2.8 Apache Spark運算資料-離線資料與線上資料 18
圖2.9 電力品質參數量測架構 18
圖2.10電力品質參數(PQ index)資料分類 19
圖2.11 電力品質參數三維空間分布 19
圖3.1電力訊號品質雲端系統架構 20
圖3.2電力訊號品質雲端系統功能運作流程圖 22
圖3.3 Node.js與MongoDB每日資料計算量 24
圖3.4 Node.js 與MongoDB運算處理時間關聯順序 24
圖3.5 Apache Hadoop叢集運算系統環境 25
圖3.6 Apache Hadoop HDFS之DataNode 運作狀態圖 25
圖3.7 Apache Hadoop YARN 做分派執行狀態圖 25
圖3.8 MongoDB雲端資料庫分散式讀寫示意圖 26
圖3.9 MongoDB 雲端資料庫分散式讀寫結果 27
圖3.10 Apache Hadoop 與 Hive 系統架構關係圖 28
圖3.11 Apache Hive資料儲存格式之原生格式與ORC格式轉換步驟 28
圖3.12 Nginx 與 Node.js 系統架構關係圖 29
圖3.13 同步I/O存取與非同步 I/O存取比較圖 30
圖3.14 Node.js之非同步I/O存取流程圖 31
圖4.1 Nginx平衡負載器與Arduino控制器及MongoDB資料庫連線圖 33
圖4.2 Nginx平衡負載器接收遠端Arduino控制器之電力訊號測試資料 33
圖4.3 五部Node.js網站伺服器實體主機運算時間過程 34
圖4.4 五部Node.js網站伺服器實體主機運算時間比較圖 35
圖4.5 (Apache Hadoop + Hive)統計分析之HQL指令 35
圖4.6 MongoDB統計分析指令 35
圖4.7 Apache Hive與MongoDB查詢比較時間趨勢圖 36
圖4.8 Hive ORC 資料格式轉換過程 37
圖4.9 MongoDB與Hadoop Hive ORC資料寫入時間(log值)比較趨勢 39
圖4.10電力訊號監測網站平台(即時資料) 40
圖4.11電力訊號監測網站平台(歷史資料) 40
圖5.1 Nginx 固定IP配對平衡負載示意圖 42
圖5.2 Apache Hadoop 2.0系統架構圖 42
圖5.3 Apache Hadoop 2.0 之MapReduce與Tez操作流程比較圖 43
圖5.4 MongoDB副本叢集系統主從式架構圖 45



表目錄
表3.1 Arduino prescaler 設定參考表 21
表3.2 電力訊號干擾事件類別 23
表3.3 (1)、多點電力訊號干擾類別表 (2)、多點電力訊號干擾日期 23
表4.1五部Node.js網站伺服器實體主機運算時間記錄 (單位微秒) 34
表4.2 Apache Hive與MongoDB查詢時間比較表 35
表4. 3. (ApacheHadoop+ Hive)原生資料格式與ORC資料格式資料查詢時間 37
表4.4 Apache Hive與MongoDB查詢時間比較表 37
表4.5 MongoDB與Hadoop Hive ORC資料寫入時間 38

[1]T. Q. Tran, L. E. Conrad, and B. K. Stallman, "Electric Shock and Elevated EMF Levels Due to Triplen Harmonic", IEEE Trans. on Power Delivery, Vol. 11, No. 2, April 1996, pp. 1041-1049.
[2]R. C. Dugan, M. F. McGranaghan, and H. W. Beaty, Electrical Power Systems Quality, McGraW-HILL International Editions, 2000.
[3]IEEE Recommended Practice for Monitoring Electric Power Quality, IEEE Std. 1159-2009, June 26, 2009.
[4]J. L.Willems, J. A. Ghijselen, and A. E. Emanuel, "The apparent power concept and the IEEE standard 1459-2000, " IEEE Trans. Power Del., Vol. 20, no. 2, pt. 1, pp. 876–884, Apr. 2005.
[5]IEEE Working group on nonsinusoidal situations: effects on meter performance and definitions, "Practical definitions for powers in systems with nonsinusoidal waveforms and unbalanced loads: A discussion, " IEEE Trans. Power Del., Vol. 11, no 1, pp. 79–87, Jan. 1996.
[6]IEEE standard definitions for the measurement of electric powerquantities under sinusoidal, non-sinusoidal, balanced and unbalanced conditions, IEEE Std 1459-2000, Sep. 2002.
[7]A. E. Emanuel, "Summary of IEEE standard 1459: definitions for the measurement of electric power quantities under sinusoidal, nonsinusoidal, balanced or unbalanced conditions, " IEEE Trans. Ind. Appl., Vol. 40, no. 3, pp. 869–876, 2004.
[8]J. Guo, S. Su, and C. Fu, "Research on Power Quality Monitoring System based on low-voltage PLC Technology, " Advanced Power System Automation and Protection (APAP), pp. 0–3., 2011.
[9]D.Sivakumar, J.P.Srividhya, and T.Shanmathi, "A Review on Power Quality Monitoring and Its Controlling Techniques," 8th International Conference on Latest Trends in Engineering and Technology (ICLTET'2016), pp. 13-19, 2016.
[10]D. Cai1, H. Tian, Y. Wang, H. Wang, H. Zheng, K. Cao and C. Zhou, "Electric Power Big Data and Its Applications," International Conference on Energy, Power and Electrical Engineering (EPEE 2016), pp. 181-184, 2016.
[11] F. Gao, G. Liu, C. Saunders, et al, "Applications of smart grid big data analytics, " Electric Power Construction, Vol. 36, no. 10, pp. 11-19, 2015.
[12] H. Daki, A, El Hannani, A. Aqqal, A. Haidine and A. Dahbi, "Big Data Management in Smart Grid: Concepts, Requirements and Implementation," Journal of Big Data,4:13, 2017.
[13]E. Gasch, J. Meyer, P. Schegner and K. Schmidt, "Efficient Power Quality Analysis of Big Data (Case Study for a Disribution Network Operator)," 23rd International Conference on Electricity Distribution (CIRED), pp.1401-1405, 2015.
[14]Arduino 控制器,
https://www.arduino.cc/, 10/2016.
[15]Y. C. Chen, Hui-Yu Shen, H. Y. Chen and C. H. Hsu, "Low Cost Arduino DAQ Development and Implementation on an Android App for Power Frequency Measurement," IEEE International Symposium on Computer, Consumer and Control, Jul. 4-6, 2016.
[16]Y. C. Chen, H. Y. Shen and C. H. Hsu, "A Power Quality Measurement System Based on the Cloud Computing Architecture by Using Fourier Series and Zero-crossing Algorithm ," 10th International Conference on Innovative Computing, Information and Control, Aug. 20-22, 2015
[17]Y. C. Chen, H. Y. Shen, H. Y. Chen and C. H. Hsu , " A Cloud-based Power Quality Measurement System Using Discrete Fourier Series and Zero-crossing Algorithm," ICIC Express Letters, Part B: Applications, Vol. 7, no. 7, pp. 1475-1481, 2016.
[18] MongoDB 分散式系統架構,
https://docs.mongodb.com/v3.2/core/distributed-queries/, 01/2017..
[19]Apache Hadoop YARN 2.0分散式系統架構,
http://www.bigdataqubes.com/posts/yarnarchitecture.php, 12/2016.
[20]R.Yokeeswaran and A.Vetrivel, "Measurement and Comparison of Power Quality Disturbances using Discrete Wavelet Transform (DWT) and Discrete Orthogonal S-Transformv (DOST)," International Journal of Engineering Research and Applications (IJERA), Vol. 2, pp.808-813, 2012.
[21]Mohammad Javad Dehghani, "Comparison of S-transform and Wavelet Transform in Power Quality Analysis," International Journal of Electrical, Computer, Energetic, Electronic and Communication Engineering, Vol. 3, no. 2, 2009.
[22]Ramesh Babu P., Ashisa Dash, Siva Nagaraju, Sangameswara Raju, "The Research of Power Quality Analysis Based on family of S-transform," International Journal of Engineering Research and Applications (IJERA), Vol. 3, pp.1760-1764, 2013.‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬‬
[23]J. Driesen & R. Belmans, "Wavelet-based power quantification approaches," IEEE Trans. Instrum. Measurement. Vol. 52, pp. 1232-1238, 2003.
[24]O. Poisson, P. Rioual and M. Meunier, " New signal processing tools applied to power quality analysis," IEEE Trans. Power Delivery, Vol. 14, no.2, pp. 561–566, 1999.
[25]P. K. Dash, B. K. Panigrahi and G. Panda, "Power Quality Analysis (PQA) using S–Transform (ST) ," IEEE Transactions on Power Delivery, Vol. 18, No. 2, pp. 406-411, 2003.
[26]I. W. C. Lee and P. K. Dash, "S-Transform-Based Intelligent System for Classification of Power Quality Disturbance Signals," IEEE Trans. on Industrial Electronics, Vol. 50, No. 4, pp. 800-805, 2006.
[27]C. Perumal and K. Vijayarajan, "Detection and Classification of Power Quality Disturbance Waveform using MRA based Modified Wavelet Transform and Neural Networks," Journal of Electrical Engineering, Vol. 61, No. 4, pp. 235–240, 2010.
[28]R. G. Stockwell, L. Mansinha, R. P. Lowe, M. Eramian and R. A. Schincariol, "Local S Spectrum analysis of 1-D and 2-D data," Physics of the Earth and Planetary Interiors, 0pp.329-336, 1997.
[29]F. Z. Zhao and R. G. Yang, "Power-Quality Disturbance Recognition using S-Transform," IEEE Trans. on Power Delivery, Vol. 22, No. 2, pp. 944-950, 2007.
[30]W. M. Lin, C. H. Wu, C. H. Lin and F. S. Cheng, "Detection and Classification of Multiple Power-Quality Disturbances with Wavelet Multiclass SVM," IEEE Trans. on Power Delivery, Vol. 23, No. 4, pp. 2575-2582, 2008.
[31]Y. C. Chen, H. Y. Shen and C. H. Hsu, "Fundamental Frequency Analysis on A Harmonic Power Signal Using Fourier Series and Zero Crossing Algorithms," Journal of Information and Multimedia Signal Processing, Vol. 6, no. 5, pp. 924-937, 2015.
[32]E. Ososanya, S. Haghani and W. H Mahmoud, "The Design of a Wi-Fi Enabled Cloud Monitoring Device," ASEE Annual Conference & Exposition (Atlanda), 11 Jul, 2013.
[33]I. A. Diaz-Diaz1 and I. Cervantes, "Design and Implementation of a Power Quality Monitoring System over Internet," 20th IMEKO TC4 International Symposium and 18th International Workshop on ADC Modelling and Testing Research on Electric and Electronic Measurement for the Economic Upturn, Sep. 15-17, 2014.
[34]L. A. Zadeh, "Recent Developments of FNET/GridEye - A Situational Awareness Tool for Smart Grid," Journal of Power and Energy Systems, vol. 2, no. 3, Sep. 2016.
[35]E. H. Mamdani, "一種基於雲計算的電壓暫降並行計算方法," Proceedings of the CSEE, Vol.34, no.31, 2014.
[36]Shyam Ra, B. G. HBa, S. Kumar Sa, P. Poornachandranb and Soman K, "Apache Spark a Big Data Analytics Platform for Smart Grid," Smart GRID Technologies, Aug. 6-8, 2015.
[37]L. A. Zadeh, "The concept of a linguistic variable and its application to approximate reasoning," Learning Systems and Intelligent Robots, pp. 1-10, 1974.
[38]en50160 電力量測標準,
http://www.ecoca.ro/NtpOneUsvRo/en50160.pdf
[39]Apache Hadoop,
http://hadoop.apache.org/, 03/2017.
[40]Apache Hadoop MapReduce,
https://hortonworks.com/apache/mapreduce/, 03/2017.
[41] MongoDB 雲端資料庫,
https://www.mongodb.com/, 03/2017.
[42]Apache Hadoop HDFS,
https://hortonworks.com/apache/hdfs/, 03/2017.
[43]Apache YARN,
https://hortonworks.com/apache/yarn/, 03/2017.
[44]Apache Hive,
https://hive.apache.org/, 03/2017.
[45]Nginx 平衡負載器,
https://nginx.org/en/.
[46]Node.js 網站伺服器,
https://nodejs.org/en/, 03/2017.
[47]MongoDB Sharding 分散式儲存架構,
https://docs.mongodb.com/v3.0/core/sharding-introduction/, 04/2017.
[48]Apache Hadoop 與 Hive 系統架構關係,
https://www.dezyre.com/hadoop-tutorial/hive-tutorial, 04/2017.
[49]Node.js 非同步IO存取流程,
http://www.codingeek.com/tutorials/nodejs/is-nodejs-single-threaded/, 04/2017.
[50]Apache Hadoop 2.0 系統架構,
http://www.edupristine.com/blog/apache-spark-vs-hadoop, 05/2017.
[51] Apache Hadoop 2.0 之MapReduce與Tez操作流程, https://www.slideshare.net/hortonworks/apache-tez-accelerating-hadoop-query-processing, 03/2017.
[52]MongoDB副本叢集系統主從式架構,
https://docs.mongodb.com/manual/core/replica-set-elections/,0 4/2017.