臺灣博碩士論文加值系統

台灣曾經有面板王國的美譽，近來4K電視硬體技術已經成熟，但是很少看到有4K解析度的電影及電視劇內容。如果把原本720*480解析度的影片，經過影像超解析度，提升成4K解析度，應會在4K UHD電影及電視劇，產生解析度質與電影數量上的改變。
硬體方面，CPU的競爭激烈，消費級設備從原本i7的4核心8執行緒，提升為目前i9的10核心20執行緒，甚至有AMD Ryzen Threadripper的64核心128執行緒；主記憶體方面，支援intel i7的主機板由最大支援32GB主記憶體，提升到64GB主憶體，支援intel i9的主機板可支援到128GB記憶體。顯示卡方面，由 NVIDIA 1080Ti的11GB顯示卡記憶體，提升至TITAN RTX的24GB，在NVIDIA Quadro RTX8000顯示卡上，可以看到48GB顯示卡記憶體。這些都顯示電腦硬體性能上面也有長足進步。
軟體及演算法方面，電腦視覺有SRGAN及Enhancenet這個類型的演算法，可以用來把解析度提升。在電腦視覺演算法出現及電腦硬體性能提升，這些進步使我產生研究動機。希望把原SD影片經過數學放大為4K UHD影片，並且把神經網路運用在最關鍵的地方，為觀眾帶來多次觀影的不同體驗。期望這個研究帶動4K及8K UHD電視機的進步，以及電影和音樂產業的發展。

Taiwan used to have a reputation as a panel kingdom. Recently, 4K TV hardware technology has matured, but it is rare to see 4K resolution movies and TV shows. If the original 720*480 resolution film is upgraded to 4K resolution through the super-resolution of the image, the resolution and the number of movies should be changed in 4K UHD movies and TV shows.
In terms of hardware, CPU competition is fierce. Consumer-grade devices have upgraded from the original i7 with 4 cores and 8 threads to the current i9 with 10 cores and 20 threads, and even AMD Ryzen Threadripper with 64 cores and 128 threads. In terms of main memory, Motherboards that support intel i7 can support up to 32GB of main memory to 64GB, and motherboards that support intel i9 can support up to 128GB of memory. In terms of graphics cards, the 11GB of NVIDIA 1080Ti graphics card memory has been upgraded to 24GB of TITAN RTX. On the NVIDIA Quadro RTX8000 graphics card, you can see 48GB of graphics card memory. All these show that the performance of computer hardware has also made great progress.
　　In terms of software and algorithms, computer vision has SRGAN and Enhancenet algorithms, which can be used to increase the resolution. With the advent of computer vision algorithms and the improvement of computer hardware performance, these advancements have motivated me to research. I hope to mathematically enlarge the original SD video into a 4K UHD video, and apply the neural network to the most critical places, so as to bring the audience multiple different experiences of watching the movie. It is hoped that this research will drive the progress of 4K and 8K UHD TVs, as well as the development of the film and music industries.

目錄
中文摘要---------------------------------------------------------------------------------V
英文摘要---------------------------------------------------------------------------------VI
誌謝---------------------------------------------------------------------------------------VII
目錄---------------------------------------------------------------------------------------VIII
圖目錄------------------------------------------------------------------------------------XI
表目錄------------------------------------------------------------------------------------XIII
第一章 導論--------------------------------------------------------------------------------1
1.1 研究動機----------------------------------------------------------------------------1
1.2 研究架構----------------------------------------------------------------------------1
第二章 相關文獻--------------------------------------------------------------------------2
2.1 神經網路運作----------------------------------------------------------------------2
2.1.1 卷積運算--------------------------------------------------------------------------2
2.1.2 填補--------------------------------------------------------------------------------4
2.1.3 步幅--------------------------------------------------------------------------------5
2.1.4 池化層-----------------------------------------------------------------------------5
2.1.5 全連接層--------------------------------------------------------------------------6
2.1.6 激勵函數--------------------------------------------------------------------------7
2.1.6.1 恒等函數(Identity Function)-------------------------------------------------7
2.1.6.2 Sigmoid Function---------------------------------------------------------------7
2.1.6.3 ReLU 函數(Rectified Linear Unit, ReLU )--------------------------------8
2.1.6.4 Leaky ReLU---------------------------------------------------------------------8
2.1.7 Gradient Descent梯度下降------------------------------------------------------8
2.2 基於數學函式之影像縮放------------------------------------------------------10
2.2.1 Nearest-neighbor interpolation-------------------------------------------------11
2.2.2 bilinear interpolation------------------------------------------------------------13
2.2.3 bicubic interpolation-------------------------------------------------------------14
2.3 基於模型學習之超解析度演算法---------------------------------------------16
2.3.1 生成對抗網路Generative Adversarial Network ---------------------------16
2.3.2 Photo-Realistic Single Image Super-Resolution Using a Generative
Adversarial Network----------------------------------------------------------17
2.3.3 EnhanceNet-----------------------------------------------------------------------22
2.4 超解析度效能評估---------------------------------------------------------------27
2.4.1 峰值信噪比(PSNR)------------------------------------------------------------27
2.4.2 結構相似性(SSIM)-------------------------------------------------------------28
第三章 基於多損失函式之影帶超解析度-------------------------------------------30
3.1 系統架構流程圖------------------------------------------------------------------30
3.1.1 前處理----------------------------------------------------------------------------30
3.1.2 基於深度學習的影像超解析度----------------------------------------------30
3.1.3 生成器網路----------------------------------------------------------------------31
3.1.4 辨別器網路----------------------------------------------------------------------32
3.2 損失函式---------------------------------------------------------------------------33
3.2.1 Perceptual loss(感知損失) -----------------------------------------------------33
3.2.2 Adversarial loss(對抗損失)----------------------------------------------------34
3.2.3 辨別器損失----------------------------------------------------------------------32
3.3 網路訓練---------------------------------------------------------------------------34
3.4 網路預測---------------------------------------------------------------------------34
3.5 資料分析---------------------------------------------------------------------------36
3.6 把每張單一超解析度影像壓縮並製成影片檔------------------------------37
第四章 實驗結果-------------------------------------------------------------------------38
4.1 實驗環境---------------------------------------------------------------------------38
4.2 實驗影像資料庫------------------------------------------------------------------38
4.3 評估方法---------------------------------------------------------------------------39
4.3.1 網路訓練-------------------------------------------------------------------------39
4.3.2 網路評估-------------------------------------------------------------------------39
4.3.3 網路預測-------------------------------------------------------------------------66
4.4 實驗數據---------------------------------------------------------------------------66
第五章 結論-------------------------------------------------------------------------------67
第六章 參考文獻-------------------------------------------------------------------------68
附錄一 -------------------------------------------------------------------------------------70













圖目錄
圖1 神經網路運作圖(1) -----------------------------------------------------------------------2
圖2 神經網路運作圖(2) -----------------------------------------------------------------------2
圖3卷積運算的範例----------------------------------------------------------------------------3
圖4卷積運算的計算步驟----------------------------------------------------------------------4
圖5卷積運算的填補處理----------------------------------------------------------------------4
圖6步幅為2的卷積運算範例----------------------------------------------------------------5
圖7最大池化的處理步驟----------------------------------------------------------------------6
圖8平均池化的處理步驟----------------------------------------------------------------------6
圖9 恒等函數的圖形---------------------------------------------------------------------------7
圖10 Sigmoid函數的圖形---------------------------------------------------------------------7
圖11 ReLU函數的圖形------------------------------------------------------------------------8
圖12 Leaky ReLU函數的圖形----------------------------------------------------------------8
圖13 Gradient Descent圖(1) ------------------------------------------------------------------9
圖14 Gradient Descent圖(2) -----------------------------------------------------------------10
圖15 Nearest-neighbor interpolation圖(1) -------------------------------------------------11
圖16 Nearest-neighbor interpolation圖(2) -------------------------------------------------11
圖17 Nearest-neighbor interpolation圖(3) -------------------------------------------------12
圖18 Nearest-neighbor interpolation圖(4) -------------------------------------------------12
圖19 bilinear interpolation圖-----------------------------------------------------------------13
圖20 bicubic interpolation圖(1) -------------------------------------------------------------14
圖21 bicubic interpolation圖(2) -------------------------------------------------------------15
圖22 bicubic interpolation圖(3)--------------------------------------------------------------15
圖23 bicubic interpolation圖(4)-------------------------------------------------------------16
圖24 DCGAN生成器網路架構圖----------------------------------------------------------17
圖25 SRGAN生成器網路與辨別器網路的架構圖--------------------------------------18
圖26 SRGAN超解析度影像和原影像比較-----------------------------------------------21
圖27原影像與bicubic、SRResNet、SRGAN效果比較圖----------------------------21
圖28 ENet-E生成影像與ENet-PAT生成影像比較-------------------------------------26
圖 29 調整後的Enhancenet影像超解析度流程圖---------------------------------------30
圖30調整後的Enhancenet生成器網路架構圖(1)---------------------------------------31
圖31 Residual learning: a building block圖------------------------------------------------31
圖32使用CIFAR10資料集在20層與56層神經網路的訓練及測試誤差---------32
圖33調整後的Enhancenet生成器網路架構圖(2)---------------------------------------32
圖34調整後的SRGAN辨別器網路架構圖----------------------------------------------33

















表目錄
表 1 Enhancenet的影像超解析度生成器網路架構表----------------------------------22
表 2 Enhancenet的影像超解析度辨別器網路架構表----------------------------------23
表 3 VGG網路架構表-----------------------------------------------------------------------25

第六章 參考文獻
[1]I. Goodfellow. Nips,2016,Generative adversarial networks,
https://arxiv.org/abs/1701.00160
[2] Alec Radford & Luke Metz, Soumith Chintala,2016, UNSUPERVISED REPRESENTATION LEARNING WITH DEEP CONVOLUTIONAL GENERATIVE ADVERSARIAL NETWORKS, https://arxiv.org/pdf/1511.06434.pdf
[3]Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero,
Andrew Cunningham,Alejandro Acosta,Andrew Aitken,Alykhan Tejani,
Johannes Totz,Zehan Wang,Wenzhe Shi,2016,Photo-Realistic Single Image
Super-Resolution Using a Generative Adversarial Network,
https://arxiv.org/abs/1609.04802
[4]Mehdi S. M. Sajjadi, Bernhard Schölkopf, Michael Hirsch,2016,EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis, https://arxiv.org/abs/1612.07919
[5]Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun,2015,Deep Residual Learning for Image Recognition,https://arxiv.org/abs/1512.03385
[6]Karen Simonyan & Andrew Zisserman,2015,Very Deep Convolutional Networks For Large-Scale Image Recognition,https://arxiv.org/abs/1409.1556
[7]Mehdi S. M. Sajjadi,Raviteja Vemulapalli,Matthew Brown,2018,Frame-Recurrent Video Super-Resolution,https://arxiv.org/abs/1801.04590v4
[8]Ben Mildenhall,Pratul P. Srinivasan,Matthew Tancik,Jonathan T. Barron,Ravi Ramamoorthi,Ren Ng,2020, Representing Scenes as Neural Radiance Fields for View Synthesis,https://arxiv.org/abs/2003.08934v2
[9]宋宜蓁，2018，基於視覺損失函數之單張超解析度影像，國立高雄科技大學，碩士論文。
[10] Sebastian Raschka，2016，Python機器學習，博碩文化股份有限公司，新北。
[11]林大貴，2017，Tensorflow+Keras深度學習人工智慧實務應用，博碩文化股份有限公司，新北。
[12]斎藤康毅，2017，用PYTHON進行深度學習的基礎理論實作，碁峰資訊股份有限公司，臺北。
[13]邢夢來、王碩、孫洋洋，2018，PyTorch深度學習與自然語言中文處理，博碩文化股份有限公司，臺北。
[14]陳雲，2018，比Tensorflow還精美的人工智慧套件PyTorch讓你愛不釋手，佳魁資訊，臺北。
[15]廖星宇，2018，PYTORCH更好用更強大更易懂，深石數位，臺北。
[16]文淵閣工作室，2016，Python初學特訓班，碁峰資訊股份有限公司，臺北。
[17]李宏毅，2017，GAN Lecture 1 (2017): Introduction of Generative Adversarial Network(GAN)，https://www.youtube.com/watch?v=G0dZc-8yIjE&list=PLJV_el3uVTsMd2G9ZjcpJn1YfnM9wVOBf&index=1
[18]周沫凡，2016，莫煩Python，https://morvanzhou.github.io/，澳洲。
[19]Ben Hu，(深度學習)ResNet之殘差學習，https://medium.com/@hupinwei/深度學習-resnet之殘差學習-f3ac36701b2f
[20]Dot CSV，¡Aumentando RESOLUCIÓN con Inteligencia Artificial! (SuperResolución)， https://www.youtube.com/watch?v=dPYwr8fihH0&list=FL6fDPQX1Hw3dMQkVBkNbd6A&index=4&t=546s
[21]Ray Lin，Activation Function: ReLU & Maxout，https://medium.com/%E5%AD%B8%E4%BB%A5%E5%BB%A3%E6%89%8D/activation-function-relu-maxout-f958f066fbfa
[22]Martin Huang，AlexNet小簡介(3) — Local Response Normalization(LRN)，https://martin12345m.medium.com/alexnet%E5%B0%8F%E7%B0%A1%E4%BB%8B-3-local-response-normalization-lrn-a0af4e5fdafd