臺灣博碩士論文加值系統

從傳統的線性插值法、鄰近像素插值法到生成式網路、對抗式生成網路，近年來隨著深度學習的興起，廣泛的影像超解析方式被提出。傳統方法相較於神經網路，雖能提升畫面解析度，但在畫面物件邊緣及細節方面無法臻至完美，在需求為高品質應用場合下仍然不足。在生成式網路方面近年來多使用深度卷積網路方式進行影像超解析，此種方法有助於單張影像在細節方面的提升，但對於由多張影像構成的視訊而言，此種方法容易使視訊因畫面各自超解析造成不連貫特性，在視訊撥放時因亮度和色度差異出現畫面閃爍、抖動，為了解決此種問題，對於視訊內容應在保持物件邊緣及細節同時注重影像連貫性，完成連續畫面之視訊超解析，此種方法需從低解析度畫面中提取空間特徵資訊、以及相鄰畫面間的時間相關性，如光流、動作資訊，以此重建出高解析度畫面，消除於播放時畫面物件邊緣鋸齒、紋理消失及亮度變化問題。
本論文提出將多影像超解析整合至視訊編解碼器中，如H.26x、AV1，稱為MIMVSR。研究如何強化解碼後影像以保有更多影像細節，並在後續利用編解碼時提供的動作向量等相關資訊，配合3維卷積方法進行計算，用以訓練生成式神經網路。在提升解碼影像細節方面，加入小波轉換作為神經網路損失，於補償編解碼過程造成之損失前提下，著重強化畫面結構及線條，完成低解析度畫面強化，以利後續使用增強後畫面及動作向量進行生成式神經網路訓練，完成多影像視訊超解析提升視訊完整性。

From traditional linear interpolation and nearest neighbor interpolation to generative networks and adversarial generative networks, in recent years, with the rise of deep learning, a wide range of image super-resolution methods have been proposed. Compared to neural networks, traditional methods can enhance the resolution of images, but they fall short of achieving perfection in terms of object edges and details in the image. They still lack adequacy in high-quality application scenarios. In recent years, deep convolutional neural networks have been widely used in the field of image super-resolution within generative networks. This approach aids in enhancing the details of single images. However, for videos composed of multiple frames, this method can lead to discontinuities due to individual frame super-resolution, resulting in flickering and jittering during video playback due to differences in brightness and color. To address this issue, it is essential to maintain both object edges and details while emphasizing image coherence within the video content. Achieving continuous video super-resolution involves extracting spatial feature information from low-resolution frames and exploiting temporal correlations between adjacent frames, such as optical flow and motion information. By doing so, high-resolution frames can be reconstructed, effectively eliminating issues such as jagged edges, texture loss, and brightness variations during playback.
This thesis proposes integrating multi-image super-resolution into video codecs such as H.26x and AV1, referred to as Multi-image Motion Vector Super-Resolution (MIMVSR). It explores methods to enhance the decoded images to retain more details and utilizes motion vectors and other related information provided during encoding and decoding, combined with 3D convolution methods, to train generative neural networks. To improve the detail of decoded images, wavelet transform is incorporated as a neural network loss function. This approach compensates for losses incurred during the encoding and decoding process by focusing on strengthening the image structure and lines. The enhanced low-resolution images and motion vectors are then used to train the generative neural network, achieving multi-image video super-resolution and improving video integrity.

Abstract..................i
摘要..................iii
Acknowledgments..................v
Table of Contents..................viii
List of Tables..................x
List of Figures..................xi
Chapter 1 Introduction..................1
1.1 Background..................1
1.2 Motivation and objective..................2
1.3 Organization of the thesis..................2
Chapter 2 Literature Review..................4
2.1 Overview..................4
2.2 Standard Video Codecs..................4
2.2.1 Motion Vector..................6
2.2.2 GOP..................7
2.3 Convolution Neural Network (CNN)..................9
2.3.1 2D Convolution..................9
2.3.2 3D Convolution..................10
2.3.3 SRCNN..................13
2.4 Wavelet Transform..................14
Chapter 3 Intra-frame and Inter-frame Learning-based Super-resolution..................15
3.1 System Overview..................15
3.2 Overall Design of the System Model..................16
3.2.1 Design of Edge Enhancement Network..................16
3.2.2 Design of MIMVSR..................19
3.3 Loss Function..................23
3.3.1 EEN Network Loss..................23
3.3.2 MIMVSR Network Loss..................26
Chapter 4 Experiments..................29
4.1 Overview..................29
4.2 Environment Settings..................29
4.3 Prepare Datasets and Training Parameter..................30
4.3.1 Preprocessing of Datasets..................31
4.3.2 Training Parameter..................32
4.4 Experimental Results..................36
4.5 Experimental Evaluation..................40
4.5.1 Effect of adding DWT and MV to the model..................40
4.5.2 Evaluation and Comparison..................46
Chapter 5 Conclusion..................54
Reference..................55
Abstract..................58

[1]Varsha Hemant Patil, and Dattatraya S. Bormane. “Interpolation for Super Resolution Imaging”, Innovations and Advanced Techniques in Computer and Information Sciences and Engineering, 2007.
[2]Wan-Chi Siu, and Kwok-Wai Hung. “Review of image interpolation and super-resolution”, Proceedings of The Asia Pacific Signal and Information Processing Association Annual Summit and Conference, Hollywood, CA, USA, pp. 1-10, 2012.
[3]Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. “Image Super-Resolution Using Deep Convolutional Networks”, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 38, no. 2, pp. 295-307, 1 Feb. 2016.
[4]Daniel Glasner, and Shai Bagon, Michal Irani. “Super-resolution from a single image”, IEEE 12th International Conference on Computer Vision, Kyoto, Japan, pp. 349-356, 2009.
[5]Zhisheng Lu, Juncheng Li, Hong Liu, Chaoyan Huang, Linlin Zhang, and Tieyong Zeng. “Transformer for Single Image Super-Resolution”, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 457-466, 2022.
[6]Xuelong Li, Yanting Hu, Xinbo Gao, Dacheng Tao, and Beijia Ning. “A multi-frame image super-resolution method”, Signal Processing, Vol 90, Issue 2, pp. 405-414, 2010.
[7]Md Rifat Arefin, Vincent Michalski, Pierre-Luc St-Charles, Alfredo Kalaitzis, Sookyung Kim, Samira E. Kahou, and Yoshua Bengio. “Multi-Image Super-Resolution for Remote Sensing Using Deep Recurrent Networks”, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 206-207, 2020.
[8]ITU Telecommunication Standardization Sector - Video Coding Experts Group. “Advanced video coding for generic audiovisual services”, SERIES H: AUDIOVISUAL AND MULTIMEDIA SYSTEMS, Infrastructure of audiovisual services – Coding of moving video, Jun 2019.
[9]Vivienne Sze, Madhukar Budagavi, and Gary J. Sullivan. “High Efficiency Video Coding: Algorithms and Architectures”, 2014.
[10]Yue Chen et al. “An Overview of Core Coding Tools in the AV1 Video Codec”, Picture Coding Symposium (PCS), San Francisco, CA, USA, pp. 41-45, 2018.
[11]Shanshe Wang, Siwei Ma, Shiqi Wang, Debin Zhao, and Wen Gao. “Rate-GOP Based Rate Control for High Efficiency Video Coding”, IEEE Journal of Selected Topics in Signal Processing, Vol 7, Issue 6, pp. 1101-1111, 03 Jul. 2013.
[12]https://ultravideo.fi/
[13]Yann LeCun et al. “Backpropagation Applied to Handwritten Zip Code Recognition”, in Neural Computation, Vol 1, no. 4, pp. 541-551, Dec. 1989.
[14]Amitha Mathew, P. Amudha, and S. Sivakumari. “Deep Learning Techniques: An Overview”, Advanced Machine Learning Technologies and Applications, pp. 599–608, 2020.
[15]Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. “Learning Spatiotemporal Features With 3D Convolutional Networks”, Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 4489-4497, 2015.
[16]A. Grossmann, and J. Morlet. “Decomposition of Hardy Functions into Square Integrable Wavelets of Constant Shape”, SIAM Journal on Mathematical Analysis, Philadelphia, Vol 15, Issue 4, Jul 1984.
[17]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. “Deep Residual Learning for Image Recognition”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
[18]Johannes Ballé, Valero Laparra, and Eero P. Simoncelli. “Density Modeling of Images using a Generalized Normalization Transformation”, arXiv:1511.06281, 2015.
[19]Wenzhe Shi et al. “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1874-1883,2016.
[20]Justin Johnson, Alexandre Alahi, and Li Fei-Fei. “Perceptual Losses for Real-Time Style Transfer and Super-Resolution”, European Conference on Computer Vision, pp 694–711, 2016.
[21]Xiaozhong Ji et al. “Real-World Super-Resolution via Kernel Estimation and Noise Injection”, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 466-467, 2020.
[22]Karen Simonyan, and Andrew Zisserman. “Very Deep Convolutional Networks for Large-Scale Image Recognition”, arXiv:1409.1556, 2014.
[23]Han Yuan Chang. “Motion-based Super-resolution Technique for Compressed Videos”, National Formosa University Department of Computer Science and Information Engineering, Master thesis, 01 Sep. 2022.
[24]https://www.twcc.ai/
[25]Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T. Freeman. “Video Enhancement with Task-Oriented Flow”, International Journal of Computer Vision, Volume 127, pp. 1106–1125, 2019.
[26]Alain Hore, and Djemel Ziou. “Image Quality Metrics: PSNR vs. SSIM.” International Conference on Pattern Recognition, 2010.
[27]Z. Wang, E.P. Simoncelli, and A.C. Bovik. “Multiscale structural similarity for image quality assessment”, The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, Pacific Grove, CA, USA, Vol.2, pp. 1398-1402, 2003.
[28]Richard Zhang et al. “The Unreasonable Effectiveness of Deep Features as a Perceptual Metric”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 586-595, 2018.