臺灣博碩士論文加值系統

傳統基於單張低解析度畫面的超解析(Single-image super-resolution; SISR)方法，如鄰近像素插值、線性插值法等，雖能提升畫面解析度，但容易使影像產生鋸齒或細節遺失。基於深度學習的SISR方法，以Ledig, Christian等人為首提出了有效提升視覺品質的方案，但作用於影片等連續畫面時，容易因為個別影像分別進行超解析，導致影片序列出現色差或抖動等現象。為改善此情況，影片內容的超解析應該使用多張連續畫面的超解析 (Multi-image super-resolution; MISR)方法。從多張鄰近低解析度畫面中提取空間細微互補資訊、以及相鄰畫面的動作資訊以重建出單張高解析度的畫面，因此播放時不具SISR方法的缺點，使得影片始終亮度一致。
本論文將提出一種將MISR整合至如H.26x和AV1等當代標準解碼器的方法，並研究如何利用解碼時的參考畫面(Reference frame; RF)和動作向量(Motion vector; MV)等資訊輔助訓練生成對抗網路(Generative adversarial networks; GAN)，進而提升影片品質。

Traditional single-image super-resolution (SISR) methods, such as linear interpolation and nearest-neighbor interpolation, can improve the resolution of the image, but usually produce jagged edges or lose details. Recent research on super-resolution (SR) methods is progressively using machine learning methods, which are good at replenishing picture details but can easily cause video content flickering or shaking. Therefore, in order to improve the video quality, it should explore the multi-image super-resolution (MISR) method. This method extracts spatially detailed complementary information and motion information from multiple neighboring low-resolution frames to reconstruct a single high-resolution frame, which can maintain consistent lightness and darkness during video playback to solve the shortcomings of SISR method. In this paper, we propose a method to integrate MISR method into standard video decoders such as H.26x and AV1, and investigate how to use the reference frame (RF) and motion vector (MV) information from video decoding to assist in training generative adversarial networks (GAN) to achieve super-resolution tasks.

Abstract..................i
摘要..................ii
Acknowledgments..................iii
Table of Contents..................vi
List of Tables..................viii
List of Figures..................ix
Chapter 1 Introduction..................1
1.1 Background..................1
1.2 Motivation and objective..................1
1.3 Organization of the thesis..................2
Chapter 2 Literature Review..................3
2.1 Overview..................3
2.2 Standard Video Codecs..................3
2.3 Learning-based Super-resolution (SR) Techniques..................5
2.3.1 Overview of Learning-based SR Techniques..................5
2.3.2 SR Convolutional Neural Network (SRCNN)..................6
2.3.3 SR Generative Adversarial Networks (GAN)..................7
2.3.4 Recurrent Back-Projection Network (RBPN)..................7
2.3.5 iSeeBetter..................9
Chapter 3 The Proposed Motion Vector-based SR (MVSR)..................10
3.1 System Overview..................10
3.2 Design of MVSR Model..................10
3.2.1 Design of SR Model..................11
3.2.2 Design of Discriminator..................14
3.2.3 Loss Function..................15
Chapter 4 Experiments..................17
4.1 Overview..................17
4.2 Environment Settings..................17
4.3 Prepare Datasets and Training Parameter..................18
4.3.1 Preprocessing of Datasets..................18
4.3.2 Training Parameter..................20
4.4 Experimental Results..................23
4.5 Experimental Evaluation..................25
4.5.1 Effect of adding MV to the model..................25
4.5.2 Evaluation and Comparison..................28
Chapter 5 Conclusion and Future Research..................40
Reference..................41
Abstract..................44

[1]Dong, Chao, et al. "Image super-resolution using deep convolutional networks." IEEE transactions on pattern analysis and machine intelligence 38.2: 295-307. 2015.
[2]ITU Telecommunication Standardization Sector - Video Coding Experts Group, "Advanced video coding for generic audiovisual services," SERIES H: AUDIOVISUAL AND MULTIMEDIA SYSTEMS, Infrastructure of audiovisual services – Coding of moving video, Jun 2019
[3]V. Sze, M. Budagavi, and G. J. Sullivan, High Efficiency Video Coding: Algorithms and Architectures. Springer, 2014.
[4]Zhou, Ruofan, and Sabine Susstrunk. "Kernel modeling super-resolution on real low-resolution images." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.
[5]Ledig, Christian, et al. "Photo-realistic single image super-resolution using a generative adversarial network." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
[6]Goodfellow, Ian, et al. "Generative adversarial nets." Advances in neural information processing systems 27. 2014.
[7]Haris, Muhammad, Gregory Shakhnarovich, and Norimichi Ukita. "Recurrent back-projection network for video super-resolution." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
[8]Haris, Muhammad, Gregory Shakhnarovich, and Norimichi Ukita. "Deep back-projection networks for super-resolution." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
[9]He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
[10]Chadha, Aman, John Britto, and M. Mani Roja. "iSeeBetter: Spatio-temporal video super-resolution using recurrent generative back-projection networks." Computational Visual Media 6.3: 307-317. 2020.
[11]Liu, Ce, and Deqing Sun. "A bayesian approach to adaptive video super resolution." CVPR 2011. IEEE, 2011.
[12]Xue, Tianfan, et al. "Video enhancement with task-oriented flow." International Journal of Computer Vision 127.8: 1106-1125. 2019
[13]Shi, Wenzhe, et al. "Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.
[14]Ballé, Johannes, Valero Laparra, and Eero P. Simoncelli. "End-to-end optimized image compression." arXiv preprint arXiv:1611.01704. 2016.
[15]Mentzer, Fabian, et al. "High-fidelity generative image compression." Advances in Neural Information Processing Systems 33 : 11913-11924. 2020
[16]Ballé, Johannes, Valero Laparra, and Eero P. Simoncelli. "Density modeling of images using a generalized normalization transformation." arXiv preprint arXiv:1511.06281. 2015.
[17]Isola, Phillip, et al. "Image-to-image translation with conditional adversarial networks." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.
[18]Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." Proceedings of the IEEE international conference on computer vision. 2017.
[19]Johnson, Justin, Alexandre Alahi, and Li Fei-Fei. "Perceptual losses for real-time style transfer and super-resolution." European conference on computer vision. Springer, Cham, 2016.
[20]Ji, Xiaozhong, et al. "Real-world super-resolution via kernel estimation and noise injection." proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. 2020.
[21]Simonyan, Karen, and Andrew Zisserman. "Very deep convolutional networks for large-scale image recognition." arXiv preprint arXiv:1409.1556. 2014.
[22]Han-Yuan, "MVSR" https://github.com/hanyuan97/MVSR [Online] Date accessed: September 2022.
[23]Wang, Zhou, Eero P. Simoncelli, and Alan C. Bovik. "Multiscale structural similarity for image quality assessment." The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003. Vol. 2. Ieee, 2003.
[24]Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.