臺灣博碩士論文加值系統

現今的影像超解析度採用感知損失函數(perceptual loss)應用於放大單一影像並保留高頻細節，但應用於放大低解析度影片，感知損失函數還沒有被成功的使用，因為它通常會引入artifacts，會在重建的影片中產生閃爍(flicker)的現象，使得時間不一致性更加明顯，因此影片超解析度須而外採用動作補償(motion compensation)，通常採用光流法(optical flow)或光流網路(flow net) 透過輸入多張低解析度畫面，或者重複使用先前畫面的生成結果來提高時間一致性(temporal consistency)，本研究使用基於單一影像超解析度，將影片切割成多個畫面，進行放大，再合成為影片，此方法缺點為計算量大，但由於採用感知損失函數，產生的結果並無時間一致性的問題，且沒有閃爍的現象，此外本研究針對多張低亮度車牌影像額外訓練一個模型，不僅可以放大影像，還可以提高亮度，此研究共計算649張影像，使用開源版本OpenALPR(Automatic License Plate Recognition)程式進行正確率測試，原始低亮度影像辨識率為260/649(40.06%)，放大後辨識率為283/649(43.60%)，本研究設計一個自適性光線校正方法，還額外加入一個統計式特徵損失函數，更進一步地提高超解析度後的車牌影像辨識率，辨識率為304/649(46.84%)，辨識率提高了6.78%，本研究希望未來能應用於夜間行車紀錄器或夜間監視器畫面，不僅可以降低硬體成本花費，還可以針對交通事故提高肇事逃逸車牌辨識率，此外，本研究最大的貢獻在於可應用於低亮度影像辨識之資料預處理，提高其深度學習影像辨識的正確率。

Nowadays, image super-resolution with perceptual loss function was applied to amplify a single image and keep the high-frequency detail. However, the perceptual loss function has not been used successfully for amplifying low-resolution videos. Because it usually introduces artifacts that will flicker in the reconstructed video, making the temporal inconsistency even more apparent so the resolution of the video has to adapt motion compensation. Optical flow or flow net is usually used to improve temporal consistency by inputting multiple low-resolution frames or reusing the generated results of previous frames. In this study, the video was cut into multiple frames based on the single image super-resolution, which was amplified and then synthesized into a video. The disadvantage of this method is that it requires a large amount of computation, but because of the use of the perceptual loss function, there is no problem of temporal consistency and no flicker produced. Besides, this study trains an additional model for multiple low-brightness license plate images, which can not only amplify the image but also improve the brightness. A total of 649 images were calculated in this study, and the accuracy rate test was carried out by using OpenALPR(Automatic License Plate Recognition), an open-source version. The original low-brightness images recognition rate is 260/649(40.06%), and after amplification, the recognition rate is 283/649(43.60%). In this study, an adaptive light correction method was designed, and a statistic feature loss function was added to further improve the recognition rate, which is 304/649(46.84%) and the accuracy rate was improved by 6.78%. This study is expected to be applied to night driving recorder or night monitor in the future. It could not only reduce the cost of hardware but also improve the recognition rate of the hit-and-run driver’s license plate. Also, the most significant contribution of this study is that it can be applied to preprocessing low brightness images recognition and improve the accuracy of deep learning images recognition.

摘要 i
ABSTRACT ii
誌謝 iii
目錄 iv
圖目錄 vi
第一章 導論 1
1.1 研究動機 1
1.2 研究架構 1
第二章 相關文獻 2
2.1 深度學習之影像超解析度 2
2.1.1 Super-Resolution Convolutional Neural Network (SRCNN) 2
2.1.2 Fast Super-Resolution Convolutional Neural Network (FSRCNN) 3
2.1.3 Super Resolution using a Generator Adversarial Network (SRGAN) 6
2.2 影片超解析度 9
第三章 系統架構 11
3.1 輸入影像前處理 11
3.2 網路架構 12
3.3 Progressive Multi-scale Super-resolution 13
3.3.1 Pyramidal Decomposition 14
3.3.2 Dense Compression Units 16
3.3.3 Progressive GAN 16
3.3.4 Curriculum Learning 17
3.4訓練過程介紹 18
3.4.1 鑑別器和生成器損失 19
3.4.2 Curriculum Learning loss 20
3.4.3 統計式特徵損失(statistic feature loss) 20
3.5自適性光線校正(adaptive light correction) 21
第四章 實驗結果 22
4.1 實驗環境 22
4.2 實驗影像資料庫 22
4.3 評估方法 23
4.4 實驗數據 25
第五章 結論 26
第六章 參考 27

[1] C. Dong, C.C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution,” ECCV, pp. 184-199, 2014.
[2] C. Dong, C. C. Loy, and X. Tang, “Accelerating the Super-Resolution Convolutional Neural Network,” ECCV, 2016.
[3] M. Lin, Q. Chen, and S. Yan, “Network in network,” CVPR, 2014.
[4] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” ICCV, p. 1026–1034, 2015.
[5] C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang and W. Shi, “Photo-realistic single image super-resolution using a generative adversarial network,” CVPR, 2016.
[6] J. Bruna, P. Sprechmann, and Y. LeCun, “Super-Resolution with Deep Convolutional Sufficient Statistics,” CVPR, 2015.
[7] W. Shi, J. Caballero, F. Huszar, J. Totz, A. P. Aitken, R. Bishop,D. Rueckert, and Z. Wang, “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network,” CVPR, p. 1874–1883, 2016.
[8] C. Dong, C. C. Loy, K. He, and X. Tang, “Image super-resolution using deep convolutional networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, p. 38(2):295–307, 2016.
[9] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” International Conference on Learning Representations, 2015.
[10] A. Dosovitskiy and T. Brox, “Generating images with perceptual similarity metrics based on deep networks,” NIPS, 2016.
[11] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” CVPR, 2016.
[12] L. Gatys, A. S. Ecker, and M. Bethge, “Texture synthesis using convolutional neural networks,” NIPS, 2015.
[13] J. Johnson, A. Alahi, and F. Li, “Perceptual losses for real-time style transfer and super- resolution,” ECCV, p. 694–711, 2016.
[14] D. Ulyanov, V. Lebedev, A. Vedaldi, and V. Lempitsky, “Feed-forward synthesis of textures and stylized images,” ICML, 2016.
[15] Y. Jo, S.W. Oh, J. Kang, and S.J. Kim, “Deep video superresolution network using dynamic upsampling filters without explicit motion compensation,” CVPR, p. 3224–3232, 2018.
[16] X. Tao, H. Gao, R. Liao, J. Wang, and J. Jia, “Detail-revealing deep video super-resolution,” ICCV, 2017.
[17] D. Liu, Z. Wang, Y. Fan, X. Liu, Z. Wang, S. Chang, and T. Huang, “Robust video super-resolution with learned temporal dynamics,” ICCV, p. 2526–2534, 2017.
[18] R. Liao, X. Tao, R. Li, Z. Ma, and J. Jia, “Video superresolution via deep draft-ensemble learning,” ICCV, p. 531–539, 2015.
[19] M.S. Sajjadi, R. Vemulapalli, and M. Brown, “Framerecurrent video super-resolution,” CVPR, 2018.
[20] J. Caballero, C. Ledig, A.P. Aitken, A. Acosta, J. Totz, Z. Wang, and W. Shi, “Real-time video super-resolution with spatio-temporal networks and motion compensation,” CVPR, 2017.
[21] C. Liu and D. Sun, “A bayesian approach to adaptive video super resolution,” CVPR, 2011.
[22] W. Zhao and H.S. Sawhney, “Is super-resolution with optical flow feasible,” ECCV, 2002.
[23] A. Kappeler, S. Yoo, Q. Dai, and A.K. Katsaggelos, “Video super-resolution with convolutional neural networks,” IEEE Transactions on Computational Imaging, pp. Vol. 2, No. 2, 2016.
[24] O. Makansi, E. Ilg, and T. Brox, “End-to-end learning of video super-resolution with motion compensation,” GCPR, 2017.
[25] M.S.M. Sajjadi, R. Vemulapalli, and M. Brown, “Frame-recurrent video superresolution,” CVPR, 2018.
[26] T.H. Kim, M.S.M. Sajjadi, M. Hirsch, and B. Schölkopf, “Spatio-temporal transformer network for video restoration,” ECCV, 2018.
[27] E. Perez-Pellitero, M.S. Sajjadi, M. Hirsch, and B. Scholkopf, “Photorealistic video super resolution,” arXiv preprint arXiv, 2018.
[28] M.S.M. Sajjadi, B. Schölkopf, and M. Hirsch, “EnhanceNet: single image superresolution through automated texture synthesis,” ICCV, 2017.
[29] M. Chu, Y. Xie, L. Leal-Taxixe, and N. Thuerey, “Temporally coherent gans for video super-resolution (TecoGAN),” arXiv preprint arXiv, 2018.
[30] M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu, “Spatial transformer networks,” NIPS, 2015.
[31] Wang, Yifan and Perazzi, Federico and McWilliams, Brian and Sorkine-Hornung, Alexander and Sorkine-Hornung, Olga and Schroers, Christopher, “A Fully Progressive Approach to Single-Image Super-Resolution,” The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
[32] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Deep laplacian pyramid networks for fast and accurate super-resolution,” In IEEE Conference on Computer Vision and Pattern Recognitionresolution, 2017.
[33] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang, “Fast and accurate image super-resolution with deep laplacian pyramid networks,” arXiv preprint arXiv:1710.01992, 2017.
[34] G. Huang, Z. Liu, K. Q. Weinberger, and L. van der Maaten, “Densely connected convolutional networks,” arXiv preprint arXiv, p. 1608.06993, 2016.
[35] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” In Proceedings of the IEEE conferenceon computer vision and pattern recognition, p. 770–778, 2016.
[36] D. Balduzzi, M. Frean, L. Leary, J. P. Lewis, K. W.-D. Ma, and B. McWilliams, “The shattered gradients problem: If resnets are the answer, then what is the question?,” D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, p. 342–350, Aug 2017.
[37] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D.Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Gen-erative adversarial nets,” In Advances in neural information processing systems, p. 2672–2680, 2014.
[38] C. Ledig, L. Theis, F. Husz´ar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al, “Photo-realistic single image super-resolution using a generative adversarial network,” arXiv preprint arXiv, 2016.
[39] M. S. M. Sajjadi, B. Sch¨olkopf, and M. Hirsch, “Enhancenet: Single image super-resolution through automated texture synthesis,” CoRR, abs, 2016.
[40] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. P. Smolley, “Least squares generative adversarial networks,” arXiv preprint ArXiv, 2016.
[41] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
[42] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” In Proceedings of the 26th annual international conference on machine learning, p. 41–48, 2009.
[43] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” arXiv preprint arXiv, 2017.
[44] M. Arjovsky, S. Chintala, and L. Bottou, “Wasserstein GAN,” arXiv preprint arXiv, 2017.
[45] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv, 2014.