臺灣博碩士論文加值系統

動畫黑白草稿上色是將只有線條的動畫黑白草稿著以各種顏色，使其產生有顏色的著色圖片，且色彩豐富多樣化。著色問題在深度學習領域中不算是新的研究方向，但仍然有不少人在做相關的研究，動漫黑白草稿上色為其中一種研究方向。相較於普通、自然圖片上色，其有紋理或是陰影等等這些已有的資訊和對應事物的顏色讓神經網路來學習，由於動畫黑白草稿著色沒有固定的顏色，也沒有紋理或是陰影等可以作為參考資訊，讓神經網路學習起來比較困難。動漫黑白草稿著色分成自動著色和有參考著色，自動著色靠網路自己學習生成出隨機顏色的著色結果；有參考著色則是使用者會先給予網路想要的顏色及要著色的地方或是參考的圖片色彩，即顏色hint或參考圖片，然後網路會根據給予的顏色hint或參考圖片生成出著色結果。在生成對抗網路(GANs)出現後，許多應用在各式各樣研究任務上的GAN也隨之出現，其中就有應用於著色問題的研究，像是有參考的動畫黑白草稿著色和黑白視頻、圖片著色等等；除了GAN也用其他網路做著色問題，像是用U-net來做自動動漫黑白草稿著色等等，但因上面所述動漫黑白草稿著色沒有固定顏色、紋理和陰影且因個別作畫家的畫風不同，不論是GAN還是其他網路所做出來的動畫黑白草稿著色效果有限。本研究提出了一種方法基於深度殘差網路，引入GAN之概念，在網路中加上鑑別器的架構，希望使著色圖片的顏色可以與使用者給予的想要顏色一致，並取得良好的著色結果。

Anime sketch colorization is to fill the various colors into the anime sketch, to make it colorful and diverse, just like the artist draws. The coloring problem is not a new research direction in the field of deep learning technology, but there are still many people researching related study, anime sketch colorization is one of them. Because coloring of the anime sketch does not have fixed color and also texture or shadow which can be reference information, so it is difficult and complex to let neural network learn how to colorization. Anime sketch coloring problem is divided into automatic coloring and coloring with reference. Automatic coloring is generating coloring results with random color by the network itself. While coloring with reference is the user will first give the network the desired color and where would be coloring or reference colored image, that is the color hint, and then the network will generate coloring results according to the given color hint or reference colored image. After generative adversarial networks (GANS) was proposed, some used GAN to do coloring research, like style2paint to do anime sketch with style image, but also video or photos colorization etc., although achieved some result, the coloring effect is limited. This study proposes a method use deep residual network, and adding discriminator to network, that expect the color of colored images can consistent with the desired color by the user, and can achieve good coloring results.

摘要......I
ABSTRACT......II
誌謝......III
目 錄......IV
表目錄......V
圖目錄......VI
一、導論......1
1.1 研究動機......1
1.2 研究架構......1
二、相關文獻......2
2.1 著色問題......2
2.2 自動著色方法......4
2.3 有參考著色方法......7
2.3.1 Hint......7
2.3.2 參考圖片......8
2.4 相關網路......13
2.4.1 U-net[3]......13
2.4.2 VGG[2]......15
2.4.3 Generative Adversarial Networks[4]......16
三、系統架構......17
3.1 系統著色流程......17
3.2 載入網路前後動漫彩圖前處理......18
3.2.1 草稿提取......18
3.2.2 產生顏色hint......19
3.3 基於深度學習之著色......20
3.4 著色網路......21
3.5 訓練過程介紹......23
3.5.1 Per-pixel Loss......24
3.5.2 Perceptual loss(感知損失)......25
3.5.3 Sigmoid Cross Entropy（Sigmoid交叉熵）......26
四、實驗結果......27
4.1 實驗環境......27
4.2 實驗資料......27
4.3 評估方法......28
4.4 實驗結果......29
五、結論......35
參考文獻......36

[1]Anonymous, The Danbooru Community, Branwen, G., Gokaslan, Danbooru2018: A Large-Scale Crowdsourced and Tagged Anime Illustration Dataset. https://www.gwern.net/Danbooru2018
[2]K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” International Conference on Learning Representations, 2015.
[3]O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” International Conference on Medical Image Computing and Computer-Assisted Intervention, p. 234–241, 2015.
[4]IJ. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial networks,” Conference and Workshop on Neural Information Processing Systems, p. 2672–2680, 2014.
[5]G. Liu, X. Chen, and Y. Hu, “Anime Sketch Coloring with Swish-Gated Residual U-Net,” Computational Intelligence and Intelligent Systems, p. 190-204, 2019.
[6]Q. Chen, and V. Koltun, ”Photographic image synthesis with cascaded refinement networks, “ Proceedings of International Conference on Computer Vision, 2017.
[7]L. Zhang, Y. Ji, X. Lin, “Style transfer for anime sketches with enhanced residual U-net and auxiliary classifier GAN,” Proceedings of Asian Conference on Pattern Recognition, 2017.
[8]Taizan Yonetsuji. “Paintschainer,” github.com/pfnet/Paintschainer, 2017.
[9]Y. Liu, Z. Qin, T. Wan, and Z. Luo, ”Auto-painter: cartoon image generation from sketch by using conditional Wasserstein generative adversarial networks,” Neurocomputing 311, p. 78–87, 2018.
[10]K. He, X. Zhang, S. Ren, and J. Sun, ”Deep residual learning for image recognition,” Proceedings of 29th IEEE Conference on Computer Vision and Pattern Recognition, p. 770–778 , 2016.
[11]Z. Zhang, Q. Liu, and Y. Wang, ”Road extraction by deep residual U-net,” IEEE Geoscience and Remote Sensing Letters, p. 749–753, 2018.
[12]D.P. Kingma, and J. Ba, ” Adam: a method for stochastic optimization, “ Proceedings of the 3rd International Conference for Learning Representations, 2015.
[13]P. Isola, J.Y. Zhu, T. Zhou, and A.A. Efros, ”Image-to-image translation with conditional adversarial networks,“ IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[14]J.Y. Zhu, T. Park, P. Isola, and A. A. Efros ,“Unpaired image-to-image translation using cycle-consistent adversarial networks,” IEEE International Conference on Computer Vision, 2017.
[15]Z. Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli, ”Image quality assessment: from error visibility to structural similarity,” IEEE Transactions on Image Processing, p. 600–612, 2004.
[16]P. Ramachandran, B. Zoph, and Q.V. Le, ”Searching for activation functions,” CoRR abs/1710.05941, 2017.
[17]J.L. Ba, J.R. Kiros, and G.E. Hinton, “Layer normalization,” CoRR abs/1607.06450, 2016
[18]Z. Cheng, Q. Yang, and B. Sheng, “Deep colorization,” IEEE International Conference on Computer Vision,2015.
[19]L. Zhang, C. Li, T.T. Wong, Y. Ji, and C. Liu, “Two-stage sketch colorization,” ACM Transactions on Graphics (SIGGRAPH Asia 2018 issue), p. 261:1-261:14 , 2018.
[20]R. Zhang, P. Isola, and A. A. Efros, “Colorful image colorization,” European Conference on Computer Vision, 2016.
[21]E. S. L. Gastal, and M. M. Oliveira. “Domain transform for edge-aware image and video processing,” ACM SIG International Conference on Computer Graphics and Interactive Techniques, 2011.
[22]H. Heo, and Y. Hwang, “Automatic Sketch Colorization using DCGAN,” 18th International Conference on Control, Automation and Systems,2018.
[23]L. Fang,, L. Wang, G. Lu, D. Zhang, and J. Fu, “Hand-drawn grayscale image colorful colorization based on natural image,” The Visual Computer. ,2018.
[24]L.A. Gatys, A.S. Ecker, and M. Bethge, “A neural algorithm of artistic style,” arXiv preprint arXiv:1508.06576 ,2015.
[25]L.A. Gatys, A.S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[26]J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. “Unpaired imageto-image translation using cycle-consistent adversarial networks,” IEEE International Conference on Computer Vision , 2017.
[27]A. Odena, C. Olah, and J. Shlens. “Conditional image synthesis with auxiliary classifier gans,” arXiv preprint arXiv:1610.09585, 2016.
[28]J. Johnson, A. Alahi, and F.F. Li,” Perceptual Losses for Real-Time Style Transfer and Super-Resolution,” arXiv:1603.08155, 2016.
[29]A. Radford, L. Metz, and S. Chintala. “Unsupervised representation learning with deep convolutional generative adversarial networks,” arXiv preprint arXiv:1511.06434, 2015.
[30]M. Mirza, and S. Osindero. “Conditional generative adversarial nets,” CoRR, abs/1411.1784, 2014.
[31]C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang and W. Shi, “Photo-realistic single image super-resolution using a generative adversarial network,” IEEE Conference on Computer Vision and Pattern Recognition, 2016.