臺灣博碩士論文加值系統

隨著人工智慧技術發展，圖像生成和語言描述已成為計算機視覺和自然語言處理領域的重要研究方向，視覺語言融合是這兩個領域交叉的焦點。
本篇論文將基於 Stable Diffusion、Realistic Vision和 BLIP的視覺語言融合方法改進了圖像生成和語言描述的一致性。
本篇論文以 Stable Diffusion作為生成模型，並使用 Realistic Vision此技術利用逼真的視覺資訊，來使生成的圖像更貼近真實圖像，從而增強了圖像生成的質量和可信度，同時，再引入BLIP影像預訓練模型來增強圖像和語言資料關聯性，來將Stable Diffusion模型所產生的圖像朝著更逼真的圖像進行改進。
實驗結果顯示該方法在預訓練資料集上取得了出色的表現，與傳統方法相比，在圖像生成的質量上有顯著的提升，將傳統方法所生成的圖像中的FID分數從18.6進步至 14.9，展示了該方法在視覺語言融合方法的進步。

With the advancement of artificial intelligence technologies, image generation and language description have become crucial research directions in the fields of computer vision and natural language processing. The fusion of vision and language is a focal point where these two domains intersect.

This paper proposes an enhancement to the consistency of image generation and language description based on the integration of Stable Diffusion, Realistic Vision, and BLIP. Stable Diffusion serves as the generative model, while Realistic Vision leverages realistic visual information to make generated images more closely resemble real ones, thereby improving the quality and credibility of image generation. Additionally, the introduction of the BLIP pre-trained model enhances the correlation between image and language data, further refining the images generated by the Stable Diffusion model towards a more realistic representation.

Experimental results demonstrate outstanding performance on pre-training datasets. Compared to traditional methods, there is a significant improvement in the quality of image generation, with the FID score reduced from 18.6 to 14.9. This showcases the advancement of this method in visual-language fusion approaches.

摘要 i
英文摘要 ii
目錄 iii
圖目錄 iv
表目錄 v
壹、導論 1
貳、文獻探討 3
一、基於圖像生成的相關研究 3
二、深度學習之圖片生成領域的相關研究 3
三、文字轉圖像之相關生成式模型 10
參、研究方法 13
一、研究工具 14
二、訓練資料 15
三、研究架構 19
肆、研究結果 23
一、研究限制 23
二、參數設定 23
三、圖像資料 24
四、以其他描述詞產生之圖像 27
伍、結論 30
陸、參考文獻 31

[1] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, Honglak Lee (2016). Generative Adversarial Text to Image Synthesis. Proceedings of The 33rd International Conference on Machine Learning, in Proceedings of Machine Learning Research 48:1060-1069 Available from https://proceedings.mlr.press/v48/reed16.html.
[2] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio (2014). Generative adversarial nets. In Advances in neural information processing systems (pp. 2672-2680).
[3]Kingma, D. P., & Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114.
[4]Radford, A., Metz, L., & Chintala, S. (2015). Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.
[5]Brock, A., Donahue, J., & Simonyan, K. (2019). Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations.
[6] Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T.J., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D. (2020). Language Models are Few-Shot Learners. ArXiv, abs/2005.14165.
[7]Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L, Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems (pp. 5998-6008).
[8]Zhang, H., Goodfellow, I., Metaxas, D., & Odena, A. (2019, May). Self-attention generative adversarial networks. In International conference on machine learning (pp. 7354-7363). PMLR.
[9] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ArXiv, abs/2010.11929
[10]Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., & Metaxas, D. N. (2018). StackGAN++: Realistic image synthesis with stacked generative adversarial networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(8), 1947-1962.
[11]Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., & Hochreiter, S. (2017). Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30.
[12]Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., & Abbeel, P. (2016). Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems (pp. 2172-2180).
[13]Zhao, J., Mathieu, M., & LeCun, Y. (2016). Energy-based generative adversarial network. arXiv preprint arXiv:1609.03126.
[14]Denton, E. L., Chintala, S., Szlam, A., & Fergus, R. (2015). Deep generative image models using a Laplacian pyramid of adversarial networks. In Advances in neural information processing systems (pp. 1486-1494).
[15]Isola, P., Zhu, J. Y., Zhou, T., & Efros, A. A. (2017). Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1125-1134).
[16]Kim, T., Cha, M., Kim, H., Lee, J. K., & Kim, J. (2017). Learning to discover cross-domain relations with generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 (pp. 1857-1865).
[17]Salimans, T., Goodfellow, I., Zaremba, W., Cheung, V., Radford, A., & Chen, X. (2016). Improved techniques for training GANs. In Advances in neural information processing systems (pp. 2234-2242).
[18]Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016). Generative adversarial text to image synthesis. arXiv preprint arXiv:1605.05396.
[19]Zhang, H., Goodfellow, I., Metaxas, D., & Odena, A. (2018). Self-attention generative adversarial networks. arXiv preprint arXiv:1805.08318.
[20]Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision (pp. 2223-2232).
[21]Ma, X., Li, Y., Zhang, Y., Zheng, H., & Sun, J. (2018). Shifting more attention to video salient object detection. In Proceedings of the European Conference on Computer Vision (ECCV) (pp. 30-46).
[22]Tulyakov, S., Liu, M. Y., Yang, X., Kautz, J., & Catanzaro, B. (2018). MoCoGAN: Decomposing motion and content for video generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1526-1535).
[23]Kim, T., Cha, M., Kim, H., Lee, J. K., & Kim, J. (2017). Learning to discover cross-domain relations with generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 (pp. 1857-1865).
[24]Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. [3]Soumith chintala, Luke metz, & Alec radford. (2015, November 19). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. https://arxiv.org/abs/1511.06434
[25]Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, S.K., Ayan, B.K., Mahdavi, S.S., Lopes, R.G., Salimans, T., Ho, J., Fleet, D.J., & Norouzi, M. (2022). Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. ArXiv, abs/2205.11487.