臺灣博碩士論文加值系統

從AlexNet在ILSVRC競賽上獲得令人驚艷的表現開始，卷積神經網路（Convolutional Neural Network，簡稱CNN）的發展在影像辨識方面表現得越來越優異。為了提升準確率，CNN也逐漸往更深更寬的方向發展。雖然這些大型CNN能勝任大部分的影像辨識任務，甚至於準確率已經超越人類，但同時也存在一些缺點，包括計算成本高、訓練時間長等。在物聯網的應用中，邊緣運算被用來減少物聯網節點對雲端伺服器的依賴而造成網路頻寬的需求，以達到即時反應的需求。然而，物聯網的邊緣運算能力和記憶容量通常有限，因此設計出具有較少參數量和低計算成本的CNN架構成為一個新的研究方向。SqueezeNet就是這樣的一款輕量化模型，它使用了1×1卷積降維的方法來減少普通卷積層的參數量，並且由於它整體網路層數較少，因此計算成本也相對較低。由於1×1卷積降維這種方法不會侷限於某種特定卷積，因此我們可以替換並結合不同的卷積技術來優化模型的各項評估指標。本文以SqueezeNet為基底模型，將其Fire模組內的普通卷積層替換成水平非對稱卷積、垂直非對稱卷積、深度可分離卷積和多層感知器卷積層，探討不同卷積結構對原始SqueezeNet的影響，並試圖結合這些替換結構來進一步優化模型，以達到更高的準確率、更短的運算時間和更少的參數量。由於優化模型是將原先SqueezeNet的卷積置換成不同的卷積架構，所以將其命名為SqueezeNetRP(Replacement)系列。本文所提出的兩個非對稱卷積模型SqueezeNetRP-VA(Vertical Asymmetry)與SqueezeNetRP-HA(Horizontal Asymmetry)不僅維持著與SqueezeNet相當的準確率與更短的預測時間外，還將模型參數量減少了三分之一。其中SqueezeNetRP-VA(Vertical Asymmetry)在三種小型數據集的測試準確率上還比SqueezeNet分別高出了1.3%、0.62%、1.29%。預測時間相較於SqueezeNet分別減少11.14%、11.45%、9.08%。除此之外，本文還探索了在深度可分離卷積中使用Batch Normalization層和ReLU層所帶來的效果，並提出了SqueezeNetRP-DSC(Depthwise Separable Convolution)。這個模型相較於SqueezeNet-DSC，除了維持深度可分離卷積的低參數量特色外，在ImageNet1000的Top-1分類準確率上，超過了SqueezeNet-DSC 0.94％，並且在訓練時間上也比SqueezeNet-DSC少了4.99%。最後，本文使用三個不同的小型數據集來訓練模型，證實了SqueezeNetRP系列模型在各項評估指標的表現上具有高度的一致性。

Starting from the impressive performance of AlexNet in the ILSVRC competition, the development of Convolutional Neural Networks (CNNs) has increasingly excelled in image recognition. To enhance accuracy, CNNs have been evolving towards deeper and wider architectures. While these large-scale CNNs excel in most image recognition tasks and even surpass human performance, they also suffer from drawbacks such as high computational costs and lengthy training times. In IoT applications, edge computing is used to reduce the dependence of IoT nodes on cloud servers and the demand for network bandwidth to meet the needs of immediate response. However, IoT edge computing capabilities and memory capacity are typically limited, necessitating the design of CNN architectures with fewer parameters and lower computational costs. SqueezeNet is one such lightweight model that utilizes 1×1 convolutions for dimensionality reduction, reducing the parameter count compared to regular convolutional layers. And because it has fewer overall network layers, the computational cost is relatively low. Since the method of 1×1 convolution dimensionality reduction is not limited to a specific convolution, we can replace and combine different convolution techniques to optimize various evaluation indicators of the model. This paper adopts SqueezeNet as the base model and replaces the regular convolutional layers within its Fire module with horizontal asymmetric convolution, vertical asymmetric convolution, depthwise separable convolution, and multilayer perceptron convolutional layers. The aim is to explore the impact of different convolutional structures on the original SqueezeNet and attempt to optimize the model further by combining the characteristics of these replacement structures. The desired effects include higher accuracy, shorter computation time, and reduced parameter count. Since the convolutional layers of the original SqueezeNet are replaced with different convolutional architectures, this approach is named the SqueezeNetRP (Replacement) series. Within the SqueezeNetRP series, this paper introduces two asymmetric convolution models, SqueezeNetRP-VA(Vertical Asymmetry) and SqueezeNetRP-HA(Horizontal Asymmetry). These models not only maintain comparable accuracy to SqueezeNet but also exhibit shorter prediction times while reducing the model parameters by one-third. Among them, SqueezeNetRP-VA is 1.3%, 0.62%, and 1.29% higher than SqueezeNet in the test accuracy of three small data sets. Compared with SqueezeNet, the prediction time is reduced by 11.14%, 11.45%, and 9.08%, respectively. Furthermore, the paper explores the effects of incorporating Batch Normalization and ReLU layers in depthwise separable convolution and proposes SqueezeNetRP-DSC (Depthwise Separable Convolution). This model, compared to SqueezeNet-DSC, not only maintains the low parameter count characteristic of depthwise separable convolution but also surpasses SqueezeNet-DSC by 0.94% in the Top-1 classification accuracy on ImageNet1000. Moreover, it reduces training time by 4.99% compared to SqueezeNet-DSC. Finally, this paper uses three different small data sets to train the models, which confirms that the SqueezeNetRP series models have a high degree of consistency in the performance of various evaluation indicators.

中文摘要.....................................................................................................................................i
英文摘要....................................................................................................................................ii
誌謝...........................................................................................................................................iv
目錄............................................................................................................................................v
表目錄......................................................................................................................................vii
圖目錄.....................................................................................................................................viii
第一章 緒論...............................................................................................................................1
第二章 卷積神經網路模型探討...............................................................................................3
2.1 CNN 基本架構.........................................................................................................................3
2.1.2 池化層 (Pooling Layer).........................................................................................................4
2.1.3 激活層 (Activation Layer) ....................................................................................................4
2.1.4 全連接層 (Fully Connected Layer).......................................................................................4
2.2 AlexNet....................................................................................................................................5
2.2.1 ReLU.....................................................................................................................................5
2.2.2 Dropout .................................................................................................................................6
2.3 NIN Net....................................................................................................................................7
2.3.1 多層感知器卷積(Multilayer Perceptron Convolution).........................................................7
2.3.2 全域平均池化層(Global Average Pooling Layer)................................................................7
2.4 VGG Net ..................................................................................................................................8
2.5 GoogLeNet.............................................................................................................................10
2.5.1 1×1 卷積.............................................................................................................................11
2.5.2 Batch Normalization............................................................................................................11
2.5.3 非對稱卷積.........................................................................................................................11
2.6 SqueezeNet.............................................................................................................................13
2.7 MobileNet ..............................................................................................................................16
2.7.1 深度可分離卷積.................................................................................................................17
2.7.2 ReLU6.................................................................................................................................18
2.8 SqueezeNet-DSC....................................................................................................................19
第三章 SqueezeNet 卷積神經網路之優化 .............................................................................21
3.1 設計目標與理念....................................................................................................................21
3.2 SqueezeNetRP-NF (NoFire)...................................................................................................21
3.3 SqueezeNetRP-NFV2.............................................................................................................22
3.4 SqueezeNetRP-NIN................................................................................................................23
3.5 SqueezeNetRP-VA (Vertical Asymmetry).............................................................................25
3.6 SqueezeNetRP-HA (Horizontal Asymmetry).........................................................................27
vi
3.7 SqueezeNetRP-DSC（Depthwise Separable Convolution）.................................................28
3.8 SqueezeNetRP-NF&HA (No Fire & Horizontal Asymmetry)................................................30
第四章 模型訓練與預測.........................................................................................................31
4.1 實驗流程................................................................................................................................31
4.1.1 實驗環境.............................................................................................................................31
4.1.2 數據前處理.........................................................................................................................31
4.1.3 數據擴增.............................................................................................................................33
4.1.4 模型訓練.............................................................................................................................34
4.1.5 評估指標.............................................................................................................................34
4.2 實驗結果................................................................................................................................36
第五章 結論.............................................................................................................................47
參考文獻..................................................................................................................................48
Extended Abstract

[1] LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P., 1998, "Gradient-based learning applied to document recognition", Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324.
[2] Alex Krizhevsky , Ilya Sutskever and Geoffrey E. Hinton, 2012. "ImageNet Classification with Deep Convolutional Neural Networks", In Advances in Neural Information Processing Systems 25, pp. 1106–1114.
[3] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, 2009, "ImageNet: A Large-Scale Hierarchical Image Database", In CVPR09.
[4] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R, 2014 , "Dropout: A simple way to prevent neural networks from overfitting", The Journal of Machine Learning Research, 15(1), 1929-1958.
[5] M. Lin, Q. Chen and S. Yan, 2013, "Network in network", CoRR, vol. abs/1312.4400.
[6] K. Simonyan and A. Zisserman, 2014, "Very deep convolutional networks for large-scale image recognition", CoRR, vol. abs/1409.1556.
[7] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich, 2014, "Going Deeper with Convolutions", CoRR, vol. abs/1409.4842.
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, 2015, "Deep Residual Learning for Image Recognition", CoRR, vol. abs/1512.03385.
[9] Forrest N. Iandola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, Kurt Keutzer, 2016, "SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size", CoRR, vol. abs/1602.07360.
[10] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, Hartwig Adam, 2017, "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications", CoRR, vol. abs/1704.04861.
[11] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, Jian Sun, 2017, "ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices", CoRR, vol. abs/1707.01083.
[12] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le, 2017, "Learning Transferable Architectures for Scalable Image Recognition", CoRR, vol. abs/1707.07012.
[13] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, Quoc V. Le, 2018, "MnasNet: Platform-Aware Neural Architecture Search for Mobile", CoRR, vol. abs/1807.11626
[14] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, Zbigniew Wojna, 2015, "Rethinking the Inception Architecture for Computer Vision", CoRR, vol. abs/1512.00567.
[15] A. G. Santos, C. O. de Souza, C. Zanchettin, D. Macedo, A. L. I. Oliveira, T. Ludermir, 2018, "Reducing SqueezeNet storage size with depthwise separable convolutions", in Proc. Int. Joint Conf. Neural Netw. (IJCNN), pp. 1–6.
[16] Sergey Ioffe, Christian Szegedy, 2015, "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift", CoRR, vol. abs/1502.03167.
[17] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, Alex Alemi, 2016, "Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning", CoRR, vol. abs/1602.07261.
[18] https://chih-sheng-huang821.medium.com/卷積神經網路-convolutional-neural-network-cnn-1-1卷積計算在做什麼-7d7ebfe34b8
[19] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, Liang-Chieh Chen, 2018, "MobileNetV2: Inverted Residuals and Linear Bottlenecks", CoRR, vol. abs/1801.04381.
[20] https://www.kaggle.com/datasets/utkarshsaxenadn/fast-food-classification-dataset
[21] https://www.kaggle.com/datasets/alxmamaev/flowers-recognition
[22] https://www.kaggle.com/datasets/alessiocorrado99/animals10