臺灣博碩士論文加值系統

近年來，臉部表情辨識（FER） 在情感計算中已成為一個流行的研究課題，但FER在自動辨識由於多種原因仍具有挑戰性，例如樣本數據質量的控制、有效特徵的提取、模型的構建與多特徵融合等方面仍有許多尚未研究的區域。有鑑於深度學習的成熟發展， FER 中採用深度學習方法逐年熱門，但深度學習的缺點在於需要大量數據來有效地訓練，因此許多研究皆使用遷移學習來彌補此缺點，但一直都沒有通用性的方法來作為FER的遷移學習指標。在本研究中，我們使用五個FER中的經典模型（ResNet-50、Xception、EfficientNet-B0、Inception與DenseNet121）作為代表，進行一系列的實驗：資料處理、遷移學習訓練模式與多重源域訓練的可用性，並進行一系列討論。

結果的趨勢表明，使用類權重進行資料平衡是最好的，不論大小資料庫，所有模型的訓練模式採用加入凍結的微調訓練都較提高準確度，除了在多重源域訓練時。在FER2013上，相較僅用ImageNet來進行遷移學習，透過ImageNet與AffectNet雙重源域的遷移學習，在Xception、EfficientNet-B0與Inception上都能帶來更好的效果，同時也發現因此降低訓練方式對模型的影響性，兩種訓練模式表現趨勢一樣。最後我們也針對不同模型找出最適合的源域與遷移學習的訓練方式，在AffectNet中，所有模型都適合加入凍結的微調訓練。而在FER2013中，ResNet50適合使用源域為ImageNet的微調訓練；Xception適合使用多重源域搭配微調訓練；Inception適合使用多重源域搭配微調訓練；EfficientNet-B0適合使用多重源域與加入凍結的微調訓練；DenseNet121單一與多重源域都適合，搭配加入凍結的微調訓練來進行遷移學習訓練能獲得最佳效果。

而為了證明我們使用遷移學習方法的優勢，我們與同樣資料庫、同個源域、同個模型的文獻相比，在AffectNet中， ResNet-50、Xception、EfficientNet-B0、Inception與DenseNet121分別提升8.37%、9.7%、1.45%、8.55%與9.47%；而在FER2013中，ResNet-50、Xception、EfficientNet-B0、Inception與DenseNet121分別提升5.72%、2%、10.45%、5%與9%，由此證明我們實驗的可靠性與在FER中使用遷移學習方法不單只是用來比較，對於實際應用也有的可用性。

Recently, Facial Expression Recognition （FER） has become a popular research topic in affective computing, but FER is still challenging in automatic recognition due to various reasons, such as control of sample data quality, extraction of effective features, the construction of models There are still many unstudied areas in aspects such as multi-feature fusion. In view of the mature development of deep learning, the deep learning method used in FER has become popular year by year, but the disadvantage of deep learning is that it requires a large amount of data to train effectively; so many studies use transfer learning to make up for this shortcoming, but there is no universal one method as transfer learning metric for FER.

The trend of the results shows that using class weights for data balancing is the best, regardless of the size of the database, the training mode of all models with frozen fine-tuning training improves the accuracy, except when training on multi-source domain. On FER2013, compared to using only ImageNet for transfer learning, the transfer learning of ImageNet and AffectNet dual source domain can bring better results on Exception, EfficientNet-B0 and Inception, and it is also found that the training method is reduced accordingly.The impact on the model, the two training modes show the same trend. Finally, we also found the most suitable source domain and transfer learning training method for different models. In AffectNet, all models are suitable for performing frozen+fine-tuning training. In FER2013, ResNet50 is suitable for fine-tuning training with ImageNet as the source domain; Xception is suitable for fine-tuning training with multi-source domain; Inception is suitable for fine-tuning training with multi-source domain; EfficientNet-B0 is suitable for frozen+fine-tuning training with multi-source domain ; DenseNet121 is suitable for both single and multi-source domain, and the best results can be obtained by adding frozen+fine-tuning training for transfer learning training.

In order to prove the advantages of our transfer learning method, compared with the literature of the same database, the same source domain, and the same model, in AffectNet, ResNet-50, Xception, EfficientNet-B0, Inception and DenseNet121 were improved by 8.37%、9.7%、1.45%、8.55% and 9.47%; and in FER2013, ResNet-50, Xception, EfficientNet-B0, Inception and DenseNet121 increased by 5.72%、2%、10.45%、5% and 9%， respectively.This demonstrates the reliability of our experiments and the usability of using transfer learning methods in FER not only for comparison but also for practical applications.

摘要...i
Abstract...ii
誌謝...iv
目錄...v
表目錄...vii
圖目錄...viii
第一章 緒論...1
1.1 研究背景...1
1.2研究動機與目標...2
1.3文獻探討...3
1.3.1基於遷移學習的FER...3
1.4論文架構...4
第二章相關技術探討...5
2.1情緒定義...5
2.2卷積神經網路運算基本架構...5
2.3卷積神經網路模型...6
2.3.1 Resnet50...6
2.3.2 Xception...6
2.3.3 Inception...6
2.3.4 DenseNet-121...6
2.3.5 EfficientNet...7
2.4 遷移學習...9
2.5資料庫...10
2.5.1 ImageNet...10
2.5.2 FER Datasets...10
第三章 研究方法...13
3-1研究架構...13
3.2資料處理...14
3.2.1數據歸一化...14
3.2.2類權重（Class weight）...14
3.2.3資料擴增（Data augmentation）...15
3.2.4超參數設定...16
3.3遷移學習模型訓練模式（Training Type）...16
3.3.1微調訓練（Fine-tuning）...16
3.3.2加入凍結的微調訓練（Frozen +Fine-tuning）...16
3.4表現評估...16
3.5實驗環境設定...17
第四章 結果分析...18
4.1資料處理實驗結果...18
4.1.1在AffectNet 上的資料處理...18
4.1.2在FER2013 上的資料處理...23
4.2遷移學習實驗結果...29
4.2.1在AffectNet上的遷移學習...29
4.2.2在FER2013上的遷移學習...34
4.2.3在FER2013上多重源域的遷移學習...40
4.3討論...45
4.4與最新文獻比較...48
第五章 結論與未來展望...49
5.1結論...49
5.2未來展望...49
參考文獻...50
Extended Abstract...53

[1]R. W. Picard, "Affective computing (No. 321)", M.I.T Media Laboratory Perceptual Computing Section Technical Report No. 321, 1995.
[2]H. Silva, A. Lourenço, and A. Fred, "In-vehicle driver recognition based on hand ECG signals," in Proceedings of the 2012 ACM international conference on Intelligent User Interfaces, pp. 25-28, 2012. doi: 10.1145/2166966.2166971
[3]J. Zhu et al., "Toward Depression Recognition Using EEG and Eye Tracking: An Ensemble Classification Model CBEM," 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2019, pp. 782-786, doi: 10.1109/BIBM47256.2019.8983225.
[4]A. Ghandeharioun et al., "Objective assessment of depressive symptoms with machine learning and wearable sensors data," 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), pp. 325-332, 2017, doi: 10.1109/ACII.2017.8273620.
[5]L. Subramainan, M. A. Mahmoud, M. S. Ahmad and M. Z. M. Yusoff, "A conceptual emotion-based model to improve students engagement in a classroom using agent-based social simulation," 2016 4th International Conference on User Science and Engineering （i-USEr）, pp. 149-154, 2016, doi: 10.1109/IUSER.2016.7857951.
[6]H. Torkamaan and J. Ziegler, "A taxonomy of mood research and its applications in computer science," 2017 Seventh International Conference on Affective Computing and Intelligent Interaction （ACII）, 2017, pp. 421-426, doi: 10.1109/ACII.2017.8273634.
[7]K. Patel et al., "Facial Sentiment Analysis Using AI Techniques: State-of-the-Art, Taxonomies, and Challenges," in IEEE Access, vol. 8, pp. 90495-90519, 2020, doi: 10.1109/ACCESS.2020.2993803.
[8] S. P. Ristiawanto, B. Irawan, and C. Setianingsih, "Pengenalan Ekspresi Wajah Berbasis Convolutional Neural Network Menggunakan Arsitektur Residual Network-50," eProceedings of Engineering, vol. 8, no. 5, 2021.
[9]A. V. Savchenko, "Facial expression and attributes recognition based on multi-task learning of lightweight neural networks," arXiv, 2021, doi：10.48550/arXiv.2103.17107.
[10]F. BELAL, "Benchmarking of Convolutional Neural Networks for Facial Expressions Recognition," Journal of Theoretical and Applied Information Technology, vol. 98, no. 18, 2020.
[11] N. S. Abdulsattar and M. N. Hussain, "Facial Expression Recognition using Transfer Learning and Fine-tuning Strategies: A Comparative Study," 2022 International Conference on Computer Science and Software Engineering （CSASE）, pp. 101-106, 2022, doi: 10.1109/CSASE51777.2022.9759754.
[12] Hong-Wei Ng, Viet Dung Nguyen, Vassilios Vonikakis, and Stefan Winkler, “Deep Learning for Emotion Recognition on Small Datasets using Transfer Learning”, In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction , pp.443–449, 2015, doi：10.1145/2818346.2830593
[13] A. Ghandeharioun et al., "Objective assessment of depressive symptoms with machine learning and wearable sensors data," 2017 Seventh International Conference on Affective Computing and Intelligent Interaction （ACII）, pp. 325-332, 2017, doi: 10.1109/ACII.2017.8273620.
[14] Y. S. Kong, V. Suresh, J. Soh, and D. C. Ong, "A Systematic Evaluation of Domain Adaptation in Facial Expression Recognition," arXiv, 2021, doi：10.48550/arXiv.2106.15453
[15]P. Ekman et al., "Universals and cultural differences in the judgments of facial expressions of emotion," Journal of personality and social psychology, vol. 53, no. 4, p. 712, 1987, doi：10.1037/0022-3514.53.4.712
[16] Akhand, M. A. H, Shuvendu Roy, Nazmul Siddique, Md Abdus Samad Kamal and Tetsuya Shimamura, "Facial Emotion Recognition Using Transfer Learning in the Deep CNN", Electronics,vol 10, no. 9, 2021, doi： 10.3390/electronics10091036
[17] Alom MZ, Taha TM, Yakopcic C, Westberg S, Sidike P, Nasrin MS, Hasan M, Van Essen BC, Awwal AAS, Asari VK, "A State-of-the-Art Survey on Deep Learning Theory and Architectures", Electronics, Vol.8, no.3, 2019, doi：10.3390/electronics8030292
[18] K.He et al, "Deep Residual Learning for Image Recognition", in 2016 IEEE Conference on Computer Vision and Pattern Recognition （CVPR） , pp.770 - 778, 2016, doi: 10.1109/CVPR.2016.90
[19] F. Chollet, "Xception: Deep Learning with Depthwise Separable Convolutions",in 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR） , pp.1800-1807, 2017, doi: 10.1109/CVPR.2017.195
[20]C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, ‘‘Rethinking the inception architecture for computer vision,’’ in Proc. IEEE Conf.Comput. Vis. Pattern Recognit. （CVPR）, Las Vegas, NV, USA, Jun. 2016, pp. 2818–2826.doi：10.1109/CVPR.2016.308
[21]G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks," 2017 IEEE Conference on Computer Vision and Pattern Recognition （CVPR） , pp. 2261 - 2269, 2017,doi: 10.1109/CVPR.2017.243
[22]M. Tan and Q. Le, "Efficientnet: Rethinking model scaling for convolutional neural networks," in Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research（PMLR）, 2019.
[23] S. J. Pan and Q. Yang, "A Survey on Transfer Learning," in IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345-1359, 2010, doi: 10.1109/TKDE.2009.191.
[24]C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu, "A survey on deep transfer learning," in International conference on artificial neural networks, Springer, pp. 270-279, 2018, doi：10.1007/978-3-030-01424-7_27.
[25]J. Deng, W. Dong, R. Socher, L. Li, L. Kai, and F.-F. Li, "ImageNet: A large-scale hierarchical image database," in 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255,June 2009, doi: 10.1109/CVPR.2009.5206848.
[26] Ko, Byoung Chul, "A brief review of facial emotion recognition based on visual information," Sensors, vol. 18, no. 2, 2018, doi：10.3390/s18020401
[27]A. Mollahosseini, B. Hasani and M. H. Mahoor, "AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild," in IEEE Transactions on Affective Computing, vol. 10, no. 1, pp. 18-31, 1 Jan.-March 2019, doi: 10.1109/TAFFC.2017.2740923.
[28]I. J. Goodfellow et al., "Challenges in representation learning: A report on three machine learning contests," in International conference on neural information processing, Springer, pp. 117-124, 2013, doi：10.1007/978-3-642-42051-1_16
[29]P. Giannopoulos, I. Perikos, and I. Hatzilygeroudis, "Deep learning approaches for facial emotion recognition: A case study on FER-2013," in Advances in hybridization of intelligent methods: Springer, pp. 1-16, 2018, doi：10.1007/978-3-319-66790-4_1.
[30]Tom,"資料的正規化（Normalization）及標準化（Standardization）." [online] Available: https://aifreeblog.herokuapp.com/posts/54/data_science_203/
[31]S. Picek, A. Heuser, A. Jovic, S. Bhasin, and F. Regazzoni, "The curse of class imbalance and conflicting metrics with machine learning for side-channel evaluations," IACR Transactions on Cryptographic Hardware and Embedded Systems, vol. 2019, no. 1, pp. 1-29, 2019, doi：10.13154/tches.v2019.i1.209-237.
[32]А. В. Савченко, "Вычислительно эффективные алгоритмы классификации изображений на основе последовательного анализа," Записки научных семинаров ПОМИ, vol. 499, no. 0, pp. 267-283, 2021.doi: 10.36535/0235-5019-2020-12-1
[33]A. Radford et al, "Learning transferable visual models from natural language supervision," in International Conference on Machine Learning, Proceedings of Machine Learning Research（PMLR）, vol.139, pp. 8748-8763,2021.