人臉表情是人類內部情感的一種象徵，它在日常生活的彼此交流佔有很大的地
位。大部份的人臉表情辨識使用單一特徵來表示一張人臉影像，這樣會造成在某些特
定的人臉影像會資訊不足，而且會使系統變得比較不穩定。因此，本篇論文提出了一
個多重特徵的模型來表示一張人臉影像，此模型採用了以小波轉換為基礎的二維主成
份分析和區域二元特徵做為我們的多重特徵。藉由結合這兩種特徵，多重特徵的辨識
系統將獲得這兩種特徵各自的長處，並且勝過單一特徵的辨識系統。在此，我們建議
可以採用線性權重的策略來做為特徵合並的方法。本篇論文所提出的方法將在Yale、
JAFFE、NimStim 三個人臉資料庫做模擬。從實驗結果可以得知我們提出的方法成功
的改善了系統的辨識率。

Facial expression is a representation of human internal emotion. It plays a important role in the interaction in the daily life. Most common facial expression recognitions use singlefeature to represent a facial image. It is insufficient in some specific face images, and will make system unstable. Thus, this thesis presents a multi-feature model to represent a facial image, which adopts wavelet-based two dimensional principal component analysis (2DPCA) and local binary pattern (LBP) for our integrated feature. By combining two features, the recognition system of multi-feature model accepts the merit of each feature, which outperforms single-feature model. A linear weighting strategy is suggested for the feature concatenation.
The proposed method is evaluated on three facial expression database: Yale, JAFFE and NimStim. Experimental results reveal that our method is successful in improving the recognition accuracy.

Abstract i
Contents ii
1 Introduction 1
I Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
II Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
III Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2 Literature Survey 4
3 Mathematics Foundation 8
I Feature Extraction By Principal Component Analysis . . . . . . . . . . . . . 8
II Feature Extraction By Two-Dimensional Principal Component Analysis . . . 10
III Feature Extraction By Local Binary Pattern . . . . . . . . . . . . . . . . . . 12
IV Feature Extraction ByWavelet Transform . . . . . . . . . . . . . . . . . . . 15
V Identification by Reconstruction Classifier . . . . . . . . . . . . . . . . . . . 17
VI Identification by Radial Basis Function Network . . . . . . . . . . . . . . . . 17
VII Identification by Support Vector Machine . . . . . . . . . . . . . . . . . . . . 19
4 System Architecture of Proposed Model 22
I Global Feature Extraction vs. Local Feature Extraction . . . . . . . . . . . . 22
II The Proposed Hybrid Architecture . . . . . . . . . . . . . . . . . . . . . . . 23
5 Experimental Results 26
I Facial Expression Data Sets and EvaluationMethod . . . . . . . . . . . . . . 26
II Experiments on Yale Database . . . . . . . . . . . . . . . . . . . . . . . . . . 29
III Experiments on JAFFE Database . . . . . . . . . . . . . . . . . . . . . . . . 34
IV Experiments on NimStimDatabase . . . . . . . . . . . . . . . . . . . . . . . 40
V Comparison of Computational Efficiency . . . . . . . . . . . . . . . . . . . . 45
6 Conclusion

[1] N. Fragopanagos and J. Taylor. Emotion recognition in human-computer interaction. Neural Networks, 18:389–405, 2005.
[2] B. Fasel and J. Luettin. Automatic facial expression analysis: A survey. Pattern Recognition, 36(1):259–275, 2003.
[3] A.M. Mart´inez and A.C. Kak. PCA versus LDA. IEEE Trans. on Pattern Analysis and Machine Intelligence, 23(2):228–233, Febrauary 2001.
[4] M.S. Bartlett, J.R. Movellan, and T.J. Sejnowski. Face recognition by independent component analysis. In IEEE Trans. on Neural Networks, volume 13, pages 1450–1464, November 2002.
[5] J. Yang, D. Zhang, A. Frangi, and J.-Y. Yang. Two-dimensional PCA: a new approach to appearance-based face representation and recognition. IEEE Trans. on Pattern Analysis and Machine Intelligence, 26(1):131–137, 2004.
[6] S. Kongsontana and Y. Rangsanseri. Face recognition using 2dlda algorithm. 2005.
[7] M. Pietik¨ainen T. Ojala and T. M¨aenp¨a¨a. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. on Pattern Analysis and Machine Intelligence, 24(7):971–987, 2002.
[8] T. Ahonen, A. Hadid, and M. Pietik¨ainen. Face recognition with local binary patterns. In T. Pajdla and J. Matas, editors, European Conference on Computer Vision, volume 3021, pages 469–481, 2004.
[9] P. Ekman and W.V. Friesen. The facial action coding system: A technique for the measurement of facial movement. In San Francisco:Consulting Psychologists Press, 1978.
[10] Y.-L. Tian, T. Kanade, and J. Cohn. Recognizing action units for facial expression analysis. IEEE Trans. on Pattern Analysis and Machine Intelligence, 23(2):97–115, February 2001.
[11] Y. Yacoob, H.M. Lam, and L.S. Davis. Recognizing faces showing expressions. In International Workshop on Automatic Face and Gesture Recognition, Zurich, 1995.
[12] I.A. Essa and A.P. Pentland. Coding, analysis, interpretation, and recognition of facial expressions. IEEE Trans. on Pattern Analysis and Machine Intelligence, 19(7):757–763, July 1997.
[13] Y. Zhu, L.C. De Silva, and C.C. Ko. Using moment invariants and hmm in facial expression recognition. In Image Analysis and Interpretation, 2000, pages 305–309, April 2000.
[14] I. Cohen, N. Sebe, Fabio G. Cozman, M. C. Cirelo, and T. S. Huang. Learning bayesian network classifiers for facial expression recognition using both labeled and unlabeled data. In Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 1, pages 595–601, 2003.
[15] Y. Wang, H. Ai, B. Wu, and C. Huang. Real time facial expression recognition with adaboost. In Proceedings of the 17th International Confeence on Pattern Recognition, volume 3, pages 926–929, Aug. 2004.
[16] L.C. De Silva, I.T. Miyasato, and R. Nakatus. Facial emotion recognition using multimodal information. In Information, Communications and Signal Processing, 1997. ICICS, volume 1, pages 397–401, September 1997.
[17] W.F. Liu and Z.F.Wang. Facial expression recognition based on fusion of multiple gabor features. In The 18th International Conference on Pattern Recognition, volume 3, pages 536–539, August 2006.
[18] I. Buciu and I. Pitas. Application of non-negative and local non negative matrix factorization to facial expression recognition. In Proceedings of the 17th International Confeence on Pattern Recognition, volume 1, pages 288– 291, Aug. 2004.
[19] C.-S. Lee and A. Elgammal. Nonlinear shape and appearance models for facial expression analysis and synthesis. In Proceedings of the 18th International Conference on Pattern Recognition, pages 494–502, 2006.
[20] D. Shepard. A two-dimensional interpolation function for irregularly-spaced data function. In Proceedings of the 1968 ACM National Conference, pages 517–524, 1968.
[21] C. Cortes and V. Vapnik. Support-vector network. Machine Learning, 20:273–297, 1995.
[22] J.C. Platt, N. Cristianini, and J. Shawe-Taylor. Large margin DAGs for multiclass classification. In S.A. Solla, T.K. Leen, and K.-R. M¨uller, editors, Advances in Neural Information Processing Systems 12, 2000.
[23] The Yale Face Database. http://cvc.yale.edu/projects/yalefaces/yalefaces.html.
[24] M. J. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba. Coding facial expressions with gabor wavelets. In Third IEEE International Conference on Automatic Face and Gesture Recognition, pages 200–205, April 1998.
[25] N. Tottenham, A. Borscheid, K. Ellertsen, D.J. Marcus, and C.A. Nelson. Categorization of facial expressions in children and adults: Establishing a larger stimulus set, 2002.
[26] D. Makris and T. Botley. Facial expression recognition from still images. http://staffnet.kingston.ac.uk/∼ku32195/FacialExpression.html, 2005.
[27] Z. Jin and F. Davoine. Facial expression analysis by using KPCA. In 2003 IEEE International Conference on Robotics Intelligent Systems and Signal Processing, volume 2, pages 736–741, October 2003.
[28] Y. Shinohara and N. Otsu. Facial expression recognition using fisher weight maps. In Proceedings of the Sixth IEEE International Confeence on Automatic Face And Gesture Recognition, pages 499 – 504, 2004.
[29] X. Feng. Facial expression recognition based on local binary patterns and coarse-to-fine classification. In Proceedings of the Fourth International Conference on Computer and Information Technology, pages 178–183, September 2004.
[30] M. Lyons, J. Budynek, and S. Akamatsu. Automatic classification of single facial images. 21(12):1357–1362, December 1999.
[31] X. Feng, M. Pietikainen, and A. Hadid. Facial expression recognition with local binary patterns and linear programming. Pattern Recognition and Image Analysis, 15(2):546–548, 2005.