Journal of Guangdong University of Technology ›› 2023, Vol. 40 ›› Issue (03): 10-16.doi: 10.12052/gdutxb.210179

Previous Articles     Next Articles

Facial Expression Recognition Based on Multi-feature Fusion

Lai Dong-sheng1, Feng Kai-ping2, Luo Li-hong2   

  1. 1. School of Computer Science and Technology, Guangdong University of Technology, Guangzhou 510006, China;
    2. School of Art and Design, Guangdong University of Technology, Guangzhou 510090, China
  • Received:2021-11-17 Online:2023-05-25 Published:2023-06-08

HTML

PDF (PC)

1165

Abstract: For the insufficient generalization ability of lightweight facial expression recognition algorithms, a facial expression recognition method based on multi-feature fusion and attention mechanism is proposed. LBP is used to reduce the interference of meaningless information of original face image, the dual-branch neural network extracts the features of original face image and LBP image respectively, merges the middle-level and high-level feature information extracted by these two networks and strengthens important features through attention mechanism for facial expression recognition, which can generate a large amount of discriminative features by multi-feature fusion to improve the recognition performance with a small number of parameters. The experimental results show that the recognition rate of this method has reached 70.21% on Fer2013 dataset and 95.59% on CK+ dataset respectively, which effectively improves the performance of lightweight expression recognition algorithm.

Key words: facial expression recognition, lightweight network, local binary pattern, multi-feature fusion, attention mechanism

CLC Number: 

  • TP391
[1] SHI D, TANG H. Face recognition algorithm based on self-adaptive blocking local binary pattern [J]. Multimedia Tools and Applications, 2021, 80: 23899-23921.
[2] YAO A, CAI D, HU P, et al. HoloNet: towards robust emotion recognition in the wild[C]// Proceedings of the 18th ACM International Conference on Multimodal Interaction. Tokyo: Association for Computing Machinary, 2016: 472-478.
[3] DING H, ZHOU P, CHELLAPPA R. Occlusion-adaptive deep network for robust facial expression recognition[C]// IEEE International Joint Conference on Biometrics. Houston: IEEE, 2020: 1-9.
[4] FAN Y, LAM J, LI V. Multi-ensemble convolutional neural network for facial expression recognition[C]// International Conference on Artificial Neural Networks. Rhodes: Springer International Publishing, 2018: 84-94.
[5] WANG K, PENG X, YANG J, et al. Region attention networks for pose and occlusion robust facial expression recognition [J]. IEEE Transactions on Image Processing, 2020, 29: 4057-4069.
[6] WANG K, CHEN J, ZHANG X, et al. Facial expression recognition based on deep convolutional neural network[C]// IEEE 8th Annual International Conference on CYBER Technology in Automation, Control, and Intelligent Systems. Tianjin: IEEE, 2018: 629-634.
[7] MA Y, WANG X, WEI L. Multi-level spatial and semantic enhancement network for expression recognition [J]. Applied Intelligence, 2021, 51: 1-14.
[8] LI J, JIN K, ZHOU D, et al. Attention mechanism-based CNN for facial expression recognition [J]. Neurocomputing, 2020, 411: 340-350.
[9] HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 7132-7141.
[10] CHEN S, LIU Y, GAO X, et al. Mobile Face Nets: efficient cnns for accurate real-time face verification on mobile devices[C]// Biometric Recognition. Urumqi: Springer International Publishing, 2018: 428-438.
[11] 徐琳琳, 张树美, 赵俊莉. 构建并行卷积神经网络的表情识别算法[J]. 中国图像图形学报, 2019, 24(2): 227-236.
XU L L, ZHANG S M, ZHAO J L. Expression recognition algorithm for parallel convolutional neural networks [J]. Journal of Image and Graphics, 2019, 24(2): 227-236.
[12] MOLLAHOSSEINI A, CHAN D, MAHOOR M H. Going deeper in facial expression recognition using deep neural networks[C]// 2016 IEEE Winter Conference on Applications of Computer Vision. Lake Placid: IEEE, 2016: 1-10.
[13] JEON J, PARK J C, JO Y J, et al. A real-time facial expression recognizer using deep neural network[C]// Proceedings of the 10th International Conference on Ubiquitous Information Management and Communication. Danang: Association for Computing Machinery, 2016: 1-4.
[14] SANG D V, DAT N V, THUAN D P. Facial expression recognition using deep convolutional neural networks[C]// 9th International Conference on Knowledge and Systems Engineering. Hue: IEEE, 2017: 130-135.
[15] MA N N, ZHANG X Y, ZHENG H T, et al. ShuffleNet V2: practical guidelines for efficient cnn architecture design[C]// Proceedings of the European Conference on Computer Vision. Munich: Springer International Publishing, 2018: 116-131.
[16] 何志超, 赵龙章, 陈闯. 用于人脸表情识别的多分辨率特征融合卷积神经网络[J]. 激光与光电子学进展, 2018, 55(7): 370-375.
HE Z C, ZHAO L Z, CHEN C. Convolution neural network with multi-resolution feature fusion for facial expression recognition [J]. Laser & Optoelectronics Progress, 2018, 55(7): 370-375.
[17] QIAN Z, MU J, ZHANG J, et al. Facial expression recognition based on sas-net attention mechanism[C]// International Conference on Computer Network, Electronic and Automation. Xian: IEEE, 2021: 159-163.
[1] Lai Zhi-mao, Zhang Yun, Li Dong. A Survey of Deepfake Detection Techniques Based on Transformer [J]. Journal of Guangdong University of Technology, 2023, 40(06): 155-167.
[2] Zeng An, Chen Xu-zhou, Ji Yu-Zhu, Pan Dan, Xu Xiao-Wei. Cardiac Multiclass Segmentation Method Based on Self-attention and 3D Convolution [J]. Journal of Guangdong University of Technology, 2023, 40(06): 168-175.
[3] Wu Jun-xian, He Yuan-lie. Channel Attentive Self-supervised Network for Monocular Depth Estimation [J]. Journal of Guangdong University of Technology, 2023, 40(02): 22-29.
[4] Liu Hong-wei, Lin Wei-zhen, Wen Zhan-ming, Chen Yan-jun, Yi Min-qi. A MABM-based Model for Identifying Consumers' Sentiment Polarity―Taking Movie Reviews as an Example [J]. Journal of Guangdong University of Technology, 2022, 39(06): 1-9.
[5] Teng Shao-hua, Dong Pu, Zhang Wei. An Attention Text Summarization Model Based on Syntactic Structure Fusion [J]. Journal of Guangdong University of Technology, 2021, 38(03): 1-8.
[6] Liang Guan-shu, Cao Jiang-zhong, Dai Qing-yun, Huang Yun-fei. An Unsupervised Trademark Retrieval Method Based on Attention Mechanism [J]. Journal of Guangdong University of Technology, 2020, 37(06): 41-49.
[7] Zeng Bi-qing, Han Xu-li, Wang Sheng-yu, Xu Ru-yang, Zhou Wu. Sentiment Classification Based on Double Attention Convolutional Neural Network Model [J]. Journal of Guangdong University of Technology, 2019, 36(04): 10-17.
[8] Gao Jun-yan, Liu Wen-yin, Yang Zhen-guo. Object Tracking Combined with Attention and Feature Fusion [J]. Journal of Guangdong University of Technology, 2019, 36(04): 18-23.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
Copyright © 2017 Journal of Guangdong University of Technology
Add：729 East Dongfeng RD., Guangzhou PRC. Postcode: 510090
Tel：020-37626653,020-37626139,020-37626396
Email：xbzrb@gdut.edu.cn
Supported:Beijing Magtech