3D face recognition based on RGB-D data: a survey
DOI:
https://doi.org/10.61173/9hh86v72Keywords:
RGB-D face recognition, Feature-level fusion, Hybrid Fusion, Deep Learning Face Representation, 3D face dataAbstract
Face recognition, as a convenient, natural, and widely applied emerging technology, has achieved many significant research results in recent years. 2D face recognition has drawn extensive studies, while previously,2D face recognition is too sensitive to variations in features like facial expressions. To avoid the shortcoming, more attention was paid to the optimization of algorithms, stronger computational capabilities, and fusion strategies, which contributed greatly to the accuracy of face recognition and made it more outstanding. Compared to existing methods, RGB-D images tend to be more robust and reliable. Based on different processing methods of RGB-D 3D face data, researchers have proposed numerous 3D face recognition methods, such as 3D reconstruction methods from monocular RGB-D images, methods based on point cloud data, and methods based on image depth map data. This paper focuses mainly on the image depth map data method, analyzing its rich development history and its unique advantages and disadvantages in RGB-D 3D face recognition. Additionally, we introduced some common RGB-D face datasets, analyzing data collection methods.
References
[1] Sukanya C M, Gokul R, Paul V. A survey on object recognition methods[J]. International Journal of Science, Engineering and Computer Technology, 2016, 6(1): 48.
[2] Chihaoui M, Elkefi A, Bellil W, et al. A survey of 2D face recognition techniques[J]. Computers, 2016, 5(4): 21.
[3] Günther M, El Shafey L, Marcel S. 2D face recognition: An experimental and reproducible research survey[J]. 2017.
[4] Blanz V, Vetter T. Face recognition based on fitting a 3D morphable model[J]. IEEE Transactions on pattern analysis and machine intelligence, 2003, 25(9): 1063-1074.
[5] Khan M S, Jehanzeb M, Babar M I, et al. Face Recognition Analysis Using 3D Model[C]//Emerging Technologies in Computing: First International Conference, iCETiC 2018, London, UK, August 23–24, 2018, Proceedings 1. Springer International Publishing, 2018: 220-236.
[6] Abiantun R, Prabhu U, Savvides M. Sparse feature extraction for pose-tolerant face recognition[J]. IEEE transactions on pattern analysis and machine intelligence, 2014, 36(10): 2061- 2073.
[7] Zhong Y, Pei Y, Li P, et al. Face denoising and 3D reconstruction from a single depth image[C]//2020 15th IEEE international conference on automatic face and gesture recognition (FG 2020). IEEE, 2020: 117-124.
[8] Luo C, Zhang J, Bao C, et al. Robust 3D face modeling and tracking from RGB-D images[J]. Multimedia Systems, 2022, 28(5): 1657-1666.
[9] Tran L, Liu X. Nonlinear 3d face morphable model[C]// Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 7346-7355.
[10] Jiang D, Jin Y, Zhang F L, et al. Reconstructing recognizable Dean&Francis 3d face shapes based on 3d morphable models[C]//Computer Graphics Forum. 2022, 41(6): 348-364.
[11] Zhu X, Yang F, Huang D, et al. Beyond 3dmm space: Towards fine-grained 3d face reconstruction[C]//Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VIII 16. Springer International Publishing, 2020: 343-358.
[12] Li P, Pei Y, Zhong Y, et al. Robust 3D face reconstruction from single noisy depth image through semantic consistency[J]. IET Computer Vision, 2021, 15(6): 393-404.
[13] Zhong Y, Pei Y, Li P, et al. Face denoising and 3D reconstruction from a single depth image[C]//2020 15th IEEE international conference on automatic face and gesture recognition (FG 2020). IEEE, 2020: 117-124.
[14] Dutta K, Bhattacharjee D, Nasipuri M, et al. Complement component face space for 3D face recognition from range images[J]. Applied Intelligence, 2021, 51(4): 2500-2517.
[15] Zhu Z, Sui M, Li H, et al. CMANET: Curvature-Aware Soft Mask Guided Attention Fusion Network for 2D+ 3D Facial Expression Recognition[C]//2022 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2022: 1-6.
[16] Boumedine A Y, Bentaieb S, OUAMRI A. 3D Face Identification based on Normal Maps[C]//Proceedings of the International Conference on Advances in Communication Technology, Computing and Engineering, Meknes. 2022: 260- 269.
[17] Sui M, Zhu Z, Zhao F, et al. FFNet-M: Feature fusion network with masks for multimodal facial expression recognition[C]//2021 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2021: 1-6.
[18] Wardeberg H. Mesh-based 3D face recognition using Geometric Deep learning[D]. NTNU, 2021.
[19] Sanyal S, Bolkart T, Feng H, et al. Learning to regress 3D face shape and expression from an image without 3D supervision[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019: 7763-7772.
[20] Zeng X, Peng X, Qiao Y. Df2net: A dense-fine-finer network for detailed 3d face reconstruction[C]//Proceedings of the IEEE/ CVF International Conference on Computer Vision. 2019: 2315- 2324.
[21] Dutta K, Bhattacharjee D, Nasipuri M. SpPCANet: A simple deep learning-based feature extraction approach for 3D face recognition[J]. Multimedia Tools and Applications, 2020, 79(41): 31329-31352.
[22] Li J, Qiu T, Wen C, et al. Robust face recognition using the deep C2D-CNN model based on decision-level fusion[J]. Sensors, 2018, 18(7): 2080.
[23] He R, Cao J, Song L, et al. Adversarial cross-spectral face completion for NIR-VIS face recognition[J]. IEEE transactions on pattern analysis and machine intelligence, 2019, 42(5): 1025- 1037.
[24] He M, Zhang J, Shan S, et al. Enhancing face recognition with self-supervised 3d reconstruction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 4062-4071.
[25] Cui J, Han H, Shan S, et al. RGB-D face recognition: A comparative study of representative fusion schemes[C]// Biometric Recognition: 13th Chinese Conference, CCBR 2018, Urumqi, China, August 11-12, 2018, Proceedings 13. Springer International Publishing, 2018: 358-366.
[26] Atik M E, Duran Z. Deep learning-based 3D face recognition using derived features from point cloud[C]//Innovations in Smart Cities Applications Volume 4: The Proceedings of the 5th International Conference on Smart City Applications. Springer International Publishing, 2021: 797-808.
[27] Hu Z, Gui P, Feng Z, et al. Boosting depth-based face recognition from a quality perspective[J]. Sensors, 2019, 19(19): 4124.
[28] Lee J, Bhattarai B, Kim T K. Face parsing from RGB and depth using cross-domain mutual learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 1501-1510.
[29] Li Z, Zou H, Sun X, et al. 3d expression-invariant face verification based on transfer learning and siamese network for small sample size[J]. Electronics, 2021, 10(17): 2128.
[30] Cardia Neto J B. 3D face recognition with descriptor images and shallow convolutional neural networks[J]. 2020.
[31] Feng Z, Zhao Q. Robust face recognition with deeply normalized depth images[C]//Biometric Recognition: 13th Chinese Conference, CCBR 2018, Urumqi, China, August 11- 12, 2018, Proceedings 13. Springer International Publishing, 2018: 418-427.
[32] Huang Y H, Chen H H. Deep face recognition for dim images[J]. Pattern Recognition, 2022, 126: 108580.
[33] Zhao P, Ming Y, Meng X, et al. LMFNet: A lightweight multiscale fusion network with hierarchical structure for lowquality 3-D face recognition[J]. IEEE Transactions on Human- Machine Systems, 2022, 53(1): 239-252.
[34] Khan F, Shariff W, Farooq M A, et al. A robust lightweight fused-feature encoder-decoder model for monocular facial depth estimation from single images trained on synthetic data[J]. IEEE Access, 2023.
[35] Ding C, Tao D. Robust face recognition via multimodal deep face representation[J]. IEEE transactions on Multimedia, 2015, 17(11): 2049-2058.
[36] Hu W. Improving 2D face recognition via fine-level facial depth generation and RGB-D complementary feature learning[J]. arXiv preprint arXiv:2305.04426, 2023.
[37] Cai Y, Lei Y, Yang M, et al. A fast and robust 3D face recognition approach based on deeply learned face representation[J]. Neurocomputing, 2019, 363: 375-397.
[38] Zeng X, Peng X, Qiao Y. Df2net: A dense-fine-finer network for detailed 3d face reconstruction[C]//Proceedings of the IEEE/ CVF International Conference on Computer Vision. 2019: 2315- 2324.
[39] Garg S, Mittal S, Kumar P, et al. DeBNet: multilayer deep network for liveness detection in face recognition system[C]//2020 7th International Conference on Signal Dean&Francis Processing and Integrated Networks (SPIN). IEEE, 2020: 1136- 1141.
[40] Grati N, Ben-Hamadou A, Hammami M. Learning local representations for scalable RGB-D face recognition[J]. Expert Systems With Applications, 2020, 150: 113319.
[41] Uppal H, Sepas-Moghaddam A, Greenspan M, et al. Depth as attention for face representation learning[J]. IEEE Transactions on Information Forensics and Security, 2021, 16: 2461-2476.
[42] Zhang F, Liu N, Duan F. Coarse-to-Fine depth superresolution with adaptive RGB-D feature attention[J]. IEEE Transactions on Multimedia, 2023.
[43] Lin W C, Chiu C T, Shih K C. RGB-D Based Pose-Invariant Face Recognition Via Attention Decomposition Module[C]// ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023: 1-5.
[44] Wang H, Li M, Li S, et al. Exploring Depth Information for Face Manipulation Detection[J]. arXiv preprint arXiv:2212.14230, 2022.
[45] Yu C, Zhang Z, Li H, et al. Meta-learning-based adversarial training for deep 3D face recognition on point clouds[J]. Pattern Recognition, 2023, 134: 109065.
[46] Jin B, Cruz L, Goncalves N. Pseudo RGB-D face recognition[J]. IEEE Sensors Journal, 2022, 22(22): 21780- 21794.
[47] Gecer B, Bhattarai B, Kittler J, et al. Semi-supervised adversarial learning to generate photorealistic face images of new identities from 3d morphable model[C]//Proceedings of the European conference on computer vision (ECCV). 2018: 217- 234.
[48] Uppal H. Attention and Depth Hallucination for RGB-D Face Recognition with Deep Learning[D]. Queen’s University (Canada), 2021.
[49] Uppal H, Sepas-Moghaddam A, Greenspan M, et al. Two-level attention-based fusion learning for RGB-D face recognition[C]//2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021: 10120-10127.
[50] Ghosh S, Singh R, Vatsa M, et al. RGB-D face recognition using reconstruction based shared representation[C]//2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021). IEEE, 2021: 1-8.
[51] Zhu Y, Gao J, Wu T, et al. Exploiting enhanced and robust RGB-D face representation via progressive multimodal learning[J]. Pattern Recognition Letters, 2023, 166: 38-45.
[52] Lin S, Jiang C, Liu F, et al. High quality facial data synthesis and fusion for 3D low-quality face recognition[C]//2021 IEEE International Joint Conference on Biometrics (IJCB). IEEE, 2021: 1-8.
[53] Thamizharasan V, Das A, Battaglino D, et al. Face attribute analysis from s
[54] Pecoraro R, Basile V, Bono V. Local multi-head channel self-attention for facial expression recognition[J]. Information, 2022, 13(9): 419. tructured light: an end-to-end approach[J]. Multimedia Tools and Applications, 2023, 82(7): 10471-10490.
[55] You Z, Yang T, Jin M. Multi-channel deep 3D face recognition[J]. arXiv preprint arXiv:2009.14743, 2020.
[56] Lin T Y, Chiu C T, Tang C T. RGB-D based multimodal deep learning for face identification[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020: 1668-1672.
[57] Zhao P, Ming Y, Hu N, et al. DSNet: Dual-stream multiscale fusion network for low-quality 3D face recognition[J]. AIP Advances, 2023, 13(8).
[58] Zhang F, Liu N, Chang L, et al. Edge‐guided single facial depth map super‐resolution using CNN[J]. IET Image Processing, 2020, 14(17): 4708-4716.
[59] Cardia Neto J B. 3D face recognition with descriptor images and shallow convolutional neural networks[J]. 2020.
[60] Jiang L, Zhang J, Li C, et al. Rgb-d face recognition via spatial and channel attentions[C]//2021 IEEE 5th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC). IEEE, 2021, 5: 2037-2041.
[61] Li H, Sui M, Zhu Z, et al. MFEViT: A Robust Lightweight Transformer-based Network for Multimodal 2D+ 3D Facial Expression Recognition[J]. arXiv preprint arXiv:2109.13086, 2021.
[62] Zheng H, Wang W, Wen F, et al. A complementary fusion strategy for rgb-d face recognition[C]//International Conference on Multimedia Modeling. Cham: Springer International Publishing, 2022: 339-351.
[63] Khan F, Farooq M A, Shariff W, et al. Towards monocular neural facial depth estimation: Past, present, and future[J]. IEEE Access, 2022, 10: 29589-29611.
[64] Jiang L, Zhang J, Deng B. Robust RGB-D face recognition using attribute-aware loss[J]. IEEE transactions on pattern analysis and machine intelligence, 2019, 42(10): 2552-2566.
[65] Zhang W, Shu Z, Samaras D, et al. Improving heterogeneous face recognition with conditional adversarial networks[J]. arXiv preprint arXiv:1709.02848, 2017.
[66] Kemelmacher-Shlizerman I, Basri R. 3D face reconstruction from a single image using a single reference face shape[J]. IEEE transactions on pattern analysis and machine intelligence, 2010, 33(2): 394-405.
[67] Dib A, Thebault C, Ahn J, et al. Towards high fidelity monocular face reconstruction with rich reflectance using selfsupervised learning and ray tracing[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021: 12819-12829.
[68] Xiao M, Yi H, Huang Y, et al. Effective Key Region- Guided Face Detail Optimization Algorithm for 3D Face Reconstruction[J]. Journal of Sensors, 2022, 2022.
[69] Petkova R, Manolova A, Tonchev K, et al. 3D face reconstruction and verification using multi-view RGB-D data[C]//2022 Global Conference on Wireless and Optical Technologies (GCWOT). IEEE, 2022: 1-6.
[70] Koestinger M, Wohlhart P, Roth P M, et al. Annotated facial landmarks in the wild: A large-scale, real-world database for facial landmark localization[C]//2011 IEEE international Dean&Francis conference on computer vision workshops (ICCV workshops). IEEE, 2011: 2144-2151.
[71] Devi P R S, Baskaran R. SL2E-AFRE: Personalized 3D face reconstruction using autoencoder with simultaneous subspace learning and landmark estimation[J]. Applied Intelligence, 2021, 51(4): 2253-2268.
[72] Belhumeur P N, Jacobs D W, Kriegman D J, et al. Localizing parts of faces using a consensus of exemplars[J]. IEEE transactions on pattern analysis and machine intelligence, 2013, 35(12): 2930-2940.
[73] Le V, Brandt J, Lin Z, et al. Interactive facial feature localization[C]//Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part III 12. Springer Berlin Heidelberg, 2012: 679-692.
[74] Huang G B, Learned-Miller E. Labeled faces in the wild: Updates and new reporting procedures[J]. Dept. Comput. Sci., Univ. Massachusetts Amherst, Amherst, MA, USA, Tech. Rep, 2014, 14(003).
[75] Yi D, Lei Z, Liao S, et al. Learning face representation from scratch[J]. arXiv preprint arXiv:1411.7923, 2014.
[76] Sengupta S, Chen J C, Castillo C, et al. Frontal to profile face verification in the wild[C]//2016 IEEE winter conference on applications of computer vision (WACV). IEEE, 2016: 1-9.
[77] Kumar N, Berg A C, Belhumeur P N, et al. Attribute and simile classifiers for face verification[C]//2009 IEEE 12th international conference on computer vision. IEEE, 2009: 365- 372.
[78] Gao W, Cao B, Shan S, et al. The CAS-PEAL largescale Chinese face database and baseline evaluations[J]. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 2007, 38(1): 149-161.
[79] Zhu Z, Huang G, Deng J, et al. Webface260m: A benchmark unveiling the power of million-scale deep face recognition[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021: 10492-10502.
[80] Yang S, Luo P, Loy C C, et al. Wider face: A face detection benchmark[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 5525-5533.
[81] Klare B F, Klein B, Taborsky E, et al. Pushing the frontiers of unconstrained face detection and recognition: Iarpa janus benchmark a[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2015: 1931-1939.
[82] Liu Z, Luo P, Wang X, et al. Deep learning face attributes in the wild[C]//Proceedings of the IEEE international conference on computer vision. 2015: 3730-3738
[83] Heseltine T, Pears N, Austin J. Three-dimensional face recognition using combinations of surface feature map subspace components[J]. Image and Vision Computing, 2008, 26(3): 382- 396.
[84] Savran A, Alyüz N, Dibeklioğlu H, et al. Bosphorus database for 3D face analysis[C]//Biometrics and Identity Management: First European Workshop, BIOID 2008, Roskilde, Denmark, May 7-9, 2008. Revised Selected Papers 1. Springer Berlin Heidelberg, 2008: 47-56.
[85] Colombo A, Cusano C, Schettini R. UMB-DB: A database of partially occluded 3D faces[C]//2011 IEEE international conference on computer vision workshops (ICCV workshops). IEEE, 2011: 2113-2119.
[86] Vijayan V, Bowyer K W, Flynn P J, et al. Twins 3D face recognition challenge[C]//2011 international joint conference on biometrics (IJCB). IEEE, 2011: 1-7.
[87] Gupta S, Castleman K R, Markey M K, et al. Texas 3D face recognition database[C]//2010 IEEE Southwest Symposium on Image Analysis & Interpretation (SSIAI). IEEE, 2010: 97-100.
[88] Phillips P J, Flynn P J, Scruggs T, et al. Overview of the face recognition grand challenge[C]//2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05). IEEE, 2005, 1: 947-954.
[89] Zhang J, Huang D, Wang Y, et al. Lock3dface: A largescale database of low-cost kinect 3d faces[C]//2016 International Conference on Biometrics (ICB). IEEE, 2016: 1-8.
[90] Urbanová P, Ferková Z, Jandová M, et al. Introducing the FIDENTIS 3D face database[J]. AnthropologicAl review, 2018, 81(2): 202-223.
[91] Goswami G, Bharadwaj S, Vatsa M, et al. On RGB-D face recognition using Kinect[C]//2013 IEEE Sixth International Conference on Biometrics: Theory, Applications and Systems (BTAS). IEEE, 2013: 1-6.
[92] Min R, Kose N, Dugelay J L. Kinectfacedb: A kinect database for face recognition[J]. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2014, 44(11): 1534-1548.
[93] Yin L, Wei X, Sun Y, et al. A 3D facial expression database for facial behavior research[C]//7th international conference on automatic face and gesture recognition (FGR06). IEEE, 2006: 211-216.
[94] La Cava S M, Orrù G, Goldmann T, et al. 3D face reconstruction for forensic recognition-a survey[C]//2022 26th International Conference on Pattern Recognition (ICPR). IEEE, 2022: 930-937.
[95] Wang M, Deng W. Deep face recognition: A survey[J]. Neurocomputing, 2021, 429: 215-244.
[96] Liu F, Chen D, Wang F, et al. Deep learning based single sample face recognition: a survey[J]. Artificial Intelligence Review, 2023, 56(3): 2723-2748.
[97] Zhou S, Xiao S. 3D face recognition: a survey[J]. Humancentric Computing and Information Sciences, 2018, 8(1): 35.
[98] Ning X, Nan F, Xu S, et al. Multi‐view frontal face image generation: a survey[J]. Concurrency and Computation: Practice and Experience, 2023, 35(18): e6147.
[99] Sharma S, Kumar V. 3d face reconstruction in deep learning era: A survey[J]. Archives of Computational Methods in Engineering, 2022, 29(5): 3475-3507.
[100] Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural networks[J]. Advances in neural information processing systems, 2012, 25.
Downloads
Published
Issue
Section
License
Copyright (c) 2024 by the authors.
This work is licensed under a Creative Commons Attribution 4.0 International License.
