微信服务号

微信订阅号

2025-5-30- 17
Home > Archive>Volume 35, Issue 5, 2024 >2176-2191. DOI:10.13328/j.cnki.jos.007029
PDF HTML XML Export Cite reminder
AU-aware Algorithm for 3D Facial Reconstruction
Author:
Affiliation:

  • Article
    • | |
  • Metrics
    • |
  • Reference [61]
    • |
  • Related [20]
    • | | |
  • Comments
    Abstract:

    As a critical task in computer vision and animation, facial reconstruction can provide 3D model structures and rich semantic information for multi-modal facial applications. However, monocular 2D facial images lack depth information and the parameters of the predicted facial model are not reliable, which causes poor reconstruction results. This study proposes to employ facial action unit (AU) and facial keypoints which are highly correlated with model parameters as a bridge to guide the regression of model-related parameters and thus solve the ill-posed monocular facial reconstruction. Based on existing facial reconstruction datasets, this study provides a complete semi-automatic labeling scheme for facial AUs and constructs a 300W-LP-AU dataset. Furthermore, a 3D facial reconstruction algorithm based on AU awareness is put forward to realize end-to-end multi-tasking learning and reduce the overall training difficulty. Experimental results show that it improves the facial reconstruction performance, with high fidelity of the reconstructed facial model.

    Reference
    [1] Paysan P, Knothe R, Amberg B, Romdhani S, Vetter T. A 3D face model for pose and illumination invariant face recognition. In:Proc. of the 6th IEEE Int'l Conf. on Advanced Video and Signal Based Surveillance. Genova:IEEE, 2009. 296-301.
    [2] Luo C, Song SY, Xie WC, Shen LL, Gunes H. Learning multi-dimensional edge feature-based au relation graph for facial action unit recognition. In:Proc. of the 31st Int'l Joint Conf. on Artificial Intelligence. Vienna:IJCAI, 2022. 1239-1246.
    [3] Zhang R, Tsai PS, Cryer JE, Shah M. Shape-from-shading:A survey. IEEE Trans. on Pattern Analysis and Machine Intelligence, 1999, 21(8):690-706.
    [4] Quéau Y, Mélou J, Castan F, Cremers D, Durou JD. A variational approach to shape-from-shading under natural illumination. In:Proc. of the 11th Int'l Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition. Venice:Springer, 2018. 342-357.
    [5] 曾小星. 深度人脸检测与三维重建方法研究[博士学位论文]. 深圳:中国科学院大学(中国科学院深圳先进技术研究院), 2021.
    Zeng XX. Deep learning methods for face detection and 3D reconstruction[Ph.D. Thesis]. Shenzheng:Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, 2021.
    [6] Blanz V, Vetter T. A morphable model for the synthesis of 3D faces. In:Proc. of the 26th Annual Conf. on Computer Graphics and Interactive Techniques. Los Angeles:ACM, 1999. 187-194.
    [7] Jourabloo A, Liu XM. Large-pose face alignment via CNN-based dense 3D model fitting. In:Proc. of the 2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). Las Vegas:IEEE, 2016. 4188-4196.
    [8] Sela M, Richardson E, Kimmel R. Unrestricted facial geometry reconstruction using image-to-image translation. In:Proc. of the 2017 IEEE Int'l Conf. on Computer Vision (ICCV). Venice:IEEE, 2017. 1585-1594.
    [9] Tran AT, Hassner T, Masi I, Medioni G. Regressing robust and discriminative 3D morphable models with a very deep neural network. In:Proc. of the 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). Honolulu:IEEE, 2017. 1493-1502.
    [10] Yi HW, Li C, Cao Q, Shen XY, Li S, Wang GP, Tai YW. MMFace:A multi-metric regression network for unconstrained face reconstruction. In:Proc. of the 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). Long Beach:IEEE, 2019. 7655-7664.
    [11] Wu F, Bao L, Chen Y, et al. MVF-net:Multi-view 3D face morphable model regression. In:Proc. of the 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). Long Beach:IEEE, 2019. 959-968.
    [12] Chen AP, Chen Z, Zhang GL, Mitchell K, Yu JY. Photo-realistic facial details synthesis from single image. In:Proc. of the 2019 IEEE/CVF Int'l Conf. on Computer Vision (ICCV). Seoul:IEEE, 2019. 9428-9438.
    [13] Lattas A, Moschoglou S, Gecer B, Ploumpis S, Triantafyllou V, Ghosh A, Zafeiriou S. AvatarMe:Realistically renderable 3D facial reconstruction "in-the-wild". In:Proc. of the 2020 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). Seattle:IEEE, 2020. 757-766.
    [14] van Rootseler RTA, Spreeuwers LJ, Veldhuis RNJ. Using 3D morphable models for face recognition in video. In:Proc. of the 33rd Symp. on Information Theory in the Benelux and the 2nd Joint WIC/IEEE Symp. on Information Theory and Signal Processing in the Benelux. Boekelo:WIC, 2012. 235-242. http://purl.utwente.nl/publications/80462
    [15] Lee YJ, Lee SJ, Park KR, Jo J, Kim J. Single view-based 3D face reconstruction robust to self-occlusion. EURASIP Journal on Advances in Signal Processing, 2012, 2012:176.
    [16] Huber P, Feng ZH, Christmas W, Kittler J, Rätsch M. Fitting 3D morphable face models using local features. In:Proc. of the 2015 IEEE Int'l Conf. on Image Processing (ICIP). Quebec City:IEEE, 2015. 1195-1199.
    [17] Romdhani S, Vetter T. Estimating 3D shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior. In:Proc. of the 2005 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. San Diego:IEEE, 2005. 986-993.
    [18] Zhu XY, Lei Z, Yan JJ, Yi D, Li SZ. High-fidelity pose and expression normalization for face recognition in the wild. In:Proc. of the 2015 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). Boston:IEEE, 2015. 787-796.
    [19] Zhao SW, Wang XM, Zhang DH, Zhang GY, Wang ZY, Liu HH. FM-3DFR:Facial manipulation-based 3-D face reconstruction. IEEE Trans. on Cybernetics, 2024, 54(1):209-218.
    [20] Chen YJ, Wu FZ, Wang ZY, Song YB, Ling YG, Bao LC. Self-supervised learning of detailed 3D face reconstruction. IEEE Trans. on Image Processing, 2020, 29:8696-8705.
    [21] Wu CY, Xu QG, Neumann U. Synergy between 3DMM and 3D landmarks for accurate 3D facial geometry. In:Proc. of the 2021 Int'l Conf. on 3D Vision (3DV). London:IEEE, 2021. 453-463.
    [22] Jung H, Oh MS, Lee SW. Learning free-form deformation for 3D face reconstruction from in-the-wild images. In:Proc. of the 2021 IEEE Int'l Conf. on Systems, Man, and Cybernetics (SMC). Melbourne:IEEE, 2021. 2737-2742.
    [23] Chen Z, Guan T, Wang YS, Luo YW, Xu LY, Liu WK. Learning 3-D face shape from diverse sources with cross-domain face synthesis. IEEE MultiMedia, 2023, 30(1):7-16.
    [24] Zhu XY, Liu XM, Lei Z, Li SZ. Face alignment in full pose range:A 3D total solution. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019, 41(1):78-92.
    [25] Zhu XY, Lei Z, Liu XM, Shi HL, Li SZ. Face alignment across large poses:A 3D solution. In:Proc. of the 2016 IEEE Conf. on Computer Vision and Pattern Recognition. Las Vegas:IEEE, 2016. 146-155.
    [26] Feng Y, Wu F, Shao XH, Wang YF, Zhou X. Joint 3D face reconstruction and dense alignment with position map regression network. In:Proc. of the 15th European Conf. on Computer Vision (ECCV). Munich:Springer, 2018. 557-574.
    [27] Guo JZ, Zhu XY, Yang Y, Yang F, Lei Z, Li SZ. Towards fast, accurate and stable 3D dense face alignment. In:Proc. of the 16th European Conf. on Computer Vision. Glasgow:Springer, 2020. 152-168.
    [28] Mukherjee SS, Robertson NM. Deep head pose:Gaze-direction estimation in multimodal video. IEEE Trans. on Multimedia, 2015, 17(11):2094-2107.
    [29] Ruiz N, Chong E, Rehg JM. Fine-grained head pose estimation without keypoints. In:Proc. of the 2018 IEEE Conf. on Computer Vision and Pattern Recognition Workshops. Salt Lake City:IEEE, 2018. 2074-2083.
    [30] Hsu HW, Wu TY, Wan S, Wong WH, Lee CY. QuatNet:Quaternion-based head pose estimation with multiregression loss. IEEE Trans. on Multimedia, 2019, 21(4):1035-1046.
    [31] Yang TY, Chen YT, Lin YY, Chuang YY. FSA-NET:Learning fine-grained structure aggregation for head pose estimation from a single image. In:Proc. of the 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Long Beach:IEEE, 2019. 1087-1096.
    [32] Cao ZW, Chu ZC, Liu DF, Chen YJ. A vector-based representation to enhance head pose estimation. In:Proc. of the 2021 IEEE Winter Conf. on Applications of Computer Vision. Waikoloa:IEEE, 2021. 1187-1196.
    [33] Hempel T, Abdelrahman AA, Al-Hamadi A. 6d rotation representation for unconstrained head pose estimation. In:Proc. of the 2022 IEEE Int'l Conf. on Image Processing (ICIP). Bordeaux:IEEE, 2022. 2496-2500.
    [34] Rosenberg EL, Ekman P. What the Face Reveals:Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). 3rd ed., Oxford:Oxford University Press, 2020.
    [35] 李勇, 曾加贝, 刘昕, 山世光. 面部动作单元检测方法进展与挑战. 中国图象图形学报, 2020, 25(11):2293-2305.
    Li Y, Zeng JB, Liu X, Shan SG. Progress and challenges in facial action unit detection. Journal of Image and Graphics, 2020, 25(11):2293-2305.
    [36] 闫衍芙, 吕科, 薛健, 王聪, 甘玮. 基于深度学习和表情AU参数的人脸动画方法. 计算机辅助设计与图形学学报, 2019, 31(11):1973-1980.
    Yan YF, Lyu K, Xue J, Wang C, Gan W. Facial animation method based on deep learning and expression AU parameters. Journal of Computer-aided Design & Computer Graphics, 2019, 31(11):1973-1980.
    [37] Cohn JF, Ambadar Z, Ekman P. Observer-based measurement of facial expression with the Facial Action Coding System. In:Coan JA, Allen JJB, eds. Handbook of Emotion Elicitation and Assessment. Oxford:Oxford University Press, 2007. 203-221.
    [38] Gan W, Xue J, Lu K, Yan YF, Gao PC, Lyu JY. FEAFA+:An extended well-annotated dataset for facial expression analysis and 3D facial animation. In:Proc. of the 14th Int'l Conf. on Digital Image Processing (ICDIP 2022). Wuhan:SPIE, 2022. 1234211.
    [39] Yan YF, Lu K, Xue J, Gao PC, Lyu JY. FEAFA:A well-annotated dataset for facial expression analysis and 3D facial animation. In:Proc. the 2019 IEEE Int'l Conf. on Multimedia & Expo Workshops (ICMEW). Shanghai:IEEE, 2019. 96-101.
    [40] Yin X, Liu XM. Multi-task convolutional neural network for pose-invariant face recognition. IEEE Trans. on Image Processing, 2018, 27(2):964-975.
    [41] Feng ZH, Kittler J, Awais M, Huber P, Wu XJ. Wing loss for robust facial landmark localisation with convolutional neural networks. In:Proc. of the 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Salt Lake City:IEEE, 2018. 2235-2245.
    [42] Ruan ZY, Zou CQ, Wu LH, Wu GS, Wang LM. SADRNet:Self-aligned dual face regression networks for robust 3D dense face alignment and reconstruction. IEEE Trans. on Image Processing, 2021, 30:5793-5806.
    [43] Köstinger M, Wohlhart P, Roth PM, Bischof H. Annotated facial landmarks in the wild:A large-scale, real-world database for facial landmark localization. In:Proc. of the 2011 IEEE Int'l Conf. on Computer Vision Workshops (ICCV Workshops). Barcelona:IEEE, 2011. 2144-2151.
    [44] Bulat A, Tzimiropoulos G. How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In:Proc. of the 2017 IEEE Int'l Conf. on Computer Vision. Venice:IEEE, 2017. 1021-1030.
    [45] Sandler M, Howard A, Zhu ML, Zhmoginov A, Chen LC. Mobilenetv2:Inverted residuals and linear bottlenecks. In:Proc. of the 2018 IEEE Conf. on Computer Vision and Pattern Recognition. Salt Lake City:IEEE, 2018. 4510-4520.
    [46] He KM, Zhang XY, Ren SQ, Sun J. Deep residual learning for image recognition. In:Proc. of the 2016 IEEE Conf. on Computer Vision and Pattern Recognition. Las Vegas:IEEE, 2016. 770-778.
    [47] Cao XD, Wei YC, Wen F, Sun J. Face alignment by explicit shape regression. Int'l Journal of Computer Vision, 2014, 107(2):177-190.
    [48] Yu R, Saito S, Li HX, Ceylan D, Li H. Learning dense facial correspondences in unconstrained images. In:Proc. of the 2017 IEEE Int'l Conf. on Computer Vision. Venice:IEEE, 2017. 4733-4742.
    [49] Bhagavatula C, Zhu CC, Luu K, Savvides M. Faster than real-time facial alignment:A 3D spatial transformer network approach in unconstrained poses. In:Proc. of the 2017 IEEE Int'l Conf. on Computer Vision. Venice:IEEE, 2017. 4000-4009.
    [50] Tu XG, Zhao J, Xie M, Jiang ZH, Balamurugan A, Luo Y, Zhao Y, He LX, Ma Z, Feng JS. 3D face reconstruction from a single image assisted by 2D face images in the wild. IEEE Trans. on Multimedia, 2021, 23(99):1160-1172.
    [51] Sun B, Shao M, Xia SY, Fu Y. Deep evolutionary 3D diffusion heat maps for large-pose face alignment. In:Proc. of the 2018 British Machine Vision Conf. 2018. Newcastle:BMVC, 2018. 256.
    [52] Shang JX, Shen TW, Li SW, Zhou L, Zhen MM, Fang T, Quan L. Self-supervised monocular 3D face reconstruction by occlusion-aware multi-view geometry consistency. In:Proc. of the 16th European Conf. on Computer Vision. Glasgow:Springer, 2020. 53-70.
    [53] Deng Y, Yang JL, Xu SC, Chen D, Jia YD, Tong X. Accurate 3D face reconstruction with weakly-supervised learning:From single image to image set. In:Proc. of the 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition Workshops. Long Beach:IEEE, 2019. 285-295.
    [54] Yang TY, Huang YH, Lin YY, Hsiu PC, Chuang YY. SSR-Net:A compact soft stagewise regression network for age estimation. In:Proc. of the 27th Int'l Joint Conf. on Artificial Intelligence. Stockholm:IJCAI, 2018. 1078-1084.
    [55] Liu ZX, Chen ZZ, Bai JQ, Li SH, Lian SG. Facial pose estimation by deep learning from label distributions. In:Proc. of the 2019 IEEE/CVF Int'l Conf. on Computer Vision Workshop. Seoul:IEEE, 2019. 1232-1240.
    [56] Zhang H, Wang MM, Liu Y, Yuan Y. FDN:Feature decoupling network for head pose estimation. In:Proc. of the 2020 AAAI Conf. on Artificial Intelligence. New York:AAAI, 2020. 12789-12796.
    [57] Valle R, Buenaposada JM, Baumela L. Multi-task head pose estimation in-the-wild. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2021, 43(8):2874-2881.
    [58] Liu YJ, Jourabloo A, Ren W, Liu XM. Dense face alignment. In:Proc. of the 2017 IEEE Int'l Conf. on Computer Vision Workshops. Venice:IEEE, 2017. 1619-1628.
    Cited by
    Comments
    Comments
    分享到微博
    Submit
Get Citation

章毅,吕嘉仪,兰星,薛健.结合面部动作单元感知的三维人脸重建算法.软件学报,2024,35(5):2176-2191

Copy
Share
Article Metrics
  • Abstract:956
  • PDF: 3801
  • HTML: 1627
  • Cited by: 0
History
  • Received:April 10,2023
  • Revised:June 08,2023
  • Online: September 11,2023
  • Published: May 06,2024
Links:Institute of Software, CASChina Computer FederationCASNSFC
You are the first2049223Visitors
Copyright: Institute of Software, Chinese Academy of Sciences Beijing ICP No. 05046678-4
Address：4# South Fourth Street, Zhong Guan Cun, Beijing 100190,Postal Code：100190
Phone：010-62562563 Fax：010-62562533 Email：jos@iscas.ac.cn
Technical Support：Beijing Qinyun Technology Development Co., Ltd.

Beijing Public Network Security No. 11040202500063