CVPR 2022 · New Orleans
EMOCA
Emotion Driven Monocular Face Capture and Animation
Max Planck Institute for Intelligent Systems, Germany
Capturing emotion in 3D.
As 3D facial avatars become more widely used for communication, it is critical that they faithfully convey emotion. Unfortunately, the best recent methods that regress parametric 3D face models from monocular images are unable to capture the full spectrum of facial expression, such as subtle or extreme emotions. We find the standard reconstruction metrics used for training (landmark reprojection error, photometric error, and face recognition loss) are insufficient to capture high-fidelity expressions.
The result is facial geometries that do not match the emotional content of the input image. We address this with EMOCA (EMOtion Capture and Animation), by introducing a novel deep perceptual emotion consistency loss during training, which helps ensure that the reconstructed 3D expression matches the expression depicted in the input image.
While EMOCA achieves 3D reconstruction errors that are on par with the current best methods, it significantly outperforms them in terms of the quality of the reconstructed expression and the perceived emotional content. We also directly regress levels of valence and arousal and classify basic expressions from the estimated 3D face parameters.
On the task of in-the-wild emotion recognition, our purely geometric approach is on par with the best image-based methods, highlighting the value of 3D geometry in analyzing human behavior. The model and code are publicly available.
Expressive reconstructions
Examples from the original project showing facial geometry and expression recovered from monocular input.
Video presentation
Expression capture in motion
DECA and EMOCA with detail displacements, applied to the same video sequence. Compare how each reconstruction preserves the changing expression over the full three-minute sequence.
Transfer the expression to a new face
The recovered expression can animate other identities. The original reconstruction and three target identities play in sync below, driven by the same Aff-Wild2 sequence.
Examples from the EMOCA presentation. These extend the short animated previews with full video comparisons and retargeting results.
Explore the work
Original downloads · MPI research page · Questions: emoca@tue.mpg.de
Project updates
- February 2023 — EMOCA v2 released, with better lip sync, landmark alignment, and eye closure.
- April 2022 — Supplementary video, code, trained models, and preprint available.
- March 2022 — Accepted to CVPR 2022.
Citation
@inproceedings{EMOCA:CVPR:2021,
title = {{EMOCA}: {E}motion Driven Monocular Face Capture and Animation},
author = {Danecek, Radek and Black, Michael J. and Bolkart, Timo},
booktitle = {Conference on Computer Vision and Pattern Recognition (CVPR)},
pages = {20311--20322},
year = {2022}
}