SIGGRAPH Asia 2023 · Conference
EMOTE
Emotional Speech-Driven Animation with Content-Emotion Disentanglement
1 Max Planck Institute for Intelligent Systems, Germany
2 KTH Royal Institute of Technology, Sweden
Speech with expression.
Given audio input and an emotion label, EMOTE generates an animated 3D head that has state-of-the-art lip synchronization while expressing the emotion. The method is trained from 2D video sequences using a novel video emotion loss and a mechanism to disentangle emotion from speech.
One voice. Eight emotions.
Keep the speech fixed and change the emotion condition. Each row shows the same utterance with eight different expressions, across two identities.
Audio input: LRS3. Turn on sound to compare lip articulation. On a narrow screen, scroll the comparison horizontally or open the full-size video.
Second identity · M022
Comparisons with prior methods
Speech-driven animation from VOCA, MeshTalk, CodeTalker, FaceFormer, and EMOTE. The training dataset is shown beneath each method; all columns play together with the same input audio.
VOCASETMeshTalk
Private datasetCodeTalker
VOCASETFaceFormer
VOCASETFaceFormer
MEADEMOTE
MEAD
Example 2
VOCASETMeshTalk
Private datasetCodeTalker
VOCASETFaceFormer
VOCASETFaceFormer
MEADEMOTE
MEAD
Example 3
VOCASETMeshTalk
Private datasetCodeTalker
VOCASETFaceFormer
VOCASETFaceFormer
MEADEMOTE
MEAD
What makes EMOTE work?
Compare the full model with variants that remove disentanglement, emotion supervision, lip-reading, FLINT, or perceptual losses, and with a static emotion loss and FaceFormer EMO. See the paper for the full ablation discussion.
disentanglementWithout
emotion lossWithout
lip-readingWithout
FLINTWithout
perceptual lossesStatic
emotion lossFaceFormer
EMO
Example 2
disentanglementWithout
emotion lossWithout
lip-readingWithout
FLINTWithout
perceptual lossesStatic
emotion lossFaceFormer
EMO
Example 3
disentanglementWithout
emotion lossWithout
lip-readingWithout
FLINTWithout
perceptual lossesStatic
emotion lossFaceFormer
EMO
Change the emotion as speech unfolds
A gradual interpolation of the emotion condition (left) compared with a sudden change (right). Each pair shares a single player to keep the speech and animations synchronized.
Sad → angry
Happy → sad
Neutral → disgust
See EMOTE in motion
Watch the supplementary video for animated results and an explanation of content–emotion disentanglement.
Use the player controls to turn on audio. Open or download the video ↗
Explore the work
Code and models are available through the GitHub repository. See the repository for current training-code and data availability.
Citation
@inproceedings{EMOTE,
title = {Emotional Speech-Driven Animation with Content-Emotion Disentanglement},
author = {Daněček, Radek and Chhatre, Kiran and Tripathi, Shashank and Wen, Yandong and Black, Michael and Bolkart, Timo},
publisher = {ACM},
month = dec,
year = {2023},
doi = {10.1145/3610548.3618183},
url = {https://emote.is.tue.mpg.de/index.html},
month_numeric = {12}
}