Seeing Speech: Learning Visible Articulatory Dynamics for Speech-Driven 3D Facial Animation

Recent progress in speech-driven 3D facial animation has improved vertex-level reconstruction quality, but speech-consistent visible articulation remains difficult. This is because speech production follows structured and constrained articulators' coordination and the mapping from acoustics to motion is inherently one-to-many. Motivated by the structured patterns of visible articulation, we propose a novel articulation-aware framework that models visible speech through directional articulatory motions and composes them into surface-consistent 3D facial motion. To represent visible articulation with three directional articulatory motions, spreading, opening, and protrusion, we propose a Speech–Articulatory Memory (SAM) that captures the correspondence between speech and these motions under phonetic context through retrieval and decoding based on a key-value memory structure. Then, a Topology-aware Articulatory Composition (TAC) integrates the predicted directional articulatory motions under mesh topology to produce surface-consistent 3D facial motion. Experiments on VOCASET and TFHP show that our method achieves state-of-the-art performance on standard reconstruction metrics and improves visible articulatory distance and velocity errors for lip articulation, while a user study confirms clear preference in lip sync and realism.

Our method achieves the lowest FVE, LVE, and LDTW on both VOCASET and TFHP, showing that articulatory modeling improves lip-region reconstruction while preserving overall reconstruction quality.

We measure inter-commissural width (ICW, lip spreading), inter-labial distance (ILD, lip opening), and lip protrusion (LP) from lip-surface anchors. Our method achieves the lowest distance errors on all three measures and the lowest velocity errors on ILD and LP.

Beyond per-frame accuracy, we check whether each method preserves the relative magnitude of directional articulatory motions. For each paired motion (e.g., opening–closing), we compare predicted and ground-truth magnitudes and plot them relative to the ground truth (1.0). Our method stays closest to the ground truth across all directions, indicating more balanced directional articulation.

Mesh renderings (top) and per-vertex L1 error maps against the ground truth (bottom, 0 in blue to 0.015 in red) on TFHP at phoneme-aligned frames. Our method produces distinct articulatory configurations at salient phonemes with low errors across the lips and lower face, while the upper face remains comparable to competing methods.

(a) Lip ROI and landmarks defining the horizontal (LMC–RMC) and vertical (UL–LL) slit lines. (b) Horizontal and (c) vertical slit-scans over time for the utterance adopted. Our trajectories show distinct rounding/spreading and opening/closing patterns with steeper slopes at articulatory transitions, matching the ground truth.

Trajectories along spreading–rounding, opening–closing, and protrusion–retraction, aligned with the speech waveform and per-phoneme mesh renderings. Baselines deviate in direction-specific ways: flat opening trajectories at opening-dominant phonemes (e.g., /aɪ/, /æ/, /aʊ/), attenuated protrusion at protrusion-dominant phonemes (e.g., /tr/, /v/), or noisy spreading at high-magnitude segments. Our method closely follows the ground truth in all three directions.

Pairwise A/B preference study on VOCASET with 24 valid participants (15 clips per competitor). Our method is preferred over every competitor, with average preference of 70.3% for lip sync and 70.7% for realism (one-sided binomial test, $p<0.0001$ for every competitor).