AnyTalk generates 3D speech animation for arbitrary characters without animation data

Researchers presented AnyTalk, a method for generating 3D speech animation for arbitrary characters that requires neither animation data nor manual rigging. Instead, the authors fine-tune a ready-made video diffusion model on renders of the target character with "zero" audio embeddings, and then lift the resulting talking-head video into 3D by optimizing blendshape parameters. Code is promised on the project page, but as of the publication of this news note, the authors have not released it. Details are in the arXiv paper.

What it means

AnyTalk does not directly compete with the video generators in our AI video section: it tackles the task of 3D speech animation rather than generating clips from text or images. However, the method relies on a video diffusion model as a source of motion prior — that is, the same class of models as LTX-Video, CogVideoX, and Pyramid Flow. The fact sheets for these tools contain no information about fine-tuning for a specific character or lifting video into 3D animation: their tasks are text-to-video and image-to-video. AnyTalk shows that a video diffusion model can be adapted to the narrow task of lip sync with minimal data, but practical use will require code that is not yet available. If the authors release the implementation, it will become clear how applicable the method is to the open-source models in our reference and what hardware requirements it imposes.