VibeVoice-1.5B-hf: Microsoft's new open text-to-audio model on HuggingFace

The model vibevoice/VibeVoice-1.5B-hf with the text-to-audio task has been published on HuggingFace. It is designed to generate long conversational audio with multiple speakers — for example, podcasts. The author claims speech synthesis of up to 90 minutes with up to 4 speakers. The model uses continuous speech tokenizers at 7.5 Hz and a diffusion approach on top of an LLM. According to the model's fact sheet: MIT license, 2.7 billion parameters, the largest weights file is 1.9 GB, with a total of 5 GB in the repository in safetensors format. It runs via transformers. The code has been released as open source.

What it means

In our audio generation reference CPU3D there are several text-to-audio tools, and VibeVoice differs from them in a number of parameters. ChatTTS is the closest in terms of the text-to-audio task. Its weights take up 1.1 GB, the weights license is cc-by-nc-4.0, the code is AGPL-3.0. VibeVoice is noticeably heavier: 5 GB of weights, but the MIT license is more permissive for commercial use. ChatTTS requires about 4 GB of VRAM for 30 seconds of audio; for VibeVoice the author does not specify VRAM requirements. XTTS from Coqui solves the text-to-speech task, not text-to-audio. Its weights are 1.9 GB, the weights license is Coqui Public Model License 1.0.0: non-commercial use only, the code is MPL-2.0. XTTS is focused on voice cloning, whereas VibeVoice emphasizes long dialogues with multiple speakers. MMAudio is the heaviest of our tools: 50.5 GB of weights, the weights license is cc-by-nc-4.0, the code is MIT. It solves text-to-audio and video-to-audio, and requires about 6 GB of VRAM. VibeVoice is an order of magnitude lighter in weights and is focused only on text-to-audio. VibeVoice stands out for its claimed generation duration and support for up to 4 speakers, which is not reflected in the fact sheets of our tools. However, the model was published recently, and the practical hardware requirements have not yet been documented.