VibeVoice-1.5B-hf: Microsoft's new open-source text-to-audio model on HuggingFace

The vibevoice/VibeVoice-1.5B-hf model has been published on HuggingFace with a text-to-audio task. It is designed for generating long conversational audio with multiple speakers — for example, podcasts. The author claims speech synthesis of up to 90 minutes with up to 4 speakers. The model uses continuous speech tokenizers at 7.5 Hz and a diffusion approach on top of an LLM. According to the model fact sheet: MIT license, 2.7 billion parameters, the largest weights file is 1.9 GB, with a total of 5 GB in the repository in safetensors format. It runs via transformers. The code is released as open source.

What it means

In our CPU3D reference on audio generation, there are several text-to-audio tools, and VibeVoice differs from them in a number of parameters. ChatTTS is the closest in terms of the text-to-audio task. Its weights take up 1.1 GB, the weights license is cc-by-nc-4.0, and the code is AGPL-3.0. VibeVoice is noticeably heavier: 5 GB of weights, but the MIT license is more permissive for commercial use. ChatTTS requires about 4 GB of VRAM for 30 seconds of audio; for VibeVoice, the author does not specify VRAM requirements. XTTS from Coqui solves the text-to-speech task rather than text-to-audio. Its weights are 1.9 GB, the weights license is other, and the code is MPL-2.0. XTTS is focused on voice cloning, while VibeVoice emphasizes long dialogues with multiple speakers. MMAudio is the heaviest of our tools: 50.5 GB of weights, the weights license is cc-by-nc-4.0, and the code is MIT. It handles text-to-audio and video-to-audio, requiring about 6 GB of VRAM. VibeVoice is an order of magnitude lighter in weights and focused only on text-to-audio. VibeVoice stands out with its claimed generation duration and support for up to 4 speakers, which is not reflected in the fact sheets of our tools. However, the model was published recently, and practical hardware requirements are not yet documented.