LTX-Video and CogVideoX Get a New Zero-Shot Captioning Method via Synthetic Data

Researchers published a paper on arXiv, Watching Synthetic Videos: Aligning Cross-modal Representations with Visual Synthesis for Zero-shot Video Captioning, proposing the WSV method for zero-shot video captioning. The approach uses a pretrained text-to-video model to generate synthetic video latents during training, then trains a "polisher" to align the distributions of real and synthetic video. At inference, the input video is encoded by a 3D Causal VAE and fed directly into a prompter that controls GPT-2 to generate captions. The authors report results of 52 B@4 and 95.7 CIDEr on the MSVD, MSR-VTT, and VATEX datasets. The authors did not release the code.

What it means

The work addresses the task of video captioning — automatically describing video content with text. This is a related area to video generation, but not the same thing: here, text is generated from video, not the other way around. In our section on video generators, there are no direct analogs to this task. The closest tools in topic — LTX-Video and CogVideoX — solve the inverse problem: LTX-Video works as image-to-video, CogVideoX as text-to-video. Both use the diffusion approach and the diffusers library, but neither is designed to describe an already finished video. At the same time, the WSV method relies on a text-to-video model as an intermediate component: it generates synthetic video latents that are then used to train the captioner. This means the quality of text-to-video generation directly affects the quality of the final captions. However, the authors do not specify which text-to-video model they used and do not publish the code, so it is currently impossible to verify compatibility with specific generators from the reference. For users looking for a video description tool, this work remains at the level of a scientific publication for now: without code and weights, it is not possible to reproduce the method or use it in your own projects.