VGI-Bench evaluates visual reasoning of video generators on 810 tasks

arXiv published VGI-Bench, a benchmark of 27 tasks and 810 examples for testing visual reasoning in video generation models. The authors built a two-level taxonomy of domains and skills, requiring models not just to produce a plausible final frame but to correctly develop the process, and calibrated the difficulty so that tasks remain partially solvable. The strongest tested system, Seedance 2.0, scores 51.0% on the benchmark criteria. The authors did not release the code.

What it means

The benchmark directly concerns tools from the video generators section of our reference, but none of the tested models are among those we describe. LTX-Video is a video generator from Lightricks with an image-to-video task, open source code and weights. The tool's fact sheet has no information about visual reasoning or results on any benchmarks: the authors describe it as a repository for video generation and note low VRAM requirements. CogVideoX from Zhipu AI solves the text-to-video task, with open source code. The fact sheet also has no data on visual reasoning evaluation — only technical specs and descriptions of generation tasks. Pyramid Flow is a text-to-video generator with open source code; the fact sheet lists VRAM requirements and MPS support, but no information about visual reasoning capability. VGI-Bench sets a new bar for evaluating video generators, but it does not yet overlap with the tools in our reference: none of the generators we describe were tested in the work, and their fact sheets contain no data that could be compared with the benchmark results.