Sci-VBench: A Benchmark for Scientific Accuracy of Video from Generative Models

The authors of Sci-VBench presented a benchmark for evaluating video generation in scientific domains. It includes 1,253 expert-annotated examples across 60 disciplines from natural sciences, healthcare, humanities and social sciences, as well as engineering. Evaluation is carried out using rubrics, where the key criteria are Prompt Grounding and Scientific and Causal Correctness. The authors tested 16 proprietary and open-source models and found that, with similar visual quality scores, models diverge noticeably in scientific and causal correctness, with proprietary systems ahead of open-source ones. The authors did not release the code.

What it means

The benchmark directly concerns video generators from our reference, but none of them is named in the paper — the authors refer to 16 models without a list. Therefore, the results of specific tools cannot be compared based on this study. Still, the context matters for all three tool pages in the AI Video section. LTX-Video is an open-source image-to-video model from Lightricks; its fact sheet has no information on how it handles scientific content or causal relationships. CogVideoX from Zhipu AI tackles the text-to-video task, and its description also lacks data on the scientific correctness of generation. Pyramid Flow is an open-source text-to-video model whose fact sheet likewise does not reflect capabilities for scientific reasoning. The authors' conclusion about the gap between proprietary and open-source models in scientific correctness is worth considering when choosing a tool for tasks where not only visual plausibility matters but also substantive accuracy. At the same time, specific numbers for each model are not publicly available.