Latent-to-4D generates 4D scenes directly from video latents without intermediate RGB

The authors of the paper Beyond Pixels: From Video Priors to 4D Worlds propose a method called Latent-to-4D that skips the RGB video reconstruction stage and predicts a dynamic 3D scene directly from the final denoised latents of a video model. The approach relies on the fact that video generators from the same family share a common VAE, so the latent space can be used as a portable interface. A single checkpoint, trained on roughly a thousand clips, transfers without changes between different video diffusion transformers within the same VAE family. The authors have not released the code.

What it means

The work concerns the "video generator — 4D scene" pipeline, which in our reference falls under the AI video and 3D section. Among the tools we cover, the closest in topic is Wan: it is an open-source video generator from Alibaba with a text-to-video task, code and weights available under Apache-2.0, and the model runs through diffusers. The Wan fact sheet does not mention that it can expose intermediate latents for downstream tasks such as 4D reconstruction. The Latent-to-4D authors mention Wan in the context of comparison with "video + 4D reconstructor" cascades, but that comparison is within their own benchmarks, not a stated integration with Wan itself. The Wan developer does not describe such a use case. There is little practical takeaway for the user here for now: there is no code, the checkpoint is not published, and the method itself depends on access to the latents of a specific VAE family. For those working with Wan, this is more of a signal about the direction in which the video-and-4D pipeline may evolve than a ready-made tool.