CapFrame converts a text description of a frame into a camera pose for 3D Gaussian Splatting

Researchers presented CapFrame, a method that, given a text instruction, finds a 6-DoF camera pose in a 3D Gaussian Splatting scene so that the rendered frame matches the description. The work is published on arXiv, and the code is released as open source. The authors introduce a new task, Text-Instructed Viewpoint Grounding (TIVG), and solve it with a Retrieve-Translate-Refine pipeline: first, relevant views are selected using an MLLM, then the instruction is translated into geometric pseudo-labels for orientation and layout, after which the camera pose is refined with differentiable optimization. Details are in the arXiv paper.

What it means

CapFrame is directly related to Gaussian Splatting (3DGS) from our reference: the method works on top of scenes created by this 3D generator. The 3DGS fact sheet states that it is an open-source implementation from Inria and MPII, written in Python with CUDA extensions, requiring 24 GB of VRAM for training to the quality described in the paper. CapFrame does not change the scene training process itself — it adds a stage after: given a ready 3DGS scene and a text query, it searches for a camera pose. For reference users, this means that controlling a virtual camera in 3DGS scenes may no longer be manual. Currently, the 3DGS fact sheet says nothing about language-based viewpoint control — the original implementation focuses on synthesizing new views, not on how to select a frame by description. CapFrame fills this gap, but as an external tool, not as part of the main 3DGS repository. Limitations from the paper should be noted: the authors evaluated the method on 38 scenes and 135 instructions, which does not allow conclusions about behavior on arbitrary scenes. The license of the CapFrame code is not stated in the description, nor are the VRAM requirements for running it — these parameters will need to be checked separately before integrating into a workflow.