CapFrame Turns a Text Description of a Frame into a Camera Pose for 3D Gaussian Splatting

Researchers presented CapFrame — a method that, given a text instruction, finds the 6-DoF camera pose in a 3D Gaussian Splatting scene so that the rendered frame matches the description. The work was published on arXiv, and the code has been released as open source. The authors introduce a new task, Text-Instructed Viewpoint Grounding (TIVG), and solve it through a Retrieve-Translate-Refine pipeline: first, relevant views are selected using an MLLM, then the instruction is translated into geometric pseudo-labels for orientation and layout, after which the camera pose is refined by differentiable optimization. Details are in the arXiv paper.

What it means

CapFrame is directly related to Gaussian Splatting (3DGS) from our reference: the method works on top of scenes created by this 3D generator. The 3DGS fact sheet states that it is an open-source implementation from Inria and MPII, written in Python with CUDA extensions and requiring 24 GB of VRAM to train to the quality from the paper. CapFrame does not change the scene training process itself — it adds a stage afterward: given a finished 3DGS scene and a text query, a camera pose is searched for. For users of the reference, this means that controlling the virtual camera in 3DGS scenes may stop being manual. Currently, the 3DGS fact sheet says nothing about language-based control of the viewpoint — the original implementation focuses on novel view synthesis, not on how to choose a frame from a description. CapFrame fills this gap, but as an external tool rather than as part of the main 3DGS repository. The limitations from the paper are worth keeping in mind: the authors evaluated the method on 38 scenes and 135 instructions, which does not allow judging its behavior on arbitrary scenes. VRAM requirements for running it will have to be checked separately before integrating it into a workflow.