Test-time guidance improves the geometry of image-to-3D models from partial observations

Researchers presented a training-free method that allows partial geometric observations of an object to be taken into account at the generation stage without fine-tuning the model. The approach is described in the paper Guiding Image-to-3D Generation with Test-Time Partial Observations on arXiv. The authors guide generation via a ray-consistent likelihood over an occupancy representation, combining evidence about the surface and free space. The method was tested on SAM 3D and its multi-view extension. The authors did not release the code.

What it means

The paper deals with the image-to-3d task, which is the subject of our section on 3D generators. Among the tools in the reference, this task is handled by OpenLRM, Hunyuan3D 2.1 and TRELLIS. The method from the paper is an add-on on top of an existing model, not a new model. It does not require retraining and, as the authors intend, can be applied to various generators that use an occupancy representation. In our fact sheets, none of the three tools is listed as using occupancy: OpenLRM and TRELLIS work with latent representations, Hunyuan3D 2.1 with shape and PBR materials. The authors did not test compatibility with them. There is no practical code for reproduction yet, so it is impossible to assess applicability to specific tools from the reference. If the authors publish an implementation, it will be possible to check how well the approach fits with the architectures of OpenLRM, Hunyuan3D 2.1 or TRELLIS.