# MELON: Reconstructing 3D objects from images with unknown poses

DevFeed: [MELON: Reconstructing 3D objects from images with unknown poses](<https://devfeed.tech/articles/melon-reconstructing-3d-objects-from-images-with-unknown-poses-28561.md>)

Original publisher: [Read original article](<http://blog.research.google/2024/03/melon-reconstructing-3d-objects-from.html>)

Author: Google AI (noreply@blogger.com)

Published: 2024-03-18T18:41:00Z

Content type: article

Language: en

Sources: [Google Research](<https://devfeed.tech/sources/google-research.md>)

Topics: [3D](<https://devfeed.tech/topics/3d.md>), [Computer vision](<https://devfeed.tech/topics/computer-vision.md>), [Inference](<https://devfeed.tech/topics/inference.md>), [Google](<https://devfeed.tech/topics/google.md>)

Tags: [3d](<https://devfeed.tech/tags/3d.md>), [computer-vision](<https://devfeed.tech/tags/computer-vision.md>), [gans](<https://devfeed.tech/tags/gans.md>), [gaussian-splatting](<https://devfeed.tech/tags/gaussian-splatting.md>), [generative](<https://devfeed.tech/tags/generative.md>), [google](<https://devfeed.tech/tags/google.md>), [images](<https://devfeed.tech/tags/images.md>), [inference](<https://devfeed.tech/tags/inference.md>), [machine-learning](<https://devfeed.tech/tags/machine-learning.md>), [models](<https://devfeed.tech/tags/models.md>), [neural](<https://devfeed.tech/tags/neural.md>), [research](<https://devfeed.tech/tags/research.md>), [rgb](<https://devfeed.tech/tags/rgb.md>), [rotation](<https://devfeed.tech/tags/rotation.md>), [techniques](<https://devfeed.tech/tags/techniques.md>), [vision](<https://devfeed.tech/tags/vision.md>)

## AI overview

This Google Research article explains the challenge of reconstructing 3D objects from a small number of images when the camera poses are unknown. It covers pose inference, pseudo-symmetries, local-minimum failures, and prior approaches including NeRF, 3D Gaussian Splatting, GAN-based methods, BARF, SAMURAI, GNeRF, VMRF, SparsePose, and RUST.

## Source excerpt

Posted by Mark Matthews, Senior Software Engineer, and Dmitry Lagun, Research Scientist, Google Research A person's prior experience and understanding of the world generally enables them to easily infer what an object looks like in whole, even if only looking at a few 2D pictures of it. Yet the capacity for a computer to reconstruct the shape of an object in 3D given only a few images has remained a difficult algorithmic problem for years. This fundamental computer vision task has applications ranging from the creation of e-commerce 3D models to autonomous vehicle navigation. A key part of the problem is how to determine the exact positions from which images were taken, known as pose inference. If camera poses are known, a range of successful techniques -- such as neural radiance fields (NeRF) or 3D Gaussian Splatting -- can reconstruct an object in 3D. But if these poses are not available, then we face a difficult "chicken and egg" problem where we could determine the poses if we knew the 3D object, but we can't reconstruct the 3D object until we know the camera poses. The problem is made harder by pseudo-symmetries -- i.e., many objects look similar when viewed from different angles. For example, square objects like a chair tend to look similar every 90° rotation. Pseudo-symmetries of an object can be revealed by rendering it on a turntable from various angles and plotting its photometric self-similarity map. Self-Similarity map of a toy truck model. Left: The model is rendered on a turntable from various azimuthal angles, θ. Right: The average L2 RGB similarity of a rendering from θ with that of θ*. The pseudo-similarities are indicated by the dashed red lines. The diagram above only visualizes one dimension of rotation. It becomes even more complex (and difficult to visualize) when introducing more degrees of freedom. Pseudo-symmetries make the problem ill-posed, with naïve approaches often converging to local minima. In practice, such an approach might mistake the