Meta says its new AI model can generate 3D assets from text prompts

Meta has shared a new research paper on a generative artificial intelligence (AI) model that can generate 3D assets. The AI ​​model, called Meta 3D Gen, generates high-quality 3D models from text prompts in less than a minute. The company highlighted that the AI ​​model could create high-resolution textures and material maps end-to-end using a combination of texture generation and text-to-mesh techniques developed entirely in-house. This approach differs notably from Tencent and Stability AI, which released image-to-3D models.

Sharing her research papercurrently hosted on Meta’s website, the company claimed that its 3D Gen model can render 3D models with high prompt fidelity and guarantees high quality. The model also supposedly supports physically based rendering (PBR), allowing the generated models to be re-exposed in various real-world applications. Notably, the research paper has not been peer-verified, so some of the claims may be exaggerated. There’s no way to know for sure unless the model is made public.

The paper also explains the approaches used to develop the AI ​​model. Meta has developed two internal systems: Meta 3D AssetGen and Meta 3D TextureGen. AssetGen is responsible for generating 3D objects from a given text prompt. In phase I, it renders the 3D model using its understanding of the prompt, and in phase II, it further refines the 3D model to reconcile the view-based textures and smooths out those parts that are not visible in any view. TextureGen is responsible for finding and applying the correct textures to the model.

This method of 3D model generation has several advantages for the generated 3D assets. For example, because it uses text prompts, there is no requirement for reference images and users can generate completely unique models. The PBR support also means that these models can be taken out of the system and applied directly to other environments. The paper claims that the generation output of these models is also realistic.

Meta said its 3D Gen AI model has a 68 percent win rate in a single phase, meaning the AI ​​can generate the correct 3D asset based on the text prompt 68 out of 100 times. The company further claims the model outperforms several industry baselines in terms of prompt fidelity and visual quality for complex prompts.