Coarse3D: Accurate 3D Model Generation from 2D Images

Paper: Coarse3D: Controllable and Faithful Sparse Structure Generation for Image-to-3D

Authors: Gaige Wang, Bowen Pan, Hao Ren, Jiangjing Lyu, Chengfei Lyu

Executive Summary

Recent research introduces Coarse3D, a new model for generating 3D structures from 2D images that addresses critical limitations faced by existing methods. Current approaches often struggle with two major issues: controlling the orientation of the generated 3D models and ensuring that these models accurately reflect the input images. Coarse3D aims to overcome these challenges by focusing on producing a more reliable and precise 3D representation.

The core innovation of Coarse3D lies in its ability to produce voxel-based geometries that are both orientation-controllable and structurally faithful to the images being used. This is particularly important for industries that rely on accurate 3D modeling, such as gaming, virtual reality, and architectural design. The model’s design includes several key features that enhance its performance:

1. First-Frame Anchored Canonical Space: This feature aligns the generation process with the initial view of the input image, which helps maintain a consistent orientation for the generated 3D structure while normalizing the object's perspective.

2. Dual-Tower DiT Architecture: This architecture couples the generation of 3D structures with an auxiliary feed-forward reconstruction process. This coupling helps the model learn spatial features that are more grounded in the 2D image, resulting in more accurate representations.

3. Spatial-Prior Guided Attention: This mechanism improves the model's ability to correspond pixels from the 2D image to the voxel representation in 3D. By regularizing the cross-attention process, it enhances the model's accuracy in generating structures that align closely with the original images.

The research demonstrates that Coarse3D can significantly improve the quality of sparse structures in the initial stages of 3D generation. In benchmark tests, Coarse3D outperformed existing models, yielding state-of-the-art results in the final 3D generation process.

While these findings are based on experiments and benchmarks rather than real-world applications, the implications for various industries are promising. Companies in sectors like gaming, film production, and design could potentially utilize this technology to create more accurate and visually appealing 3D models from standard images. As the development of Coarse3D progresses, businesses may find new opportunities to enhance their visual content creation processes, leading to improved customer experiences and more immersive environments.

Academic Abstract

Coarse-to-fine 3D-native generation relies heavily on the first-stage sparse structure, which determines the global orientation and overall geometry for subsequent refinement. However, existing coarse generators suffer from two key limitations: their orientation is difficult to control due to ambiguous semantic canonicalization, and their structures are often inconsistent with input images due to weak 2D-3D grounding. We propose Coarse3D, a sparse-structure generation model that produces orientation-controllable and structurally faithful voxel geometry from arbitrary unposed images. Coarse3D introduces three designs: a first-frame anchored canonical space that aligns generation with the first input view while preserving object-centric normalization; a dual-tower DiT that couples generation with auxiliary feed-forward reconstruction to learn spatially grounded features; and spatial-prior guided attention that regularizes cross-attention with pixel-to-voxel correspondence. Coarse3D can directly replace the coarse stage of existing pipelines. Experiments show that it substantially improves sparse structure quality and consistently enhances final 3D generation, achieving state-of-the-art performance.