🎨 Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering

CVPR 2024
Kim Youwang1,2,4, Tae-Hyun Oh4,5,6, Gerard Pons-Moll1,2,3
1 University of Tübingen, 2 Tübingen AI Center, Germany, 3 Max Planck Institute for Informatics, Germany,
4 Dept. of Electrical Engineering, POSTECH, 5 Grad. School of AI, POSTECH, 6 Institute for Convergence Research and Education in Advanced Technology, Yonsei University
Paper arXiv Code

Paint-it: Given an untextured 3D mesh and the text prompt describing the desired appearance of the mesh, Paint-it synthesizes high-fidelity physically-based rendering (PBR) texture maps by neural re-parameterized texture map optimization.

Abstract

We present Paint-it, a text-driven high-fidelity texture map synthesis method for 3D meshes via neural re-parameterized texture optimization. Paint-it synthesizes texture maps from a text description by synthesis-through-optimization, exploiting the Score-Distillation Sampling (SDS). We observe that directly applying SDS yields undesirable texture quality due to its noisy gradients. We reveal the importance of texture parameterization when using SDS. Specifically, we propose Deep Convolutional Physically-Based Rendering (DC-PBR) parameterization, which re-parameterizes the physically-based rendering (PBR) texture maps with randomly initialized convolution-based neural kernels, instead of a standard pixel-based parameterization. We show that DC-PBR inherently schedules the optimization curriculum according to texture frequency and naturally filters out the noisy signals from SDS. In experiments, Paint-it obtains remarkable quality PBR texture maps within 15 min., given only a text description. We demonstrate the generalizability and practicality of Paint-it by synthesizing high-quality texture maps for large-scale mesh datasets and showing test-time applications such as relighting and material control using a popular graphics engine.


Paint-it: Text-driven PBR texture map synthesis

Given a 3D object mesh without a texture and a text describing the desired appearance of the mesh, Paint-it synthesizes realistic PBR texture maps via synthesis-through-optimization.


High-fidelity PBR texture synthesis on 3D meshes

The synthesized PBR texture maps are realistic and diverse. Therefore, Paint-it can reduce the burden of repetitive and exhaustive manual 3D texture map design pipeline.


Comparison with competing methods

We compare the rendered quality of the textured meshes of Paint-it with recent competing methods for the subset of Objaverse and RenderPeople dataset. Paint-it synthesizes much realistic and vivid appearance compared to other methods.


Application 1: Test-time texture control

Using the synthesized PBR texture maps of Paint-it and popular graphics engines, e.g., Blender & Unity, we can (1) relight the mesh by changing High-Dynamic Range (HDR) environmental lighting and (2) control the material properties at test-time.


Application 2: Dynamic virtual humans

Paint-it does not perform a re-meshing process and preserves the original UV texture coordinates. Thus, we can synthesize maps for any rigged meshes, e.g., T-posed human mesh, and create dynamic virtual humans from only a text input.

Citation

@inproceedings{youwang2024paintit,
    title = {Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering},
    author = {Youwang, Kim and Oh, Tae-Hyun and Pons-Moll, Gerard},
    booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
    year = {2024}
}