NeuraLeaf: Disentangled neural parametric modeling of leaf shape, deformation, and appearance

We develop a neural parametric model of 3D leaves for plant modeling and reconstruction that are essential for agriculture and computer graphics. While neural parametric models are actively studied for humans and animals, plant leaves present unique challenges due to their thin surface geometry and flexible deformation. To address this problem, we introduce a neural parametric model for leaves, NeuraLeaf. Capitalizing on the fact that flattened leaf shapes can be approximated as a 2D plane, NeuraLeaf disentangles the leaves’ geometry into their 2D base shapes (i.e., flattened leaves) and 3D deformations. This representation allows learning from rich sources of 2D leaf image datasets for the base shapes, and also has the advantage of simultaneously learning textures aligned with the geometry. To obtain a disentangled and controllable appearance space, NeuraLeaf also introduces a learnable appearance space with self-supervised objectives, enabling both appearance fitting to observations and appearance-conditioned texture generation. To model the 3D deformation, we propose a novel skeleton-free skinning model and create a newly captured 3D leaf dataset called DeformLeaf. We demonstrate that NeuraLeaf successfully generates a diverse range of leaf shapes with high-fidelity appearance and deformation, resulting in accurate model fitting to RGB-D observations.