OptiHead: Learning to Optimize in the Energy Landscape of Parametric Head Models

We present a novel approach for real-time tracking of parametric head models from monocular RGB-D sequences. To this end, we propose OptiHead, a fully differentiable neural optimization pipeline capable of interpolating between regression and optimization by adjusting the energy formulation used for second-order solvers, enabling real-time tracking in just a few iterations. Building on projective ICP to establish correspondences, we incorporate neural weighting of residual terms using large receptive fields and regularize the optimization process with a learned prior that accounts for the uncertainty of these weights. Moreover, our method can be trained end-to-end, which was previously not done for neural optimization pipelines for 3DMM, which is achieved by utilizing mesh rasterization that is capable of forward-mode automatic differentiation, allowing efficient and differentiable computation of the Jacobian matrix used during Gauss-Newton optimization. Our method outperforms ICP for dynamic facial expressions at 21.29 FPS, allowing for real-time applications.

The OptiHead framework is structured around a few end-to-end differentiable neural optimization iterations. At each iteration, the following steps are executed: (a) the current parameters are input into the flame model and rendered as 2D normal and point maps; (b) these 2D maps are then compared against the source Kinect scan to establish correspondences and residual terms; (c) the scan and renderings are then concatenated and processed through a neural network to predict weightings and priors used during optimization; (d) a Gauss-Newton optimization step is applied, and (e) this process is repeated for N iterations to obtain the final parameters.

We develop an efficient mesh rasterizer integrated into a differentiable Gauss-Newton optimizer capable of forward-mode automatic differentiation, allowing efficient computation of Jacobian matrices. The differentiable mesh rasterizer precomputes barycentric coordinates outside the composable function transform in OpenGL and interpolates the vertex attributes in PyTorch inside the function transform, allowing to compute a differentiable Jacobian matrix using automatic differentiation that is used in the neural analysis-through-synthetis optimization. Moreover, utilizing forward-mode automatic differentiation outperforms backward-mode automatic differentiation in computing the Jacobian matrix. It enables efficient backpropagation through the entire pipeline, allowing for real-time tracking and efficient training.

Our method achieves real-time reconstruction of highly dynamic expressions in real-world settings, showing minimal visual differences compared to the tracking results from a sophisticated optimization pipeline.