Our goal is to efficiently learn personalized animatable 3D head avatars from videos that are geometrically accurate, realistic, relightable, and compatible with current rendering systems. While 3D meshes enable efficient processing and are highly portable, they lack realism in terms of shape and appearance. Neural representations, on the other hand, are realistic but lack compatibility and are slow to train and render. Our key insight is that it is possible to efficiently learn high-fidelity 3D mesh representations via differentiable rendering by exploiting highly-optimized methods from traditional computer graphics and approximating some of the components with neural networks. To that end, we introduce FLARE, a technique that enables the creation of animatable and relightable mesh avatars from a single monocular video. First, we learn a canonical geometry using a mesh representation, enabling efficient differentiable rasterization and straightforward animation via learned blendshapes and linear blend skinning weights. Second, we follow physically-based rendering and factor observed colors into intrinsic albedo, roughness, and a neural representation of the illumination, allowing the learned avatars to be relit in novel scenes. Since our input videos are captured on a single device with a narrow field of view, modeling the surrounding environment light is non-trivial. Based on the split-sum approximation for modeling specular reflections, we address this by approximating the pre-filtered environment map with a multi-layer perceptron (MLP) modulated by the surface roughness, eliminating the need to explicitly model the light. We demonstrate that our mesh-based avatar formulation, combined with learned deformation, material, and lighting MLPs, produces avatars with high-quality geometry and appearance, while also being efficient to train and render compared to existing approaches.
@article{bharadwaj2023flare,title={{FLARE}: Fast learning of Animatable and Relightable Mesh Avatars},author={Bharadwaj, Shrisha and Zheng, Yufeng and Hilliges, Otmar and Black, Michael J. and Abrevaya, Victoria Fernandez},journal={ACM Transactions on Graphics},volume={42},pages={15},year={2023},doi={https://doi.org/10.1145/3618401},}
2020
A Novel Self-Supervised Re-labeling Approach for Training with Noisy Labels
D. Mandal*, S. Bharadwaj*, and S. Biswas
In 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), Mar 2020
The major driving force behind the immense success of deep learning models is the availability of large datasets along with their clean labels. This is very difficult to obtain and thus has motivated research on training deep neural networks in the presence of label noise. In this work, we build upon the seminal work in this area, Co-teaching and propose a simple, yet efficient approach termed mCT-S2R (modified co-teaching with self-supervision and relabeling) for this task. Firstly, to deal with significant amount of noise in the labels, we propose to use self-supervision to generate robust features without using any labels. Furthermore, using a parallel network architecture, an estimate of the clean labeled portion of the data is obtained. Finally, using this data, a portion of the estimated noisy labeled portion is re-labeled, before resuming the network training with the augmented data. Extensive experiments on three standard datasets show the effectiveness of the proposed framework.
@inproceedings{9093342,author={Mandal*, D. and Bharadwaj*, S. and Biswas, S.},booktitle={2020 IEEE Winter Conference on Applications of Computer Vision (WACV)},title={A Novel Self-Supervised Re-labeling Approach for Training with Noisy Labels},year={2020},volume={},issn={},pages={1370-1379},keywords={training;task analysis;noise measurement;data models;training data;computational modeling;robustness},doi={10.1109/WACV45572.2020.9093342},url={https://doi.ieeecomputersociety.org/10.1109/WACV45572.2020.9093342},publisher={IEEE Computer Society},address={Los Alamitos, CA, USA},month=mar}
