IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021

Model-based 3D Hand Reconstruction via Self-Supervised Learning    
Yujin Chen1
Zhigang Tu1
Di Kang2
Linchao Bao2
Ying Zhang3
Xuefei Zhe2
Ruizhi Chen1
Junsong Yuan4
1Wuhan University
2Tencent AI Lab
4State University of New York at Buffalo


Reconstructing a 3D hand from a single-view RGB image is challenging due to various hand configurations and depth ambiguity. To reliably reconstruct a 3D hand from a monocular image, most state-of-the-art methods heavily rely on 3D annotations at the training stage, but obtaining 3D annotations is expensive. To alleviate reliance on labeled training data, we propose S2HAND, a self-supervised 3D hand reconstruction network that can jointly estimate pose, shape, texture, and the camera viewpoint. Specifically, we obtain geometric cues from the input image through easily accessible 2D detected keypoints. To learn an accurate hand reconstruction model from these noisy geometric cues, we utilize the consistency between 2D and 3D representations and propose a set of novel losses to rationalize outputs of the neural network. For the first time, we demonstrate the feasibility of training an accurate 3D hand reconstruction network without relying on manual annotations. Our experiments show that the proposed self-supervised method achieves comparable performance with recent fully-supervised methods.


This video is also accessible via bilibili.


                title={Model-based 3D Hand Reconstruction via Self-Supervised Learning}, 
                author={Chen, Yujin and Tu, Zhigang and Kang, Di and Bao, Linchao and Zhang, Ying and Zhe, Xuefei and Chen, Ruizhi and Yuan, Junsong},
                booktitle={Conference on Computer Vision and Pattern Recognition},

If you have questions regarding our project, please contact us directly by email: yujin.chen at
Thanks for the website template.