Hand gesture is a potentially useful modality for visual-based interaction (VBI) in human-computer interaction (HCI), but estimating hand posture and pose under camera viewpoint variations is a challenging problem due to the high intrinsic degrees of freedom (DOF) and the self-occlusion problem. This work focuses on the problem of appearance-based 3D hand posture and pose estimation.

Firstly, we propose the Isometric Self-Organizing Map (ISOSOM) algorithm, which provides a concise representation of randomly sampled data points in high dimensional space by a small number of representatives organized in a structure in a low dimensional manifold. In order to resolve the complexity problem of ISOSOM, we proposed the Hierarchical-ISOSOM (H-ISOSOM) algorithm. The hierarchical structure follows a coarse-to-fine strategy. According to the coarse global structure, it "unfolds" the manifold at the coarse level and decomposes the sample data into small patches, then iteratively learns the nonlinearity of each patch in finer levels. We compare the proposed method with similar methods on five standard data sets. Encouraging quantitative experimental results from synthetic data sets illustrate the validity of ISOSOM and H-ISOSOM.

Then we formulate the appearance-based hand pose estimation to a nonlinear manifold learning and representation problem. We apply the H-ISOSOM algorithm to learn a concise, organized representation of the hand manifold, which is a complex, extremely non-linear, large scale, high-dimensional manifold composed of three components: hand image features, hand postures, and hand poses. We represent hand images using several feature descriptors, such as shape context based on 2D intensity edge and 2.5D depth edge. Given the hand image features, the relevant representatives with the camera viewpoint ground-truth are retrieved by the learned map, which maintains the intrinsic structure revealed by a huge number of synthetic training images.

In order to resolve the ambiguity caused by hand self-occlusion, we proposed a MAP (maximum a posteriori) framework to fuse the information obtained from multiple cameras for better retrieval accuracy. The quantitative experimental results validate the effectiveness and efficiency of H-ISOSOM and MAP framework, which shows that retrieval performance of H-ISOSOM with a multi-view approach is much better than Nearest Neighbor algorithm with single-view approach.