Monocular hand gesture recognition systems usually model a human hand as a pixel or a blob by which the motion of the whole hand are analyzed and the appearance features are extracted. But the system presented in this paper begins with the motion of a hand's local parts. Firstly by fusing on multiple information including motion, color and edge, the characteristic curves that can represent the structure of a hand are obtained. The characteristic curves are cut into short segments with the equal length, which,which are easily analyzed and and tracked.Then the planar model is adopted to model the appearance change between the consecutive images and calculated by motion of the short segments.At last,the deviation caused bu the coordinate system is also analyzed and a moving coordinate system is set up,from which translation-independent parameters of the planar model are exrtactrd for hand gesture recognition.